The demo version of a scraper is about twenty lines. Launch a browser, log in, read a page, return a value. It works on the first try and it feels like cheating.
The production version is mostly the other stuff. I would put the ratio at one part extraction to nine parts keeping the extraction alive, and the nine parts are what determine whether anybody can rely on it.
Only reach for a browser when the door is actually closed
I did not want a browser in this pipeline. The richest source of the data had no public API and no visible intention of building one, and the alternative was accepting worse data for the sake of a cleaner architecture. That trade was not worth making.
So the browser exists, and everything after this point is about containing it.
Authentication is the whole problem
The service authenticates itself. It logs in, holds short-lived headers, and hands them to the rest of the workflow through a small API protected by a key. Nothing upstream knows a browser is involved, which is the point. When I replace this approach later, nothing else changes.
On a 401 it retries once. Not three times, not with backoff. Once. A single retry distinguishes an expired session, which is routine, from a broken selector or a changed login flow, which is not. Retrying five times turns a real failure into a slow failure and buries the signal you needed.
Make failures structured, not silent
Endpoint errors come back in band. A caller gets a structured failure it can branch on rather than a timeout it has to guess about. This sounds obvious and it is the thing I see skipped most often, because a timeout looks like a network problem and nobody investigates network problems.
Alongside that: periodic health checks, and a force-refresh endpoint so I can invalidate a session by hand without redeploying. That endpoint has saved me more evenings than any test I wrote.
Alert once per outage
Alerts throttle per key. One broken source produces one message, not forty as every queued record fails in turn. An alerting system that cries wolf gets muted inside a week, and a muted alerting system is worse than none because you believe you have coverage.
What actually breaks
Sessions expire on the day you are travelling. Markup shifts without warning because somebody shipped a redesign nobody told you about. Rate limits appear overnight when your volume crosses a threshold you did not know existed.
None of that is exotic. All of it is predictable enough to budget for, and the budget is what people skip. If you are choosing to scrape, you are choosing an operational tail, and the honest version of the decision includes the cost of carrying it.
Wrap the browser behind a clean interface and the rest of your system stays indifferent to how bad a week the browser is having.