A status code is not evidence that a page loaded

Every enrichment pipeline eventually has to read websites at width. Thousands of domains, most of them small businesses, most of them fine, a meaningful minority of them broken, parked, or actively hostile to being read by anything that is not a person.

The naive version of this is a loop that fetches a URL and checks whether the response came back 200. It works right up until it doesn't, and when it fails it fails silently, which is the expensive kind.

What a 200 actually tells you

Nothing much. A 200 OK means a server chose to respond. It does not mean the response is the page you asked for.

The same status code is returned by a bot challenge, an interstitial, a cookie wall, a parked-domain placeholder, a soft 404 that serves the homepage for every unknown path, and a JavaScript shell whose actual content arrives later from somewhere else. All of those look identical to a status check. All of them will be recorded as a successful crawl, and their contents will flow downstream into whatever you build next.

That is the real failure. A crawl that errors out is annoying but honest. A crawl that succeeds and hands you a challenge page is a lie your entire pipeline now believes.

The gate is a content decision, not a transport one

The fix is to stop asking the transport layer whether things went well and start asking the body. Before a response is accepted as real, something has to look at what came back and decide whether it is a page about a business or a page about the crawler.

That check is the load-bearing part of the whole design. It is what lets everything else be cheap, because once you can reliably tell "this is content" from "this is not content," you no longer need to be careful by default. You can be fast by default and careful on demand.

Two passes, and the cost asymmetry that justifies them

A plain HTTP request is close to free. A real browser is not: it costs memory, time, and roughly an order of magnitude more per page. If you run a browser on every domain because some domains need one, you are paying the worst-case price on the entire universe.

So the first pass is the cheap one, and the large majority of the internet answers it perfectly well. Only responses that fail the content gate escalate to the expensive lane, where a real browser executes the page properly.

The economics are the whole argument. Coverage width is a function of per-domain cost, and per-domain cost is dominated by how often you reach for the expensive tool. Push that number down and you can afford to look at far more of the market.

Choosing pages instead of crawling them

The second cost lever is how much of each site you read. Crawling a site blindly means following links until something tells you to stop, and on a large site that is a great deal of fetching for information you did not need.

Most sites publish a map of themselves. Using it to select a small number of pages that actually answer your questions — what this business does, where it operates, how it talks about itself — replaces an open-ended crawl with a bounded one. Bounded work is work you can schedule, price, and retry.

Being biased toward keeping

There is one more decision that matters more than it looks, and it runs the other way.

Before any of this, a fast screen removes domains that are provably dead. The temptation is to make that screen clever and let it drop anything that looks unhealthy. That instinct is wrong. A screen that is eager to drop will quietly remove real businesses whose sites are merely unusual — an odd DNS configuration, a slow host, a homepage that redirects twice.

So the screen is deliberately biased toward keeping. It drops what is provably dead, not what looks odd. A false positive there costs you a real customer forever and you will never know it happened. A false negative costs one cheap HTTP request. Those are not the same mistake, and the design should not treat them as if they were.

The general principle

Most of this generalises past crawling: decide on the artifact, not on the signal that the artifact was delivered. A status code, an exit code, a webhook receipt, and a 200 on an API call are all claims that something was transmitted. None of them are claims about what.

Systems that check the delivery signal are fast to build and fail invisibly. Systems that check the artifact cost slightly more everywhere and fail loudly, which is the only kind of failure you can actually fix.