The screen that should be bad at its job
Before crawling anything at width you have to drop the dead domains. The instinct is to make that filter clever, and that instinct quietly costs you real customers.
Any pipeline that reads websites in bulk starts with a cheap screen: check the domain resolves, check it is not parked, make one request, drop what is dead. It keeps the expensive stages from spending time on nothing.
The design question is how aggressive that screen should be, and the intuitive answer is wrong.
Two errors that are not the same size
A screen can fail in two directions.
It can keep something dead - a parked domain, an expired site, a placeholder. The cost is one wasted request. You find out immediately, because the next stage gets nothing usable.
It can drop something alive - a real business whose site is merely unusual. An odd DNS setup, a slow host, a homepage that redirects twice, a page that renders late. The cost is that a real prospect leaves the pipeline permanently, and you never find out. They do not appear in a report. They are not in a failure log. They are simply absent from every list you build from that day forward.
Those two mistakes get treated as symmetric by almost every filter I have seen written. They are not close.
Bias the screen toward keeping
So the rule is: drop what is provably dead, not what looks odd.
No DNS record is provable. Nameservers pointing at a known parking provider is provable. A response that is unambiguously a placeholder is provable. Everything else - slow, weird, redirecting, minimal, ugly - survives to the next stage, where a real fetch either produces content or does not.
You are not trying to make the screen accurate. You are trying to make it cheap and safe, and let the expensive stage make the actual judgement with better information.
Why this generalises
The pattern shows up everywhere in data work: a cheap early filter that permanently removes records, sitting upstream of the stage that could actually evaluate them.
Deduplication that merges on a fuzzy match. A verification step that drops "risky" rather than tagging it. A classifier with a confidence floor that discards instead of routing to review. Each one silently shrinks the universe, and because the discarded rows are gone, nothing downstream can tell you what it cost.
The general form: an irreversible filter should require more evidence than a reversible one. If a step deletes, it needs proof. If a step tags, a guess is fine, because something later can overrule it.
The version that survives contact
In practice the early screen tags more than it drops. A domain that looks strange gets marked strange and carries on. If the crawl later produces nothing, the row exits with two pieces of evidence instead of one, and the pattern is available to whoever tunes the screen next.
That is the real payoff. A screen biased toward keeping produces a record of its own near misses. A screen biased toward dropping produces silence, and silence cannot be debugged.