Two-Pass Crawl Engine

Before anything gets crawled, each domain gets a fast screen — a DNS check, a parked-nameserver check, and one classified request — deliberately biased toward keeping: it drops what is provably dead, not what merely looks unusual. Survivors go through a two-pass fetch. Plain HTTP handles the large majority, and only when a content gate says the response is not real content does the request escalate to a full browser. Pages are chosen from the site's own sitemap rather than crawled blindly.

Deciding on content rather than status code is the important part. A site can return a perfectly healthy 200 alongside a challenge page or a stub, and a status check will happily record that as success — which quietly poisons everything downstream.

The crawled pages become a written business summary — model, reach, history, context — which becomes the evidence every later step reads. Classification works from that summary against the graded store and returns scope, match, confidence, and reach; routing off that decision is fully deterministic. No later step re-reads raw pages and quietly forms its own contradictory opinion.