How to Evaluate an AI Web Crawler Before You Rebuild It

Most AI crawler debates are architecture cosplay. One person wants a sitemap. Another wants recursive exploration. Someone adds a validator because “more AI” sounds safer. Three weeks later, the code is clever, the pipeline is slower, and nobody can prove it finds more of the right pages. If you are choosing an AI web crawling strategy, do not start with a rewrite. Start with an eval. Separate page discovery from data extraction, run both approaches against the same representative sites, and measure the failures that can actually poison your database.

The most expensive crawler error happens before extraction

A beautifully extracted phone number from the wrong website is still the wrong phone number. This sounds obvious, but crawler pipelines often spend their sophistication downstream: better prompts, smarter deduplication, stricter schemas. None of that repairs a bad domain match or an irrelevant page set. That is why every stage needs its own contract. Website selection should establish that you have the right domain. Discovery should identify the pages likely to contain the target information. Extraction should turn those pages into grounded records. Deduplication should merge only after provenance has been preserved. Do not bolt on an AI validation step just because it feels prudent. A validator can create false positives and false negatives of its own. Add it only when an eval proves that it improves end-to-end accuracy.

Compare sitemap selection with recursive exploration

A sitemap-first crawler begins with an explicit inventory. The Sitemaps protocol defines the XML format and sitemap indexes, while Google’s documentation explains how sites can advertise sitemap locations through robots.txt. The crawler can number the URLs, ask a model which IDs are likely to contain the target data, and fetch only those pages. The attraction is reproducibility. The candidate set is visible. You can inspect what the model rejected. You can rerun the same list with a different prompt or model. The weakness is equally clear: sitemaps are not guaranteed to contain every useful page, and very large inventories create context and selection problems. A recursive crawler starts from a small frontier, usually the homepage, then follows selected links level by level. It can discover pages missing from the sitemap and avoid loading thousands of irrelevant URLs into one prompt. But it needs disciplined rules for depth, deduplication, canonicalization, rate limits, and robots.txt compliance. If those rules change between runs, your comparison becomes theatre. Neither method wins in theory. A hybrid may be best: deterministic URL matches for obvious routes, followed by model selection when the path is ambiguous. But “hybrid sounds sensible” is not evidence. Keep it as a third branch and make it earn its complexity.

Build a two-stage AI web crawler evaluation

The first stage scores discovery. Give each crawler the same site sample, access rules, retry policy, rendering budget, and maximum work budget. Then compare relevant-page recall, target-record coverage, irrelevant pages visited, duplicate pages, wall-clock time, requests, tokens, and estimated model cost. The second stage scores extraction using the pages produced by stage one. Compare field accuracy, schema-valid output, groundedness, duplicates, and unsupported values. This distinction matters because a crawler can discover the perfect page and still extract badly—or extract perfectly from a page the other method never found. There is no single benchmark that exactly answers your private schema and website distribution. Adjacent work such as WebWalkerQA studies multi-hop web traversal, while the Web Content Extraction Benchmark focuses on separating main content from boilerplate across varied pages. They are useful references, not substitutes for your production-shaped test set.

Use a representative test set, not your friendliest sites

A thousand sites is useful only if the sample represents the mess you will face in production. Include tiny brochure sites, giant sitemap indexes, JavaScript-heavy pages, pagination, duplicate bios, ambiguous navigation, localized paths, missing sitemaps, and pages where the target data appears in several places. OpenAI’s evaluation guidance recommends defining the objective, collecting a dataset that reflects real use, specifying metrics, and continuously expanding the eval as failures appear. That last part matters. Your first benchmark is not a certificate. It is the beginning of regression testing. Keep the ground truth inspectable. For each site, record the accepted domain, relevant pages, expected entities, and known edge cases. When two methods disagree, you should be able to open the evidence and understand why.

Make source tracking part of the schema

Do not ask a model to reproduce long URLs beside every extracted field. Number the source pages and require compact source IDs with each value. If a job title appears on two pages, return both IDs. You reduce output noise, avoid URL transcription mistakes, and preserve a trail a human or evaluator can follow. Use schema-constrained output where available. OpenAI’s Structured Outputs can enforce JSON Schema conformance, but valid JSON is not the same as true data. Score schema validity and semantic correctness separately.

Let the specification survive the code

AI makes code cheap enough to rewrite. That changes what deserves to be permanent. The specification should define inputs, stage contracts, failure handling, metrics, and the report format. The code is one implementation of that document. Build each crawler approach on its own branch. Run both through the same eval harness. Keep a known-good output fixture. When the implementation gets tangled, update the doc with what you learned and regenerate the code against the clean contract instead of preserving every accidental abstraction like a family heirloom. This is also how I like to delegate with Notis. The useful unit is not “please improve my crawler.” It is a testable assignment: implement this branch from this spec, run this eval, return the HTML report, and explain every regression. Messaging-native delegation is fast, but the acceptance criteria still need teeth.

The report should make the decision boring

Your final report should show the distributions and failure cases, not just averages. A method that is marginally cheaper but consistently misses paginated directories may be a terrible trade. A method that finds more pages but doubles irrelevant fetches may still win if extraction coverage is the business-critical metric. The goal is not to crown the cleverest crawler. It is to produce enough evidence that the next engineering decision feels boring. Choose the simplest approach that meets the quality threshold, archive the eval, and rerun it whenever the model, prompt, crawl policy, or target-site distribution changes. Do that before the rewrite. Otherwise you are not rebuilding a crawler. You are replacing one set of undocumented assumptions with a newer, more expensive set.