Enterprise Web Crawling Services

Scraping fetches a list of URLs you already have. Crawling discovers the URL space first — enumerating pages you do not know about — and then fetches it.

That changes the failure modes. With scraping you can verify completeness against your list. With crawling the target size is unknown, so coverage has to be estimated with a stated basis, and the frontier has to be controlled or the crawl expands into an effectively infinite space and never finishes.

Strictly, you do not, which is why we report frontier_remaining and coverage_est_pct with a coverage_basis naming how the estimate was derived — sitemap comparison, link graph saturation, or both.

A crawl that ended with 400,000 URLs still queued is not complete, and a report that omits the frontier is hiding the one number that would tell you. We would rather report 98.2% with a basis than claim completeness nobody can verify.

Frontier control: URL normalisation with parameter allow-lists, facet combination budgets, depth and breadth caps per page type, trap detection for calendars and sort loops, and content hashing so duplicate pages do not seed further expansion.

We report the ratio of URLs fetched to distinct pages retained per run. Close to one means the crawl is controlled. Above twenty means it is fetching the same content repeatedly — cost you paid and load the target absorbed for nothing.

Slower than you would like, deliberately. Polite crawling means capped request rates per host, limited concurrency and backoff on errors, which puts a full crawl of a large retail site in hours or days rather than minutes.

We would rather quote a realistic duration than a fast one that ends in a block. Where speed genuinely matters, the honest answer is usually to narrow the URL space rather than raise the rate.

Yes, and we retain the snapshot in effect per run so the claim is verifiable rather than asserted. We also cap rates, limit concurrency and back off on rising error rates.

This is self-interest as much as ethics: aggressive crawling ends in blocking, and demonstrable burden on infrastructure moves any dispute from access questions to harm questions, which is a much worse position for the client.

Canonical resolution where a site declares one, plus URL normalisation and content-hash based near-duplicate clustering. The same page under six sort parameters is six URLs and one page, and we count pages.

Both the URL count and the distinct page count are reported, because the gap between them is diagnostic. A widening gap usually means the site changed its URL structure, and that is worth catching early.

Yes, and this is where most ongoing value sits. Change rates are observed per page type, recrawl intervals are set accordingly, and content hashing flags whether a page actually changed since the last fetch.

The effect is that effort follows change: product pages might recrawl hourly while policy pages recrawl monthly. Total fetch volume typically falls while freshness improves, because a single blanket interval is wrong for every page type simultaneously.

Yes, and it is a common first engagement. We crawl the same target independently and report coverage, frontier state and duplication, which quantifies what your current crawl is missing.

It is also a low-commitment way to evaluate us: you already know the target, so you can judge our coverage figure against your own data rather than against a claim.

We quote individually, and the drivers here differ from our other services. Cost follows URL space size, recrawl frequency and rendered share — not record count, since records are the output rather than the input.

A category-scoped crawl of one large site refreshed weekly sits at the lighter end. Multi-domain crawls in the tens of millions of URLs with daily change detection and heavy rendering sit considerably higher. One scoping call, a free pilot crawl on your own target within 24 hours with a coverage report, then a fixed monthly quote. Request a quote.