AI Training Data Collection Services

No, and any vendor who says yes is overreaching. The legal position on training on web content is genuinely unsettled and actively litigated in several jurisdictions, and it depends on where you are, what you collect and what you build.

What we can do is give your counsel accurate facts to reason from: per-document provenance, the robots.txt snapshot in effect at fetch, opt-out signals detected, declared licence and how we determined it, and a complete log of what we excluded and why. That is the difference between a legal question your counsel can answer and one they cannot.

Yes, at fetch time, and we record the robots.txt snapshot in effect so the claim is verifiable rather than asserted. Text-and-data-mining opt-out signals are detected and honoured, and documents excluded for that reason appear in the manifest with exclusion_reason.

This does reduce coverage on some sources, and we tell you which parts of a requested scope it removes during scoping. A vendor whose coverage is unaffected by opt-out signals is probably not checking for them.

Because it cannot be reconstructed. The robots.txt in effect on the day you fetched, the opt-out signals present then, and the licence the page declared at that moment are all facts about a past state. Checking today tells you today's position, not the position when the document entered your corpus.

The pattern we see repeatedly: a team collects fast, trains a good model, then enterprise procurement asks how the data was sourced and there is no answer. The remedy at that point is retraining on an accounted corpus, which costs vastly more than recording the manifest would have.

Around 31% in our typical runs, counting exact and near-duplicates but before considering partial overlap. Boilerplate within documents — navigation, cookie notices, footers — can be another third of raw text on a typical page.

We cluster near-duplicates and retain near_dup_cluster on every document rather than silently deleting, because dedup aggressiveness is a modelling decision. You collapse clusters however your objective requires without recollecting anything.

Detection with category flags as standard, and optional redaction configured to your risk posture. Common categories — names in certain contexts, email addresses, phone numbers, identifiers — are flagged with pii_flags, and pii_redacted records whether redaction was applied.

We do not claim detection is exhaustive, because no detector is. What we provide is a documented method, a measured recall figure per category on your own sample during the pilot, and the flags so your own downstream controls have something to work with.

Yes, and it is a growing share of this work. RAG needs different treatment: chunking with stable IDs, heading and section structure preserved, per-chunk source attribution for citation, freshness timestamps, and incremental refresh so the index does not drift stale.

The compliance requirements are the same — arguably more visible, since a RAG system cites its sources to end users. A citation pointing at content the publisher asked not to be used is a problem your users can see.

Yes. Held-out sets are constructed with time partitioning where the domain allows, and we check overlap against the training corpus using the same near-duplicate clustering, reporting contamination rates rather than asserting cleanliness.

The honest limitation: we can check against corpora we can see. If your model was pretrained on a public corpus we do not hold, we cannot verify overlap with it. We say which corpora the contamination check covered rather than implying a general guarantee.

Language identification with a confidence score runs on every document, and per-language quality thresholds are configurable, since a threshold tuned for English performs badly on morphologically rich or low-resource languages.

For genuinely low-resource languages, honest expectation setting matters: the web simply contains less text, and much of what exists is machine-translated. We flag suspected machine translation where detectable, because training on it degrades quality in ways that are hard to diagnose later.

We quote individually. The drivers are corpus scale, how many sources need individual vetting, processing depth — dedup, quality filtering, PII handling — and whether you need ongoing refresh or a one-time build.

Domain-specific corpora with careful source vetting cost more per token than bulk collection, and that is where the value is: general web crawl is largely a commodity through public corpora. One scoping call, a free pilot corpus with full manifest within 48 hours, then a fixed quote — monthly for ongoing refresh, project-based for one-time builds. Request a quote.