AI Crawler Access: Deciding What to Allow, and What Blocking Actually Costs

Introduction

Someone forwards a blog post about AI companies scraping the web. A directive gets added to robots.txt blocking every AI-related user-agent anyone could name. Six weeks later, nobody has measured anything, and the company has quietly removed itself from a class of answers where its own pages were the best available source. No alert fired. The cost does not appear in any dashboard.

This is the most common failure in AI crawler policy: treating one decision as three, and making it in the direction that feels safest without measuring the consequence. The categories differ, the costs differ, and the mechanisms differ in how enforceable they are.

This post is about access — who is permitted to read your pages. What those readers find when they arrive is a separate question, covered in our companion post on agent-ready product data feeds.

Three categories, not one

Lumping AI crawlers together is the core mistake. As of late 2026, three distinct behaviours matter, and they carry opposite trade-offs. Note that individual vendors change crawler names and split or merge functions frequently, so verify current identities against the operator's published documentation rather than a list you saved last year.

Training-corpus collection. A crawler gathering text at scale to be used in model training. Fetches are broad, not tied to any user's immediate request. Blocking this has no direct effect on whether you appear in answers today. The argument for blocking is about compensation and control over your content, which is a legitimate business position.

Live retrieval for a user's current question. A user has asked something right now, and the assistant fetches a small number of pages to answer it, typically with a link or citation back. This is the category where blocking is usually a mistake. If the fetch fails, your page is not in the answer. Someone else's is. The citation, the referral, and the framing of the topic all go elsewhere.

Answer-engine indexing. A crawler building a retrievable index that an assistant queries later. Behaviour resembles classic search indexing, and the visibility consequence of blocking resembles blocking a search engine: you are not findable in that surface.

The second row is where most of the unmeasured damage happens. Blocking live retrieval is a visibility decision dressed up as a security decision. Nothing is protected — the page is public, and a human can read it — but you have opted out of being the answer.

Your content is already public, or it is not

A useful test before writing any rule: is this page reachable by an unauthenticated visitor?

If yes, blocking a crawler does not protect it. It changes who can conveniently read it, and the people who go around your rules are not the ones with a published user-agent. If the content genuinely needs protecting, it needs authentication, not a robots directive.

If the page is behind a login, the discussion is moot. Crawler policy governs public surfaces.

This clears out a lot of confused reasoning. What remains is the real question: for public content, which readers do you want, and why.

The mechanisms, and what each actually enforces

Two honest notes.

robots.txt is a request. A crawler that ignores it faces no technical obstacle. It is a statement of policy, useful precisely because the major operators generally honour it — and useless against anyone who does not.

llms.txt, as of late 2026, is a convention rather than a standard. Support is uneven and the format is still being discussed. The sensible posture is to publish one because it is cheap and describes your site usefully, while assuming nothing about who reads it. Check the current convention before you write yours; do not copy a format from an old example.

If you need a limit to be real, it must be implemented server-side. Everything above that line is a signal.

Deciding per category

Work through this rather than applying one global rule.

The path-level row matters more than it looks. Most sites have sections with genuinely different answers: documentation you want cited, pricing pages you may want cited, proprietary research you may want licensed rather than absorbed, and internal tools that should not be public at all. A single site-wide directive throws away that nuance.

Write down the reasoning next to each rule in a comment. Six months from now, nobody will remember why a line is there, and undocumented rules survive long past their justification.

Measuring the consequence

A policy change with no measurement is a guess. The measurement is a data-collection exercise, and it has to start before the change.

Establish a baseline first. Assemble a set of prompts that represent how people actually ask about your category — buying questions, comparison questions, how-to questions. Query them across the assistants that matter to you, on a fixed schedule, and record which sources are cited.

Then change the policy, hold it for a meaningful window, and re-measure the same prompts. Two cautions. Answer surfaces are non-deterministic, so single observations are noise; you need repeated sampling. And these systems change independently of anything you do, so keep a control set of prompts where your policy did not change.

If you block live retrieval and citation presence falls, you have quantified the cost. That is a number you can put in front of whoever asked for the block.

The reciprocal question

If your business depends on public web data — competitor pricing, marketplace listings, public reviews, SERP data — then a blanket "no automated reading of our site" position deserves examination.

There are defensible positions here. Reading public listings to build a pricing signal, and having your editorial content absorbed into a model that competes with it, are not the same activity. You can reasonably distinguish factual public data from original creative work. You can reasonably distinguish attributed live retrieval from unattributed bulk ingestion.

What is not defensible is not having thought about it. If your policy and your own data practice point in opposite directions, someone will eventually ask, and "it seemed safer" is a weak answer. Write down the distinction you are drawing. If you cannot articulate it, that is information.

Verifying identity, because user-agent strings are free

Any rule keyed on user-agent alone is trivially bypassed. A string in a request header is whatever the sender types. This cuts both ways: bad actors impersonate legitimate crawlers to inherit their access, and your logs misattribute traffic, which corrupts the measurement above.

The actual checks:

• Reverse DNS with forward confirmation. Resolve the IP to a hostname, confirm the hostname belongs to the claimed operator's domain, then resolve that hostname back and confirm it returns the same IP. Both directions are required; one direction alone is spoofable.
• Published IP ranges. Several operators publish their crawler ranges. Where available, matching against the published list is cleaner than DNS. Refresh the list on a schedule — ranges change without announcement.
• Behavioural signals. Request rate, path patterns, header consistency and whether robots.txt was fetched at all. A "crawler" that never reads robots.txt and hammers one path is not what it claims to be.

Log the verification outcome, not just the raw user-agent. Otherwise your before-and-after measurement is built on a field anyone can forge.

A note on scope

This is a publisher's policy decision, not legal advice. The rules that apply to you depend on your jurisdiction, your content, your terms of service and your contractual commitments. Where the stakes are material, get counsel who knows your situation.

The conventions themselves are also unsettled as of late 2026. Crawler identities, directive syntax and emerging file formats are moving. Treat your policy as something with a review date, not a configuration you write once. Check the current specification of any convention before you implement it.

Actowiz Solutions builds the measurement layer around this decision — AI answer citation tracking against a defined prompt set, verified crawler-identity analysis from server logs, and before-and-after comparison on policy changes. We work as a data partner within the compliance posture you set, not around it.

Request a free sample dataset →

FAQ

Q: Does blocking AI crawlers in robots.txt actually stop them?

A: robots.txt is a request, not an enforcement mechanism. Major operators generally honour it, but a crawler that ignores it faces no technical obstacle. Only server-side controls such as WAF rules, rate limits and IP restrictions actually enforce a limit.

Q: What does blocking AI crawlers cost?

A: It depends on the category. Blocking training-corpus collection has no immediate visibility cost. Blocking live retrieval removes your pages from answers where you could have been cited, sending the citation and the referral to a competitor instead.

Q: How do you verify a crawler is who it claims to be?

A: User-agent strings are trivially spoofed. Use reverse DNS with forward confirmation — resolve the IP to a hostname, confirm it belongs to the claimed operator, then resolve back to the same IP — or match against the operator's published IP ranges where these exist.

Conclusion

You can also reach us for all your mobile app scraping, data collection, web scraping, and instant data scraper service requirements!