AI crawlers and robots.txt: How websites say which AI uses are allowed

A robots.txt file is a short public text file at the top of a website that tells automated visitors, called crawlers, which pages they may fetch. AI companies now use crawlers to collect web pages for training models and for answering questions. The file is how a publisher says yes or no to each of them.

It is a polite request, not a lock. That gap is at the center of a new regulatory case in the UK.

A sign on the door, not a lock

Picture a shop with a sign that reads “Deliveries round the back; no entry through the garden.” Honest couriers follow it. Nothing physically stops anyone else. robots.txt works the same way.

A site owner writes a rule such as User-agent: GPTBot followed by Disallow: /. The first line names a crawler, and the second says it may not fetch anything on the site. A well-behaved crawler reads the file before it visits and obeys.

The standard says plainly what this is not. RFC 9309 states that the rules are not a form of access authorization and are no substitute for real content security. Listing a path in the file also makes that path publicly visible.

Different bots for different jobs

Early on, one crawler per search engine was enough. AI changed that, because the same company may fetch pages for several purposes. OpenAI’s documentation lists three crawlers, each controlled separately:

• GPTBot collects content that may be used to train OpenAI’s foundation models. Blocking it signals that a site’s content should not be used for that training.
• OAI-SearchBot supports search in ChatGPT. Sites that block it will not appear in ChatGPT search answers, though they can still show up as navigational links.
• ChatGPT-User visits a page when a person asks ChatGPT a question. Because a user triggers the visit, OpenAI says robots.txt rules may not apply.

So a publisher can stay visible in AI search while opting out of training. That split is the practical heart of the topic.

Google offers a variation. Its documentation describes Google-Extended as a standalone control token, not a crawler. Publishers use it to decide whether content Google crawls may be used to train future Gemini models or to ground answers. Google says it does not affect inclusion in Google Search and is not a ranking signal. The ordinary Googlebot token still governs Search.

Where the file came from

Martijn Koster defined the method in 1994. It ran for decades as an informal convention that search engines chose to honor. In 2022 it became RFC 9309, a Proposed Standard on the Internet Standards Track, written by Koster, Gary Illyes, Henner Zeller and Lizzi Sassman.

The standard added precision. The most specific (longest) matching path wins. If no group matches a crawler, the * group applies, and if none applies, no rules apply. If the file returns a 500-series server error, crawlers must assume everything is disallowed. Crawlers should not use a cached copy for more than 24 hours unless the file is unreachable.

The limits that publishers keep running into

The file has three weaknesses for the AI era.

• It is voluntary. A crawler that ignores it faces no technical barrier.
• It is blunt. It names crawlers and paths, not purposes. A company that runs one crawler for several uses forces a single choice. OpenAI notes that if both of its crawlers are allowed, it may use one crawl for both purposes.
• It cannot undo the past. A new rule affects future fetches. It says nothing about content already collected.

A further limit is structural. The file only works when the party collecting content treats it as a request to respect. When the collection happens through contract terms rather than a crawler, there is nothing to block.

Why it surfaced in the Meta case

On 30 October 2026 the UK Competition and Markets Authority opened an investigation into Meta’s new commercial terms for publishers and advertisers. Our news report covers the case. According to the regulator, the terms could allow Meta to collect content from business and publisher websites and use it in its products, including AI features, without those businesses choosing whether or how it is used. The CMA says no decision has been made on whether Meta broke the law.

Meta says, according to coverage of the complaints, that the terms apply only to its advertising and business products. Publishers’ groups argue the terms leave no opt-out. The dispute shows why the technical opt-out matters. Where a site can say no in robots.txt, a publisher keeps a lever. Where access is bundled into terms that a business must accept to use a service, the lever may not exist.

Toward preferences that name the use

The Internet Engineering Task Force (IETF), which maintains the web’s standards, chartered an AI Preferences (AIPREF) working group in January 2025. The aim is a shared vocabulary for stating how content may be used, such as for training, and a way to attach it to content through robots.txt and HTTP headers. In April 2025 the group adopted a proposal from the Open Future foundation as a starting point, with Paul Keller and Martin Thomson of Mozilla as editors.

The idea is to target uses instead of individual crawlers. The work is unfinished. The latest draft this article reviewed, version 07, says it does not reflect consensus of the working group, and participants have called attaching preferences to robots.txt an open problem. Google says it intends to support the AIPREF training preference signal in early 2027.

What to watch

Three things will show whether the opt-out holds up. First, whether AIPREF finishes and major AI companies honor it. Second, whether crawlers that fetch pages on a user’s behalf, like ChatGPT-User, are treated as exempt. Third, whether regulators such as the CMA treat a missing or ineffective opt-out as a competition problem. Any preference signal is only a request, so enforcement will decide how much it matters.