Web Bot Auth: How websites verify that an AI agent is who it claims to be

Web Bot Auth is a way for an automated visitor, such as a search crawler or an AI agent, to prove who it is when it loads a web page. The bot signs each request with a secret key. The website checks the signature against a public key that the bot’s operator has published.

The goal is to replace guesswork with proof. Today a site mostly has to take a bot’s word for what it is.

The problem: A bot’s name tag is easy to fake

Picture a nightclub where guests announce their own names and the doorman has no list. Anyone can say they are on it. That is roughly how websites treat bots today.

Every web request carries a self-declared label called the User-Agent header. Agents often claim to be an ordinary Chrome browser. Sites can also allow-list the network addresses (IP addresses) a bot uses, but those are brittle. Cloudflare engineer Thibault Meunier wrote in May 2025 that addresses can be shared by many users or services and change as infrastructure changes.

This matters more now that agents act for people. In the story we covered, ChatGPT Wallet lets OpenAI’s agents complete purchases. OpenAI’s playbook for merchants tells them to configure a CDN or firewall to recognize signed agent requests through Web Bot Auth. A CDN (content delivery network) is the service that sits in front of many websites and filters their traffic.

Signing a request instead of announcing a name

The bot’s operator first creates a pair of keys. The private key stays secret. The public key is published on the operator’s own domain, so anyone can fetch it. Cloudflare’s documentation says it supports only one signature algorithm, Ed25519.

Each request then carries three extra headers:

• Signature: the signature itself.
• Signature-Input: what was signed, plus a key ID, a created time and an expires time, and the tag web-bot-auth.
• Signature-Agent: a link to the place where the operator’s public keys can be found.

The website, or a service in front of it, looks up the public key and checks the signature. If it matches, the request really came from whoever holds the private key. A forger who copies the header text cannot produce a valid signature without that key.

Where the keys are published

Cloudflare’s documentation describes a key directory hosted at /.well-known/http-message-signatures-directory on the operator’s domain, served over HTTPS. The directory is itself signed, with one signature per listed key. That stops a third party from copying the directory and registering it on the operator’s behalf.

The key ID is a thumbprint, a short fingerprint computed from the public key in JSON Web Key (JWK) format. The verifier uses it to pick the right key from the directory.

The timestamps do a second job. A copied request is only useful until its expires time. Cloudflare recommends short windows and notes that “a minute is often sufficient.” Its documentation also says it does not track nonces, which are one-time values, so the short expiry is the main protection against replay.

Built on an existing standard, now at the IETF

Web Bot Auth did not invent a new signing format. It builds on HTTP Message Signatures, published as RFC 9421, a standard for signing parts of an HTTP request. Meunier’s post gives the reasons for the choice: it is a published standard, it has reference implementations, and it works at the HTTP layer, which keeps adoption simple.

In that May 2025 post, Cloudflare said it was proposing the mechanisms and building tools. It also said it planned to integrate them into its AI Audit and Bot Management products. It added that OpenAI signs requests from its Operator agent using RFC 9421, so site owners can verify them. It also published a Chrome extension demo, a debug server and a plugin for the Caddy web server.

The work has since moved to the IETF, the body that standardizes internet protocols. It now has a working group named webbotauth. Its charter covers search crawlers, web archivers, link checkers, AI training crawlers and AI agents acting for end users. Its milestones list standards-track documents for April 2026 and an operational best-practices document for August 2026. We did not check where those drafts stand today.

What it does not solve

The charter is explicit about scope. It excludes authenticating end users, APIs, non-cryptographic methods, bot intent vocabularies and bot reputation tracking. Web Bot Auth answers one narrow question: who sent this request? It does not say whether the sender is trustworthy, or what it plans to do.

That gap is visible in commerce. OpenAI’s own playbook, as we reported, warns that risk systems may still flag legitimate agent purchases, leading to declines or cancellations. A verified identity helps fraud tools tell an agent from an attacker. It does not guarantee approval.

Adoption is also uneven. Cloudflare’s documentation describes only its own implementation and does not list other vendors that support the method. It also notes technical limits, such as unsupported signature components and ASCII-only values. Some sites may simply refuse agents even when they verify. CNBC reported that United Airlines said it will block agents from its systems and that Amazon prevents agents from buying on its platform.

Operations add a practical burden. Operators must protect private keys, rotate them and keep the directory current. The IETF charter lists key lifecycle management among the topics for its operational document.

What to watch next

• Final IETF documents. The specifications, once finished, would give vendors a shared target beyond Cloudflare’s implementation.
• Who signs. Wider signing by crawlers, browsers and agent makers would make it more useful for sites to require it.
• How sites react. Verified identity lets a site allow, charge or block a specific bot instead of every bot. Which of those choices dominates is still open.
• Links to payments. Agent checkout relies on both signed identity and single-use payment credentials. How the two fit together will shape how often legitimate agent purchases succeed.