Web Bot Auth: how AI agents prove who they are, and why your firewall should care.
A user-agent string is a claim anyone can make. A signature is proof. The agents your customers use are starting to sign their requests, and that changes how you should handle bot traffic.
Bot protection was built for a world where "automated" meant "unwanted". Now some automated visitors are your customers' assistants. You need a way to tell them apart.
The problem with user agents
Any client can send any user-agent string, so a header saying "ChatGPT" proves nothing. Some agents don't even try: ChatGPT agent browses with an ordinary Chrome user agent, and Perplexity Comet looks like Chrome too. Blocking by user agent stops honest bots and lets liars through.
How Web Bot Auth works
- The agent's operator creates a key pair and publishes the public key as a key directory at
/.well-known/http-message-signatures-directoryon its own domain. - Each request carries signature headers, including
Signature-Agent, which points to where the keys live. - The site, or its CDN, fetches the key and verifies the signature. Signatures are bound to the target site, so they can't be replayed elsewhere.
It builds on HTTP Message Signatures (RFC 9421). The IETF's webbotauth working group has adopted a draft, "HTTP Message Signatures for automated traffic", authored by engineers from Cloudflare and Google and intended for the standards track.
Who signs today
| Agent | Identity | Source |
|---|---|---|
| ChatGPT agent | Signature-Agent: chatgpt.com, with an ordinary Chrome user agent | Observed and documented by Simon Willison |
| Google-Agent | Experimenting with Web Bot Auth using https://agent.bot.goog | Google crawling documentation |
| Perplexity Comet | No reliable signature or token found | Security vendor analyses |
What to do
- Stop judging agents by user agent alone. Use verified-bot and signed-agent features where your CDN offers them; Cloudflare lists signed agents separately from verified bots.
- Decide which agents you want. Agents acting for your buyers are usually welcome on pricing, booking and checkout. Training scrapers are a separate decision.
- Log the signature headers. If your server logs
Signature-Agent, you can see signed agent traffic that user-agent matching misses. Our log analyzer covers the user-agent side. - Remember robots.txt isn't the gate here. Google says user-triggered fetchers such as Google-Agent generally ignore robots.txt; your firewall decides.
Most sites don't need to verify signatures themselves. A CDN or WAF that does it for you is the practical route.
Questions people ask
What is Web Bot Auth?
A protocol that lets bots and AI agents cryptographically sign HTTP requests using keys published at a well-known URL, so websites can verify who sent them instead of trusting the user-agent string.
Does ChatGPT agent sign its requests?
Yes. ChatGPT agent uses an ordinary Chrome user agent but signs requests with HTTP Message Signatures and a Signature-Agent header pointing to chatgpt.com.
Is Web Bot Auth a standard?
It is an IETF working group draft as of 2026, intended for the standards track, and already implemented by Cloudflare.
Want this mapped onto your product?
See how our AAO work runs, or send your store link and a founder will reply personally.
