AAO toolAI bot & agent log analyzer: who is reading your site for AI.
Paste or upload your access log. The analyzer finds every request from AI training crawlers, AI search bots, user-triggered fetchers such as ChatGPT-User and Google-Agent, and human visits referred by AI assistants. It shows which pages they read and where they were refused. Nothing leaves your browser.
Free · no signup · updated 13 October 2026
Your server or CDN access log
Nginx, Apache or any "combined" format log, or JSON lines from Cloudflare, Vercel and similar. Processed in your browser only.
Who visited
| Bot or agent | Type | Requests | Pages | Refused | Most requested |
|---|
Pages AI systems read most
| Path | AI requests | Search & user fetches | Training | Refused |
|---|
Visits referred by AI assistants
| Assistant | Visits | Top landing page |
|---|
Matches are by user agent, which any client can fake. Confirm important bots by IP range or reverse DNS, or by HTTP message signatures where the operator signs requests.
How do I see if AI agents visit my website?
Check your server or CDN access logs for AI user agents: GPTBot and ClaudeBot (training), OAI-SearchBot, Claude-SearchBot and PerplexityBot (AI search), and ChatGPT-User, Claude-User, Perplexity-User and Google-Agent (fetches made for a person). Browser agents such as ChatGPT agent use a normal Chrome user agent and can only be identified by their request signatures.
The method, in the open.
Each line is parsed as a standard "combined" access log (Nginx, Apache) or as a JSON log line (Cloudflare, Vercel and similar), reading the user agent, path, status code and referrer.
- AI bots are matched by the tokens their operators publish (the same list our AI crawler checker uses), plus Google-Agent, Google's user-triggered fetcher for agents.
- Refused means a 401, 403, 429 or 5xx response. A refused AI search bot or user fetcher is a page that can't be quoted or used on someone's behalf.
- AI referrals are human visits whose referrer or
utm_sourceis an AI assistant, such as chatgpt.com or perplexity.ai.
Static assets are counted like any other request. For a cleaner view, filter your log to page requests first.
What the numbers look like.
Using the sample log. Of 15 requests, 12 come from AI systems. OAI-SearchBot and ChatGPT-User read the pricing page and an article; GPTBot reads two articles for training; Google-Agent fetches the contact page for someone. But Claude-User gets 403 twice on the AAO page and PerplexityBot gets 429 on pricing, so 3 of 7 answer-side requests were refused. Two human visits arrived from ChatGPT and Perplexity. The fix: allow AI search and user fetchers in the firewall and rate limiter, not just in robots.txt.
What this is based on
- Google-Agent is a user-triggered fetcher for agents on Google infrastructure; user-triggered fetchers generally ignore robots.txt rules. Google crawling documentation
- ChatGPT agent sends a standard Chrome user agent and signs requests with HTTP Message Signatures, with Signature-Agent set to chatgpt.com. Simon Willison
- Cloudflare identifies signed agents cryptographically through Web Bot Auth, separately from verified crawlers. Cloudflare
- OpenAI documents GPTBot (training), OAI-SearchBot (search) and ChatGPT-User (user requests) as separate crawlers. OpenAI
What it does not tell you
- User agents can be faked. Confirm important bots by published IP ranges, reverse DNS or request signatures.
- Browser-based agents such as ChatGPT agent and Perplexity Comet look like ordinary Chrome and cannot be named from a user agent.
- If your CDN blocks a bot, the request may never reach your origin log. Analyse CDN logs where you can.
- Large logs are processed in your browser; very big files (over 20 MB) should be split or filtered first.
Questions people ask.
Which log formats does this read?
The common and combined formats used by Nginx and Apache, and JSON lines with fields such as ClientRequestUserAgent (Cloudflare) or userAgent. Other formats usually work if the user agent is in quotes.
Can it detect ChatGPT agent?
Not from the user agent. ChatGPT agent uses a normal Chrome user agent and signs its requests. Only a CDN or server that verifies HTTP message signatures can identify it reliably.
What is Google-Agent?
A Google user-triggered fetcher used by agents running on Google infrastructure. Google says user-triggered fetchers generally ignore robots.txt, so firewall rules decide whether it gets in.
What does "refused" mean?
The server answered 401, 403, 429 or 5xx. For AI search bots and user fetchers, that usually means a firewall, bot-protection or rate-limit rule is stopping them.
Is my log uploaded?
No. The file is read and analysed in your browser with JavaScript. Nothing is sent to our servers.
Keep checking.
AI crawler checker
Is GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot blocked on your site? Checks robots.txt and tests your CDN, bot by bot.
robots.txt AI policy generator
Pick a policy, such as "visible in AI search, no training", and get a commented robots.txt block for every AI bot. Merges with your existing file.
GA4 AI traffic regex generator
Track ChatGPT, Perplexity, Claude, Gemini and Copilot visits in GA4. Generates the regex, channel group steps, a Looker Studio formula and a GTM variable.
Let the right agents finish the job.
We test how ChatGPT agent, Gemini, Comet and Claude handle your key tasks, then fix access, pages and forms so they complete them with you.