Are AI crawlers visiting your site?
Your robots.txt says which AI crawlers are allowed. Your server logs say which ones actually came, which pages they read and when. Here’s how to find GPTBot, ClaudeBot, PerplexityBot and the rest, and how to tell the real ones from fakes.
First, check what robots.txt allows →What to look for
Every AI crawler sends its name in the user agent of each request. Each name tells you why it came.
| User agent contains | Company | What a visit means |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexing your pages so AI answers can find and cite them |
| Claude-SearchBot | Anthropic | Indexing your pages so AI answers can find and cite them |
| PerplexityBot | Perplexity | Indexing your pages so AI answers can find and cite them |
| GPTBot | OpenAI | Collecting pages for AI model training |
| ClaudeBot | Anthropic | Collecting pages for AI model training |
| CCBot | Common Crawl | Collecting pages for AI model training |
| Meta-ExternalAgent | Meta | Collecting pages for AI model training |
| ChatGPT-User | OpenAI | Someone asked an AI about your page just now |
| Claude-User | Anthropic | Someone asked an AI about your page just now |
| Perplexity-User | Perplexity | Someone asked an AI about your page just now |
Google-Extended and Applebot-Extended never appear in logs. They’re robots.txt settings for pages that Googlebot and Applebot fetch anyway.
Find them in your logs
On a server you run, the web server’s access log has every request. These commands work on
Linux and macOS. For Apache, use /var/log/apache2/access.log instead.
Count visits per crawler
grep -oE "OAI-SearchBot|Claude-SearchBot|PerplexityBot|GPTBot|ClaudeBot|CCBot|Meta-ExternalAgent|ChatGPT-User|Claude-User|Perplexity-User" /var/log/nginx/access.log | sort | uniq -c | sort -rn
See which pages a crawler read, most recent last
grep -i "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $4, $7, $9}' | tail -20List the IP addresses a crawler came from
grep -i "GPTBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rnInclude older, compressed logs
zgrep -oE "OAI-SearchBot|Claude-SearchBot|PerplexityBot|GPTBot|ClaudeBot|CCBot|Meta-ExternalAgent|ChatGPT-User|Claude-User|Perplexity-User" /var/log/nginx/access.log* | cut -d: -f2 | sort | uniq -c | sort -rn
Behind Cloudflare
Cloudflare sees every request before your server does, including the ones it blocks. Its AI Crawl Control page lists AI crawlers by name, how often they came and whether they were allowed. Check it before your server logs: a crawler Cloudflare blocks never reaches them.
On a hosted platform
Website builders and many hosts, such as Wix, Squarespace and Shopify, don’t give you raw access logs. Look for a logs or bot traffic page in your host’s dashboard, or put your domain behind a CDN that reports bots. Your analytics won’t show them either.
Is it really GPTBot?
Anyone can put “GPTBot” in a user agent, and scrapers often do. The real crawlers come from IP addresses their companies publish. If a request’s IP isn’t in the list, it isn’t that crawler.
Anthropic
Common Crawl
Get the addresses with the IP command above, then look them up in the company’s list.
Common questions
Why don’t AI crawlers show up in Google Analytics or Plausible?
Analytics tools count visitors by running JavaScript in the browser. Crawlers download the HTML and never run it, so they’re invisible to analytics. Only server logs, or your CDN, record them.
Why is there no Google-Extended in my logs?
Google-Extended isn’t a crawler. It’s a robots.txt token that tells Google whether Gemini may use pages that Googlebot already fetched. Applebot-Extended works the same way with Applebot. Look for Googlebot and Applebot instead.
How often do AI crawlers visit?
There’s no published schedule. Popular pages get fetched more often, new or small sites rarely. On-request fetchers like ChatGPT-User only appear when someone asks an AI about your page.
Did they fetch my updated page?
Look for the page’s address in the logs and check the date of the most recent visit from OAI-SearchBot, Claude-SearchBot or PerplexityBot. A visit after your update means the new version was read.
My robots.txt allows them, but they never visit. Why?
Check your CDN and firewall. Cloudflare, for example, can block AI crawlers with a single setting, before robots.txt is ever read. If your logs show them getting 403 errors, something in front of your site is blocking them.