Hey Meta

AI crawler checker

See which AI crawlers your robots.txt lets in: the ones that find you for ChatGPT, Claude and Perplexity answers, the ones collecting training data, and the ones fetching pages on request. Free, no account.

Or see an example: heymeta.com

heymeta.com lets every AI crawler in. Check yours.

The crawlers we check

AI companies run separate crawlers for search, training and on-request fetches, and robots.txt can treat each one differently.

AI search

Find and cite pages in ChatGPT, Claude and Perplexity answers. Blocking these keeps you out of AI search, so they count toward the AI readiness score.

  • OAI-SearchBot OpenAI
  • Claude-SearchBot Anthropic
  • PerplexityBot Perplexity

Model training

Collect pages to train AI models. Blocking them is your choice and doesn't remove you from AI answers.

  • GPTBot OpenAI
  • ClaudeBot Anthropic
  • CCBot Common Crawl
  • Google-Extended Google, Gemini training only, not Search
  • Applebot-Extended Apple
  • Meta-ExternalAgent Meta

Fetched for a user

Open a page when someone asks an AI about it. They act for a person, so they may ignore robots.txt.

  • ChatGPT-User OpenAI
  • Claude-User Anthropic
  • Perplexity-User Perplexity

Allow AI search, block training

A common setup: stay findable in AI answers while keeping your pages out of model training. Add it to your robots.txt, then run the check above again.

# Let AI search engines find and cite your pages
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Keep your pages out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Common questions

Does blocking GPTBot keep my site out of ChatGPT answers?

No. ChatGPT search finds pages with OAI-SearchBot. GPTBot only collects training data, so blocking it keeps your pages out of training, not out of answers. The same split applies to ClaudeBot and Claude-SearchBot.

Does Google-Extended affect Google Search?

No. Google-Extended controls whether Google may use your pages for its Gemini models. Google Search crawls with Googlebot, so blocking Google-Extended doesn't change whether or how you appear in Search.

My robots.txt allows a crawler. Why is it still blocked?

robots.txt is only one layer. CDNs and firewalls such as Cloudflare can block AI crawlers on their own, so check your CDN's bot settings too.

What if my robots.txt is missing or returns an error?

A missing robots.txt (a 4xx response) allows every crawler. A server error (5xx) tells crawlers to stay away from the whole site, so a broken robots.txt can block everything.