Training vs search vs assistants
Blocking a training crawler (GPTBot, CCBot) keeps your content out of model training. Blocking a search crawler (OAI-SearchBot, PerplexityBot) can remove you from AI answers and citations. Decide each separately.
Technical SEO Tool
AI Bot Access Checker
Enter a domain to see which AI crawlers it allows or blocks in robots.txt, from
training bots like GPTBot and ClaudeBot to answer engines like PerplexityBot and OAI-SearchBot.
Check a site
We fetch the live robots.txt and test each AI crawler against it.
View the robots.txt we tested
What AI bot access actually controls
Google-Extended is not Googlebot
Google-Extended only controls Gemini and Vertex AI training. It does not affect Google Search crawling or ranking, which is governed by Googlebot. Blocking one does not block the other.
robots.txt is a request, not a wall
Well-behaved crawlers honour robots.txt, but it does not technically prevent access. For hard blocking, use server-side rules or WAF controls. This tool reflects what compliant bots will do.
Want to change a verdict?
Edit your rules and re-test in the robots.txt Tester. Add a group like User-agent: GPTBot then Disallow: / to block a specific AI crawler, or Allow: / to let it in.
Not sure which AI bots to allow?
The right AI crawl policy depends on your goals, visibility in AI search versus protecting content. StudioHawk helps you decide and implement it correctly.