Technical SEO Tool
Pull a live robots.txt, edit it in place, and check whether any URL is
crawlable for Googlebot and other bots. Results follow Google's exact matching rules,
so you see what Google will really do before the change goes live.
Fetch a live file or paste your own. Edit freely and re-test.
Pick a crawler and list the URLs (or paths) to check.
| URL | Crawler | Verdict | Matching rule |
|---|
Tip: click a matching rule to highlight it in your robots.txt.
A robots.txt file is a set of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A crawler obeys the single group whose user-agent is the most specific match for its name. A bot with its own group ignores the global * group entirely, the two are never merged.
Within the matching group, the rule with the longest path wins, not the first one listed. Order does not matter. When an Allow and a Disallow are the same length, the Allow takes priority because Google uses the least restrictive rule.
* matches any run of characters and $ anchors the end of the URL. So Disallow: /*.pdf$ blocks /file.pdf but not /file.pdf?x=1, because the query string means the URL no longer ends in .pdf. Rules are matched against the path plus query string, and paths are case-sensitive.
An empty Disallow: allows everything, while Disallow: / blocks the whole site. Blocking a URL in robots.txt does not remove it from Google, it only stops crawling, so use noindex for de-indexing. AdsBot and AdSense crawlers do not obey the * group.
A single stray Disallow can hide your best pages from Google. StudioHawk's technical SEO specialists hunt down crawl issues like this every day, before they cost you rankings.