Free tool · robots.txt
Check your robots.txt.
One wrong line in robots.txt can hide a whole site from Google. Enter your address and we'll fetch the file, apply the same matching rules Google uses, and flag anything that blocks important pages, scripts or styles.
What it checks
- Whether Googlebot can reach your homepage and common sections
- Rules that block CSS, JavaScript or images Google needs to render pages
- A missing or relative Sitemap: line
- Which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) are allowed
- HTML served instead of text, oversize files and unreadable lines
Blocking AI crawlers is a choice, not an error. We show it so you can decide on purpose.
More free tools
Keep checking
llms.txt generator
Build an llms.txt file from your sitemap in seconds.
Open toolSchema markup generator
JSON-LD for Organization, LocalBusiness, Article, FAQ and Product.
Open toolSitemap checker
Validate your XML sitemap and count its URLs.
Open toolMeta tags & headings checker
Title, description, H1, canonical, Open Graph and noindex.
Open tool