← All checks

ClaudeBot robots.txt: is Anthropic allowed to crawl you?

We test the root path / against two user agents, ClaudeBot and anthropic-ai, using the group that applies to each in your robots.txt.

Why it matters

Anthropic uses these agents to collect pages that Claude can reference. If either is disallowed, Claude has no fresh copy of your pages and may rely on third-party summaries that are older or wrong. Sites often block only the legacy anthropic-ai name or only the newer ClaudeBot, so each is scored on its own: both allowed earns 3, one earns 1.

How to fix it

1. Search robots.txt for ClaudeBot and anthropic-ai groups and remove any Disallow: /.
2. Make sure the User-agent: * group does not disallow / if neither bot is named.
3. To allow both explicitly, put User-agent: ClaudeBot and User-agent: anthropic-ai above one Allow: / line.
4. Check CDN or WAF bot rules too, since robots.txt does not override a firewall block.

How to verify

Re-scan this page; the evidence reads "blocked: none" when both agents are allowed.

Check your site free

Scan a site

Related checks

• PerplexityBot allowed
• Google-Extended / Applebot-Extended not disallowed
• Bot Reality: fetch under GPTBot UA
• Rate limit
• sitemap.xml

All checks