← All checks
robots.txt checker: is it reachable and served as text?
Group A2 pointsrubric r1
We request /robots.txt on your domain and look for a 2xx response served as text/plain that contains robots directives.
Why it matters
AI crawlers such as GPTBot and ClaudeBot fetch robots.txt before any page. If it returns an HTML error page or a redirect to your homepage, the crawler cannot tell what is allowed and many treat the site as off limits. A file with robots directives but served as text/html earns only partial credit because strict parsers ignore it.
How to fix it
- Create
/robots.txtat the root of the domain, not in a subfolder. - Make it return HTTP 200 directly, with no redirect to a login or home page.
- Serve it with the header
Content-Type: text/plain; charset=utf-8. - Start with a catch-all group:
User-agent: *followed byAllow: /. - On static hosts, check that your rewrite rules do not send
/robots.txttoindex.html.
How to verify
Run curl -sI https://example.com/robots.txt and confirm 200 and content-type: text/plain.