← All checks
robots.txt syntax validator for AI crawlers
Group A2 pointsrubric r1
We parse every non-comment line of robots.txt and count the lines that are not a known directive such as User-agent, Allow, Disallow, Sitemap, Crawl-delay or Content-Signal.
Why it matters
Parsers handle bad lines differently. One may skip a malformed line, another may discard the whole group, so a rule you wrote for GPTBot can silently stop applying. A file with more than a fifth of its lines invalid scores zero, because an agent cannot trust any of its rules.
How to fix it
- Use one directive per line in the form
Name: value, with a colon. - Open each group with
User-agent:before anyAllow:orDisallow:line. - Remove HTML and pasted prose; comments must start with
#. - Delete invented directives such as
Noai:orAllow-all; useContent-Signal:if you need an AI policy. - Keep the file plain text, with no markup from a CMS editor.
How to verify
Re-scan this page and read the evidence line, which reports the total number of lines and how many were invalid.