← All checks
GPTBot robots.txt: is OpenAI allowed to crawl your site?
Group A4 pointsrubric r1
We read robots.txt, pick the group that applies to the GPTBot user agent (or the * group if none names it) and test whether the path / is disallowed.
Why it matters
GPTBot collects pages that OpenAI products can later cite. When it is disallowed, ChatGPT has no copy of your pages from that crawl, so your site is missing when a user asks about your product. A common cause is a blanket Disallow: / in the * group written for staging and shipped to production.
How to fix it
- Find any group that names
GPTBotand remove itsDisallow: /line. - If no group names GPTBot, check that the
User-agent: *group does not containDisallow: /. - To allow explicitly, add a group:
User-agent: GPTBotfollowed byAllow: /. - Remember a named group replaces the * group for that bot, so repeat any path rules you still need.
- Block only private paths, for example
Disallow: /admin/, instead of the whole site.
How to verify
Re-scan this page; the evidence reads "GPTBot not disallowed for /". With no robots.txt at all the check also passes.