Free SEO tool

Robots.txt Checker

We fetch your robots.txt and read it the way crawlers do, following RFC 9309. See whether Googlebot may crawl the page you entered, whether the whole site is blocked, which AI crawlers your rules allow, and which sitemaps the file lists.

Free. Checks the address you enter in about 20 seconds.

What this tool checks

  • The file and its status

    We request /robots.txt from the origin of the address you enter (the same scheme, host and port) and show the HTTP status and the file as served.

  • Groups matched like a crawler

    Rules are grouped by User-agent. A crawler follows the group that names it and uses the * group only when no group names it; it doesn’t mix the two.

  • Longest match wins

    Within a group, the Allow or Disallow rule with the longest matching path decides, and Allow wins a tie. * matches any run of characters and $ marks the end of the URL.

  • Your page and Googlebot

    Whether Googlebot may crawl the exact URL you entered under the rules that apply to it.

  • Whole-site blocks

    Whether every crawler is told to stay out of the entire site.

  • AI crawlers

    Whether your rules block or allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent.

  • Sitemap lines

    Every Sitemap: line in the file, so you can confirm crawlers are pointed at the right sitemap.

What it can’t check

  • Enforce anything. robots.txt is a request that well-behaved crawlers follow and others ignore. To keep content private, put it behind a login.
  • Tell you whether a page is indexed. A blocked URL can still appear in search if other pages link to it.
  • Show which crawlers actually visit your site or how often. Your server logs have that.
  • Read the files on other hosts, such as a subdomain or a CDN. Each host has its own robots.txt.
  • Cover every crawler. The list includes widely used AI crawlers, and new user-agent tokens appear regularly.

Why it matters

robots.txt is the first file most crawlers request, and one line can change how a whole site is crawled. “Disallow: /” under “User-agent: *”, left over from development, stops search engines crawling anything. A rule meant for one folder, such as “Disallow: /p”, also blocks /products and /pricing, because rules match the start of the path.

The file now does a second job: telling AI companies which content they may collect. The tokens differ by purpose. OpenAI documents GPTBot for collecting training data and OAI-SearchBot for finding pages to show in ChatGPT search. Google-Extended controls whether Google may use your content for its Gemini models, without changing how Google Search crawls. Blocking one token doesn’t block the others, so check each against what you intend.

The status code matters as much as the content. A missing file (404) is harmless: crawlers take it as permission to crawl everything. A server error is different. Google’s documentation says that when robots.txt returns a 5xx error, it stops crawling the site for the first 12 hours, then uses the last copy it could read for up to 30 days.

How to read the results

HTTP 200
The file was found, and the rules shown are the ones crawlers apply.
HTTP 404 or another 4xx
There is no robots.txt, so crawlers may crawl everything. That is fine for many sites, as long as nothing needs to be kept out.
HTTP 5xx
The server failed to return the file. Google pauses crawling while this lasts, so fix the server error first.
Googlebot: allowed or blocked
Whether Googlebot may fetch the URL you entered, under its own group if the file has one, otherwise under the * group.
Whole site blocked
Every crawler is told not to crawl any path. Expected on a staging site; almost always a mistake on a live one.
AI crawlers: blocked or allowed
What your rules say for each token. Allowed includes having no rule for that crawler at all.
Sitemaps listed
The sitemap URLs from the file. Each should be a full URL that returns a working XML sitemap.

How to fix the most common problems

  • The whole site is blocked

    Remove “Disallow: /” from the “User-agent: *” group, or leave the Disallow line empty to allow everything. If your platform generates the file, check its search visibility settings and any robots.txt template; on Shopify that is robots.txt.liquid in the theme code.

  • A rule blocks more than intended

    Rules match the start of a path. End folder rules with a slash (Disallow: /private/) and use $ to match an exact ending (Disallow: /*.pdf$). Then check the URLs that matter to you again.

  • Googlebot ignores your * rules

    When the file has a “User-agent: Googlebot” group, Googlebot follows only that group. Copy every rule you want Googlebot to keep into its own group.

  • Opting out of AI training while staying in search

    Add a group for each training crawler, for example “User-agent: GPTBot” followed by “Disallow: /”, and do the same for ClaudeBot, CCBot, Google-Extended and Applebot-Extended. Leave Googlebot and Bingbot alone. Decide separately about OAI-SearchBot, Claude-SearchBot and PerplexityBot, which help your pages appear in those services’ search answers.

  • Using robots.txt to remove pages from Google

    Disallow stops crawling, not indexing. To keep a page out of search, allow crawling and add <meta name="robots" content="noindex">, or return 404 or 410 if the page is gone. Search Console’s Removals tool hides a URL temporarily in urgent cases.

Questions about this tool

Does robots.txt stop AI companies from using my content?

Only for crawlers that choose to follow it, and only from the next time they crawl. It doesn’t remove content that was collected before you added the rule.

Will blocking Google-Extended remove my site from Google Search?

No. Google says Google-Extended doesn’t affect inclusion or ranking in Google Search. It only controls whether Google may use your content for its Gemini models; Googlebot keeps crawling as before.

Is a missing robots.txt a problem?

No. If /robots.txt returns 404, crawlers assume they may crawl everything. Add one when you need to keep crawlers out of areas such as internal search results, or to point them to your sitemap.

Can I hide private pages with robots.txt?

No. The file is public, so listing private paths in it advertises them. Protect private pages with a login, and use noindex for public pages that should stay out of search.

How quickly does Google notice a change?

Google generally caches robots.txt for up to 24 hours, so a change can take about a day to apply. The robots.txt report in Search Console shows the version Google last fetched.

Sources

Related tools

The SEO Fix Report ($69, one-time) starts from the full free scan and adds more pages, the change to make for every finding, a PDF and re-tests to verify your fixes.

About the SEO Fix Report