What it means
robots.txt is a plain text file at the root of a site that tells crawlers which paths they may request. The group “User-agent: *” with “Disallow: /” forbids every crawler that follows the file from crawling anything.
It controls crawling, not indexing. Google can still list a blocked address it finds through links, usually without a description, but it can’t read the page’s content.
Why it matters
Blocking the whole site is usually a launch leftover: development and staging sites are often blocked, and the file goes live unchanged. The result is that search engines stop reading your pages and their listings go stale or vanish.
Google says robots.txt isn’t a way to keep a page out of its results. To hide a page from search, use noindex instead and let the page be crawled.
How to find it yourself
- Open yoursite.com/robots.txt in a browser and look for Disallow: / under User-agent: *.
- Google Search Console’s robots.txt report shows the file Google last fetched and any problems with it.
How to fix it
On any website
- Edit robots.txt at the root of your site and remove the Disallow: / line from the User-agent: * group, or replace it with the specific paths you want to keep crawlers out of.
- Add a Sitemap: line with the full address of your XML sitemap.
- If the site was blocked on purpose before launch, check for other launch settings too, such as noindex tags or a site password.
On Shopify
- Shopify generates robots.txt. If it blocks everything, check whether the store is still password protected, and whether the theme has a robots.txt.liquid template with a custom rule.
On WordPress and WooCommerce
- WordPress serves a virtual robots.txt unless a real file exists. Check for a robots.txt file in the site root and for robots.txt settings in your SEO plugin.
- Untick “Discourage search engines from indexing this site” under Settings › Reading if it’s on.
How to check the fix
- Reload robots.txt and confirm the rule is gone.
- Run the robots.txt checker to see what Googlebot and other crawlers may crawl now.
How FixReports finds it
The SEO checker and the robots.txt checker fetch robots.txt and flag a site-wide Disallow: / for all crawlers. The robots.txt checker also tests the rules for Googlebot, Bingbot and AI crawlers one by one.