Robots.txt
Last updated September 2026
Robots.txt is a plain text file placed at a site's root domain that tells search engine crawlers which sections of the site they are allowed or disallowed to crawl. It controls crawling, not indexation directly, and a misconfigured rule can accidentally block an entire site from search engines.
Robots.txt lives at a fixed location, the root of a domain, and follows the Robots Exclusion Protocol, a set of simple rules crawlers check before requesting any other page on the site. It tells well-behaved crawlers which paths they may or may not fetch, giving a site owner coarse control over where crawl budget gets spent.
01
What robots.txt controls, and what it does not
Robots.txt governs crawling: whether a crawler is permitted to request a given URL at all. It does not directly control indexation. A page disallowed in robots.txt can still be indexed, without its content ever being crawled, if enough external links point to it; Google can index a URL based on those signals alone, and it will typically show it in results with no description since the content was never read. Keeping a page fully out of the index reliably requires a noindex directive, which in turn requires the page to be crawlable so the crawler can see that directive.
- 01
Disallow
blocks a crawler from requesting a specified path
- 02
Allow: carves out an exception within a broader disallowed path
- 03
Sitemap: points crawlers to the site's XML sitemap location
- 04
User-agent
targets rules at a specific crawler or applies them to all crawlers with a wildcard
02
The most common, most damaging mistake
The single most frequent robots.txt error is a leftover disallow rule from a staging environment that migrates into production during a launch, silently blocking crawlers from the entire live site. Because robots.txt failures are often invisible in a normal walkthrough of the site (a human browsing the site sees nothing wrong; only a crawler checking the file is affected), this mistake can run for weeks before someone notices organic traffic has fallen off a cliff.
03
Checking it before it becomes a crisis
Google Search Console includes a robots.txt report that flags the live file's rules and any errors. Reviewing that file, along with a manual check of the raw text at yourdomain.com/robots.txt and a look at indexation status, should be a fixed step in any pre-launch and post-launch checklist for a site migration, redesign, or platform change.
