robots.txt Explained for Modern Websites
A single line in robots.txt can stop a crawler from requesting thousands of URLs—or accidentally prevent it from reaching pages and resources you wanted discovered.
That makes robots.txt deceptively important. It looks like a small configuration file, but it sits near the beginning of how many search engines, AI crawlers, monitoring systems, and other automated clients interact with a website.
The most important distinction is this: robots.txt controls crawling access. It is not an indexing control, authentication system, or security boundary. Google explicitly documents that a URL blocked from crawling can still appear in search results if Google discovers that URL elsewhere.
What Is robots.txt?
robots.txt is a plain-text file that tells compliant automated crawlers which URL paths they are allowed or disallowed from requesting.
The Robots Exclusion Protocol was standardized as RFC 9309 in September 2022, although the mechanism itself dates back to the early web. The standard defines how crawlers should locate, parse, cache, and apply robots.txt rules.
For HTTPS websites, the file normally lives here:
https://example.com/robots.txt
It must be placed at the root of the relevant host. A file at:
https://example.com/blog/robots.txt
does not control crawling for example.com. Under RFC 9309, the standard location is /robots.txt, using the lowercase filename.
What robots.txt Actually Controls
Think of robots.txt as instructions placed at the entrance of a website:
“Crawler X may enter these paths, but should not enter those paths.”
A basic file could look like this:
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Sitemap: https://example.com/sitemap.xml
User-agent: * targets crawlers that match the general group. The Disallow rules ask them not to request URLs beginning with those paths.
The Sitemap line tells supporting crawlers where the XML sitemap is located. Google, Bing, and other major search engines support sitemap declarations in robots.txt, although Sitemap is not one of the core RFC 9309 protocol rules.
INTERNAL LINK: XML Sitemaps Explained
Crawling Is Not Indexing
This is the most common robots.txt misunderstanding.
Consider:
User-agent: Googlebot
Disallow: /private-offer/
Googlebot should not crawl content under that path. But if another website links to:
https://example.com/private-offer/
Google may still know that the URL exists. Google documents that a blocked URL may appear in search results without the page content or normal snippet.
If your actual requirement is:
“This page must not appear in search results,”
use an indexing control such as:
<meta name="robots" content="noindex">
or an HTTP header:
X-Robots-Tag: noindex
But there is an important dependency: the crawler must be allowed to fetch the page before it can see the noindex directive. Blocking the same URL in robots.txt can therefore prevent Google from discovering its noindex instruction.
INTERNAL LINK: noindex vs robots.txt
robots.txt Is Not Security
Never use robots.txt to protect passwords, internal documents, customer information, staging systems, administrative pages, or confidential files.
The file is public. Anyone can visit:
https://example.com/robots.txt
and read the paths you have listed.
RFC 9309 explicitly states that robots.txt is not a form of access authorization and should not replace real application-layer security. Authentication, authorization, network controls, or other appropriate security mechanisms should protect sensitive resources.
A rule such as:
Disallow: /secret-backups/
may actually advertise that /secret-backups/ exists.
How Rules Are Matched
A robots.txt file is organized into crawler groups.
For example:
User-agent: Googlebot
Disallow: /internal/
Allow: /internal/public/
User-agent: *
Disallow: /tmp/
The first group targets Googlebot. The second provides rules for other crawlers that match the wildcard group.
With Google, when multiple rules match a URL, the more specific path normally wins. If conflicting rules have equal specificity, Google uses the less restrictive rule.
For example:
User-agent: *
Disallow: /
Allow: /blog/
The site is generally blocked, while URLs beginning with /blog/ are allowed.
Google also supports * for pattern matching and $ for matching the end of a URL:
User-agent: Googlebot
Disallow: /*.pdf$
This rule blocks matching URLs that end in .pdf.
Crawler implementations can differ, especially around non-standard extensions, so avoid assuming every crawler supports every Google-specific behavior.
Modern JavaScript Websites
Modern websites frequently depend on JavaScript, CSS, API requests, and other resources to produce visible content.
If you block resources required to render a page, crawlers that render JavaScript may receive an incomplete version of it. Google documents that if robots.txt prevents Googlebot from fetching a URL, it skips requesting that resource; Google Search cannot render JavaScript from blocked pages or files.
For that reason, a rule like:
Disallow: /assets/
deserves careful review if /assets/ contains JavaScript or CSS required for public pages.
Blocking unnecessary backend paths can be useful. Blocking resources necessary to understand the page can create crawling or rendering problems.
INTERNAL LINK: JavaScript SEO Explained
robots.txt and Crawl Efficiency
robots.txt is especially useful when a website can generate enormous numbers of crawlable URLs.
Common examples include:
- faceted navigation
- internal search results
- calendar URLs
- filter combinations
- sorting parameters
- session-based URLs
- dynamically generated URL spaces
An e-commerce system might generate URLs such as:
/products?color=black&size=m&sort=price
/products?color=black&size=m&sort=newest
/products?color=black&size=l&sort=price
Thousands or millions of variations can consume crawler and server resources without providing useful search pages.
Google specifically recommends controlling unnecessary faceted-navigation crawling when those filtered URLs do not need to be indexed.
This matters much more on large or highly dynamic websites than on a simple site containing a few dozen pages.
robots.txt and AI Crawlers
robots.txt is no longer relevant only to traditional search engines.
AI companies also operate crawlers, and different crawler identities may represent different purposes.
For example, OpenAI documents OAI-SearchBot for content that may be discovered and surfaced through ChatGPT search experiences. OpenAI says publishers who want their content available for summaries and snippets should avoid blocking OAI-SearchBot.
Anthropic similarly documents separate crawler identities for different activities and states that its bots honor robots.txt directives.
This creates an important modern configuration question: which automated systems do you actually want to access your content?
A website may choose different policies for search discovery, AI search retrieval, model-development crawling, advertising systems, or other automated clients.
INTERNAL LINK: AI Crawlers and robots.txt
A Practical robots.txt Example
A relatively simple public website might use:
User-agent: *
Disallow: /admin/
Disallow: /internal-search/
Disallow: /api/private/
Sitemap: https://example.com/sitemap.xml
This configuration keeps ordinary crawlers away from areas that provide little crawling value while leaving public content accessible.
Do not copy this blindly. Your configuration should reflect your actual architecture, crawler requirements, and indexing strategy.
HTTP Errors Matter
Crawler behavior also depends on what happens when /robots.txt itself is requested.
RFC 9309 distinguishes between an unavailable robots.txt file and one that cannot be reached because of server or network failures.
For example, a 4xx response can be treated as the file being unavailable, allowing crawling. By contrast, when robots.txt is unreachable because of a server-side or network problem, the standard says crawlers must initially assume complete disallow. Crawlers may also cache robots.txt; RFC 9309 says cached versions generally should not be used for more than 24 hours unless the file is unreachable.
That means a broken robots.txt deployment can have very different consequences from simply having no file.
RFC 9309 also requires crawler implementations to support parsing at least 500 KiB of robots.txt content. Google currently applies a 500 KiB robots.txt size limit and ignores content beyond that point.
A normal robots.txt file should rarely be anywhere near that large.
How to Diagnose Problems
If important pages stop being crawled or rendered, check robots.txt early in the investigation.
A useful workflow is:
Symptom: a page or resource is not being crawled.
Measurement: request /robots.txt directly and inspect the rule matching the affected crawler.
Possible cause: an overly broad Disallow, wildcard pattern, crawler-specific group, or blocked rendering resource.
Confirmation: test the affected URL using crawler-specific diagnostic tools. For Google, Search Console's robots.txt reporting and URL Inspection tools can help confirm access.
Fix: narrow or remove the incorrect rule.
Re-measure: confirm that both the URL and necessary page resources are crawlable.
Also check your CDN, WAF, firewall, authentication layer, and bot-management system. A robots.txt rule may allow a crawler while another infrastructure layer still returns 403 Forbidden. OpenAI explicitly documents this as a possible cause of crawler-access failures.
Common Mistakes
The most damaging errors usually come from treating robots.txt as something it is not.
Using this:
User-agent: *
Disallow: /
on production blocks compliant general crawlers from the entire site.
Other common mistakes include blocking pages that contain noindex, blocking JavaScript required for rendering, publishing private directory names, assuming crawl-delay is universally supported, or copying rules from another website without understanding its architecture. Google, for example, does not support the crawl-delay robots.txt field.
robots.txt should be intentionally small and understandable.
SeoNest Recommendation
Treat robots.txt as part of your crawling architecture, not as a miscellaneous SEO file.
For most websites, start permissively and block only URL spaces that crawlers genuinely do not need to request. Keep sensitive information behind real access controls, use noindex when the objective is preventing supported search engines from indexing an accessible page, and periodically review crawler-specific policies as search and AI systems evolve.
Most importantly, test the final behavior rather than assuming the configuration means what it appears to mean.
FAQ
Do I need a robots.txt file?
Not necessarily. If crawlers may access your entire public website, having no robots.txt file can be acceptable. Google states that an empty file or no robots.txt file effectively allows crawling by default.
Does Disallow remove a page from Google?
No. Disallow controls crawling. A blocked URL may still be discovered and potentially appear in search results. Use an appropriate noindex mechanism if indexing is what you need to prevent.
Can robots.txt protect private pages?
No. robots.txt is publicly visible and is not an authorization mechanism. Use authentication and proper access controls.
Should CSS and JavaScript be blocked?
Usually not when those resources are necessary for crawlers to render and understand public pages. Google cannot render JavaScript from resources that robots.txt prevents it from fetching.
Can I control AI crawlers separately?
Yes, when the crawler operator publishes distinct user-agent names and supports robots.txt controls. OpenAI and Anthropic, for example, document crawler-specific controls.
Final Takeaway
robots.txt answers a narrow but important question: which URLs should compliant crawlers be allowed to request?
It does not decide whether a URL is confidential, whether a page must disappear from search results, or whether an unknown bot can technically access your server.
Once crawling, indexing, rendering, security, and crawler identity are treated as separate concerns, robots.txt becomes much easier to configure correctly—and much harder to misuse.
Sources
- IETF / RFC Editor — RFC 9309: Robots Exclusion Protocol. September 2022. RFC 9309
- Google Search Central — Introduction to robots.txt. Google documentation. Introduction to robots.txt
- Google Crawling Infrastructure — How Google Interprets the robots.txt Specification. Google documentation. Google robots.txt specification
- Google Crawling Infrastructure — Create and Submit a robots.txt File. Google documentation. Create a robots.txt file
- Google Search Central — Block Search Indexing with noindex. Last updated December 10, 2025. Block indexing with noindex
- Google Search Central — Understand JavaScript SEO Basics. Google documentation. JavaScript SEO basics
- Google Crawling Infrastructure — Managing Crawling of Faceted Navigation URLs. Google documentation. Faceted navigation crawling guidance
- OpenAI — Publishers and Developers FAQ. OpenAI Help Center. OpenAI publisher crawler guidance
- Anthropic — Does Anthropic Crawl Data from the Web, and How Can Site Owners Block the Crawler? Anthropic Help Center. Anthropic crawler guidance
