A single robots.txt rule can stop crawlers from requesting an entire section of a website. Before publishing or changing this file, identify the pages and resources that search engines need to access, review every path carefully, and test the result on the production hostname. A small punctuation or directory mistake can have much wider effects than intended.
This guide is for website owners, editors, developers, and SEO practitioners who create or review crawl instructions. It explains how user-agent groups, path rules, wildcards, environments, and resource files affect crawler access. The SatoGifts Robots.txt Generator can help prepare a draft, but the final file still requires human review and testing.
A robots.txt file provides crawl instructions for automated user agents. It normally sits at the root of a hostname, such as https://www.example.com/robots.txt. Its rules apply only to that protocol, hostname, and port. A file on www.example.com does not automatically govern shop.example.com, and an HTTPS file does not necessarily control an HTTP version.
A basic group contains a User-agent line followed by Disallow and, where supported, Allow rules. The user-agent identifies the crawler addressed by the group. A value of * addresses crawlers that do not match a more specific applicable group.
Disallow identifies paths that should not be crawled. An empty Disallow value does not block the whole site; it means no path is disallowed by that line. By contrast, Disallow: / requests that the applicable crawler avoid every path on that host.
Robots.txt manages cooperative crawler access. It is not an access-control system, a deletion mechanism, or a reliable way to keep information confidential.
Rules are organized into groups. A blank line usually separates one group from another. Crawlers look for the group whose user-agent token most specifically matches their identity. Exact behavior can vary among crawlers, especially older or nonstandard bots, so do not assume every automated system will interpret complex files identically.
| Directive | Purpose | Important caution |
|---|---|---|
| User-agent: * | Provides rules for crawlers without a more specific matching group | It does not override a more specific applicable group |
| Disallow: /private/ | Requests that matching paths under that directory not be crawled | The path may cover more URLs than expected |
| Allow: /private/public-guide | Creates a narrower permitted path where the crawler supports Allow | Confirm how the target crawler resolves competing rules |
| Disallow: | Leaves crawling unrestricted by that directive | Do not confuse it with Disallow: / |
Path matching begins from the URL path after the hostname. Paths are generally case-sensitive, so /Products/ and /products/ may be different. Query strings can also affect matching. The most specific matching path rule commonly determines the outcome when Allow and Disallow rules conflict, but crawler documentation should be checked for important cases.
Suppose a store wants to prevent crawling of internal search results under /search/. The intended instruction is Disallow: /search/. During editing, it is shortened to Disallow: /. That one missing word changes the instruction from blocking one directory to blocking the complete site for the applicable user-agent group.
The practical response is to compare each rule with a list of representative URLs: the homepage, category pages, product pages, internal search results, account pages, and static resources. Do not approve a file merely because its syntax looks tidy.
A publisher wants to block URLs ending in a print parameter and adds a broad rule resembling Disallow: /*print. Because the wildcard can match varying text and the rule is not anchored to the end, it may also match paths or parameters containing “print” in unintended positions, such as a legitimate article about printing.
If the relevant crawlers support the end marker, a more focused pattern may use $ to indicate the end of the URL pattern. Even then, test examples containing other parameters, encoded characters, uppercase variants, and similar words. Simple directory rules are usually easier to audit than elaborate wildcard patterns.
Staging sites are often blocked with Disallow: /, but robots.txt alone does not make staging private. A stronger approach is authentication or network-level access restriction. This also protects the environment from crawlers that ignore robots instructions.
A common deployment error is copying the staging robots.txt file to production. Another is leaving staging crawlable after it becomes publicly reachable. Deployment checks should therefore compare the hostname and expected robots policy. Production should not inherit a blanket staging block, while staging should not depend on a public text file for protection.
Blocking directories containing CSS, JavaScript, images, or other rendering resources can prevent a crawler from seeing a page as users see it. This may make navigation, responsive layouts, lazy-loaded content, or page context harder to evaluate.
Do not block a resource directory merely because its files are not standalone landing pages. Test whether important pages require those resources. A Page Speed Checker may help identify page-loading concerns, but performance work and crawler access are separate decisions. The resource still needs to be accessible if it is necessary for rendering.
Automated validation can identify formatting issues, but it cannot fully understand business importance. For every rule, manually ask: which exact URLs match, which valuable pages share the same prefix, and which resources do those pages require?
Paste representative paths into a simple review table and classify each as intended to be allowed or disallowed. Then compare that expectation with a crawler-specific tester where available. Finally, open the live robots.txt file directly and check server logs over time for requests to critical pages and resources.
A Link Analyzer can help reveal internal paths that deserve review, while a Broken Links Finder can identify links that complicate URL auditing. An XML Sitemap Generator may help organize canonical URLs intended for discovery. Sitemap inclusion does not override a robots.txt block, so the two files should not send contradictory operational signals.
Robots.txt behavior is not perfectly uniform across all bots. Some crawlers do not support every extension, some may cache the file, and hostile bots may ignore it entirely. URL encoding, redirects, alternate hostnames, and parameter combinations can also make outcomes less obvious.
Use the Robots.txt Generator to prepare a clear draft, then review it against real URLs and the published site structure. If an important section is already blocked, correct the rule, confirm that the pages are internally linked and included in an appropriate sitemap, and allow time for compliant crawlers to retrieve the updated file. For sensitive or high-impact changes, involve the developer or server administrator responsible for deployment and access controls.
Not reliably. It can prevent compliant crawlers from fetching the page, but the URL may still be discovered through links. Use suitable page-level indexing controls when the crawler can access the page, or require authorization when content must remain private.
Only crawlers addressed by the applicable user-agent group and willing to follow robots.txt. It does not physically prevent requests.
Usually not when they are required to render important pages. Review each resource directory according to how the site works rather than whether the files are search landing pages.
A crawler may be using a cached copy of robots.txt or may not have revisited affected URLs yet. Confirm the live file first, then monitor requests and available crawler reports.
No. The file is publicly readable and can draw attention to those paths. Protect private areas through authentication, authorization, or network controls.
No. A sitemap suggests URLs for discovery; it does not grant permission to crawl a URL blocked by robots.txt.
Support and interpretation can vary, particularly among nonstandard crawlers. Keep patterns simple, consult relevant crawler documentation, and test important cases manually.
Last reviewed: July 28, 2026