Robots.txt for SEO: Complete Guide

Robots.txt is a four-line file that can quietly cost you traffic or save your crawl budget. This guide covers the syntax that actually matters, the blocks that hurt rankings most often, what crawl budget really means, and how to verify every change before it ships.

5 min read7 sectionsJune 22, 2026

Written by

Kostadin Stamboliev

Founder and Imaginator

Founder and Imaginator at Pineido. Practitioner across IT consulting, app and web development, technical SEO, and AI / automation. Ships production code in 30-day sprints with fixed scope and clear ownership.

LinkedIn

What robots.txt does (and what it cannot do)

Robots.txt lives at the root of each domain, for example https://example.com/robots.txt. It tells compliant crawlers which URLs they are allowed to request. Google states this plainly: the file is meant to avoid overloading your site with requests, and it is not a mechanism for keeping a page out of Search.

That single sentence resolves most robots.txt confusion. Crawling is a bot fetching a URL. Indexing is Google storing that URL so it can rank. Robots.txt only governs crawling. A page you disallow can still appear in results, usually as a bare URL with no snippet, if other pages link to it. To keep a page out of Search you need a noindex meta tag, an X-Robots-Tag header, password protection, or removal, never a disallow line.

One more limit: the file is advisory. Googlebot and other reputable crawlers obey it, but a hostile scraper can ignore it entirely. Robots.txt is a traffic-management tool, not a security control. Anything that must stay private belongs behind authentication.

The syntax that actually matters

A robots.txt file is groups of rules. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules. A minimal example:

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://example.com/sitemap.xml

Field names are case-insensitive (User-agent equals user-agent), but path values are case-sensitive: /Admin and /admin are different. Paths are relative to the root and must start with a slash. Comments start with #.

Two wildcards are supported by Google, Bing, and other major engines: * matches any run of characters, and $ anchors the end of a URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf. Crawl-delay is widely listed in old guides but Google ignores it; Bing and Yandex still read it. Keep the file UTF-8 and under 500KB, which is Google's hard limit; anything beyond that is ignored.

Precedence: order does not decide, length does

The most common syntax mistake is assuming rules are read top to bottom. They are not. When several rules match a URL, the rule with the longest path wins, and on an exact tie Allow beats Disallow. This is codified in RFC 9309, the formal Robots Exclusion Protocol standard.

Example. For the URL /home/search/shirts with these rules:

Disallow: /home
Allow: *search*
Disallow: *shirts

the URL is crawlable, because the matching Allow path is longer than the matching Disallow paths. Reordering the lines changes nothing.

There is a second trap with user agents. Google merges all groups that match a specific bot into one, but it never combines a named group with the global User-agent: * group. So the moment you add a User-agent: Googlebot block, Googlebot stops reading your User-agent: * rules. If you target a specific bot, that group must repeat every rule you still want it to follow.

Blocks that quietly hurt rankings

Most robots.txt damage comes from over-blocking, not under-blocking. The worst offender is disallowing CSS and JavaScript. Google renders pages to understand them, and its own guidance is direct: if robots.txt blocks the JS, CSS, and image files your pages use, it directly harms how well Google renders and indexes your content and can result in lower rankings. The old crawl-budget myth of blocking assets was retired years ago.

The nuance from Google's Martin Splitt: blocking a script is only a real problem if content the user sees depends on it. A robotted analytics tag or chat widget is harmless. A robotted bundle that injects your main copy means Google sees an empty page.

The second quiet failure is using robots.txt to deindex. Disallow a URL and Google can never crawl it to discover a noindex tag you add later, so the page stays indexed. If you want a page gone, leave it crawlable and serve noindex. A blanket Disallow: / left over from a staging build is the catastrophic version of this, and it deindexes entire sites after launch.

Get operations insights weekly

Join service-business operators who get one actionable insight on ops, automation, and growth every week.

Checking that you're human. You can send in a moment.

No spam. Unsubscribe anytime.

You’re subscribed! Look for our next issue in your inbox.

Crawl budget: what it is and what robots.txt can do

Crawl budget is how many URLs Googlebot will fetch from your site in a given window. For sites under a few thousand URLs it rarely matters; Google crawls them fully. It becomes real on large catalogs, marketplaces, and any site that generates URLs combinatorially.

The classic waste is faceted navigation and on-site search. Filters like ?color=red&size=l&sort=price multiply into near-infinite URLs with little unique value, and internal search results pages can spawn endlessly. Google's Gary Illyes has repeatedly warned to block action URLs (add-to-cart, login, checkout, filter combinations) because Googlebot will otherwise crawl them indefinitely, even non-existent combinations.

One fact correction many guides get wrong: blocking pages does not hand the freed budget to your important pages. Google only reallocates if your server was already hitting its serving limit. Robots.txt stops waste; it does not manufacture crawl capacity. The real levers for more crawling are faster, healthier servers and higher-quality, genuinely unique content.

A practical starting template

Most service-business and content sites want something close to this, adjusted to real paths:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /*?*sort=
Disallow: /*?*filter=
Disallow: /search
Allow: /
Sitemap: https://example.com/sitemap.xml

Notes. List every XML sitemap with an absolute URL; you can have several lines. Do not block /wp-content, /assets, or any directory holding CSS, JS, or images. Be precise with parameter rules, because a sloppy Disallow: /*? can knock out clean URLs that happen to carry a tracking parameter. Keep the file simple; complex stacks of wildcard rules are where accidental over-blocking hides. Each subdomain needs its own robots.txt, and so does each protocol, so http and https are treated as separate files.

Testing with GSC and curl before you ship

Verify two things on every change: that the file Google fetches is the one you intended, and that real URLs resolve the way you expect.

Start with curl to see exactly what crawlers receive, with no caching or CDN surprises: curl -A "Googlebot" https://example.com/robots.txt. Confirm the status is 200, the content type is text/plain, and the body matches your source. A robots.txt that returns 500 makes Google treat the whole site as disallowed until it recovers, so server health matters here.

In Google Search Console, the old robots.txt Tester is gone. Use the robots.txt report under Settings to confirm Google's last-fetched version and any parse warnings. To check a specific URL, paste it into the URL Inspection tool and read the crawl section; it tells you if the URL is blocked. To find existing damage, open the Pages report and filter by Blocked by robots.txt. Remember GSC only tests Googlebot, so for Bing or AI crawlers validate separately. This testing discipline, applied before launches and migrations, is exactly the kind of safeguard a fixed-scope technical SEO sprint builds into your release process so a stray disallow never reaches production.

Ready to ship?

Tell us where you are stuck. We will review it and send a focused recommendation within one business day.

Frequently asked questions

Can I use robots.txt to keep a page out of Google?

No. Robots.txt controls crawling, not indexing. A disallowed page can still be indexed and shown as a URL if other pages link to it, and blocking it prevents Google from ever seeing a noindex tag. To remove a page from Search, leave it crawlable and use a noindex meta tag or X-Robots-Tag header, password-protect it, or delete it.

Is it bad to block CSS and JavaScript in robots.txt?

Yes, in most cases. Google renders pages to understand them and says blocking the CSS, JS, and image files your pages use directly harms rendering and indexing and can lower rankings. The only safe exception is scripts that do not affect the content users see, like a chat widget. Never block bundles that load your visible copy.

Do robots.txt rules apply in the order they are written?

No. When multiple rules match a URL, the rule with the longest matching path wins, and on a tie Allow beats Disallow. This follows RFC 9309. Reordering lines does not change behavior, so resolve conflicts by adjusting path specificity, not position.

Does blocking pages in robots.txt free up crawl budget?

Only sometimes. Blocking low-value URLs stops waste, but Google reallocates the freed budget to other pages only if your server was already hitting its serving limit. For most sites, faster servers and higher-quality unique content do more for crawl rate than blocking alone.

How do I test robots.txt changes in 2026?

Use curl with a Googlebot user agent to confirm the file returns 200 and the exact content you expect. In Google Search Console, check the robots.txt report under Settings for the fetched version and warnings, use the URL Inspection tool to test specific URLs, and filter the Pages report by Blocked by robots.txt to catch existing issues. The old standalone Tester is deprecated.

What should I actually block in robots.txt?

Block low-value, high-volume URLs that create infinite crawl space: faceted filter combinations, internal search results, sort and pagination parameters, and action pages like cart, checkout, login, and account. Always leave CSS, JS, images, and any page you want indexed crawlable.