What is the difference between noindex and robots.txt disallow, and when should I use each?

3 min readJune 22, 2026

  1. The two directives solve different problems, and that is the root of most confusion. A robots.txt disallow controls crawling, it tells search engines which URLs they may fetch. Per Google's own documentation, robots.txt exists mainly to avoid overloading your site with requests, and it is explicitly not a mechanism for keeping a page out of Google. A noindex rule controls indexing, it tells search engines to drop the page from results entirely. So the question is never just disallow or noindex, it is what outcome you actually want.

  2. Use noindex when you want a page that users can reach but that should never show up in search results, such as thank-you pages, internal search result pages, gated download confirmations, or thin filter combinations. Noindex is set either as a meta robots tag in the page head or as an X-Robots-Tag HTTP response header. Once Googlebot crawls the page and reads the rule, the page is dropped from results regardless of how many other sites link to it. That reliability is why noindex, not disallow, is the correct tool for keeping pages out of search.

  3. Use a robots.txt disallow when you want to stop crawlers from fetching a URL at all, usually to protect crawl budget or keep bots out of sections that produce no search value, like admin paths, faceted parameter URLs, staging directories, or large media you do not want fetched. But be clear about the limit, a disallowed URL can still get indexed if other pages link to it. Google may show the bare URL with a snippet such as a description for this result is not available because of this site's robots.txt. Disallow hides the content, not necessarily the listing.

  4. The single most damaging mistake, and one we see constantly during technical audits, is applying both a disallow and a noindex to the same URL. They conflict. For noindex to work, Google must crawl the page to see the tag. If robots.txt blocks the crawl, Googlebot never reads the noindex, and the page can linger in the index indefinitely. If you want a page gone from search, you must let Google crawl it, which means do not disallow it. Note also that Google does not support a noindex directive written inside robots.txt itself.

  5. This conflict surfaces in Google Search Console as two distinct reports. Indexed, though blocked by robots.txt means Google indexed a URL it could not crawl, often because of inbound links or a sitemap entry. Excluded by noindex tag means the rule was seen and honored, which is the healthy state. To fix the blocked-but-indexed case when you genuinely want the page out, remove the disallow, add or confirm the noindex, then request reindexing. Removing the internal links that point at the URL, or adding rel=nofollow as a hint, helps Google stop rediscovering it.

  6. The X-Robots-Tag HTTP header is the piece most teams overlook. A meta robots tag only works inside HTML, so it cannot govern PDFs, images, video, or other non-HTML files. The X-Robots-Tag carries the same noindex and nofollow rules in the response header and applies to any file type. On Apache you set it for all PDFs with a Files block in .htaccess, on NGINX with an add_header inside a location match. The same crawl requirement applies, if robots.txt blocks the file, the crawler never sees the header, so the rule does nothing.

  7. One last boundary worth stating plainly, neither directive is security. Robots.txt is a public file that effectively advertises the paths you do not want crawled, and a noindex still serves the full content to anyone, human or bot, who requests the URL. For anything genuinely private, such as customer data, internal tools, or pre-launch environments, use authentication or password protection. Treat noindex and disallow as visibility controls for search engines, never as access control.

  8. Getting this right across a real site means auditing every templated URL pattern, the sitemap, the robots.txt rules, and the meta and header directives together, because they interact. That cross-cutting cleanup is exactly the kind of work our website-in-30-days and stabilization engagements handle, we map which URLs should be crawled, which should be indexed, and which should be locked down, then implement the directives so they reinforce rather than fight each other.

Ready to ship?

Tell us where you are stuck. We will review it and send a focused recommendation within one business day.

Written by

Kostadin Stamboliev

Founder and Imaginator

Founder and Imaginator at Pineido. Practitioner across IT consulting, app and web development, technical SEO, and AI / automation. Ships production code in 30-day sprints with fixed scope and clear ownership.

LinkedIn