These are the two files most often confused with each other, and neither does what people expect. A robots.txt file tells crawlers which addresses they may request on your site. An XML sitemap tells them which addresses exist and which ones you consider important. Google is explicit on the point that causes the most damage: robots.txt is not a mechanism for keeping a page out of Google.
In plain English
Two notes left for a contractor working on your building.
One is taped to a door: do not go into this room. The other is a written job list: here is everything that needs doing, and when each item was last touched.
The door sign controls where the contractor walks. It doesn’t make the room invisible. Someone standing in the corridor can still tell other people the room exists, and describe it from the doorway, without ever going in.
The job list is a list. It doesn’t oblige anyone to do everything on it. It just means nothing important got missed because it was in a cupboard nobody opened.
Why this matters for your site
Both files are small, both live in a place most people never look, and both can remove a website from search results in a single line. This is the highest ratio of consequence to effort anywhere in technical SEO, the work of keeping a site reachable and readable.
The specific danger is that robots.txt is routinely used as a privacy tool by people who haven’t read what it does. Blocking a page and expecting it to disappear from search produces the opposite of what was wanted: a result with no description, which looks worse than the page did.
How it actually works
Robots.txt. It sits at the root of your site and manages crawler traffic. Google describes two legitimate uses: reducing load when a server would struggle with crawler requests, and avoiding the crawling of unimportant or near-identical pages.
The three limits. Google lists them plainly. The rules aren’t enforceable, so well-behaved crawlers obey them and others may not. Different crawlers interpret the syntax differently. And an address disallowed in robots.txt can still be indexed if other sites link to it, appearing in results without a description.
The trap. Google cautions that combining crawling rules with indexing rules can make them cancel each other out. A noindex rule is written inside the page, so a crawler blocked from fetching that page never reads it. Blocking and noindexing at once guarantees the noindex is ignored.
Resource files. You can block scripts, styles, and images with robots.txt, but Google warns against it when their absence makes the page harder to understand. A page whose stylesheet is blocked is a page Google sees differently from your visitors.
XML sitemaps. A sitemap lists pages, videos, and other files, along with information such as when each was last updated and which alternate language versions exist. Google’s position on whether you need one is unusually relaxed: if a site’s pages are properly linked, it can usually discover most of the site.
When one helps. Google names the cases: large sites, where guaranteeing every page has an inbound link is difficult. New sites with few external links, since crawlers arrive by following links from elsewhere. Sites with a lot of video or images, or that appear in Google News. And sites with archives of pages that are isolated or poorly linked to each other.
Size limits. A single sitemap file holds up to 50,000 addresses or 50 MB uncompressed, whichever comes first. Image and video sitemaps usually hit the size ceiling before the count. Beyond either, you use a sitemap index, which is a file listing other sitemaps.
Neither file grants indexing. A sitemap is a discovery aid, and Google’s own wording is blunt: submitting one is merely a hint, with no guarantee it will even be downloaded or used for crawling. Being listed in a sitemap is not a commitment by Google to store or show the page.
What this looks like in practice
A team reads that Googlebot wastes time on unimportant files and decides to help by blocking the folder holding their theme’s stylesheets, scripts, and images.
The pages still load perfectly for visitors. For Google, they now render as unstyled text with no images, because the resources needed to display them are behind a door marked do not enter.
The change was made to improve crawling efficiency and it degraded how every page on the site was understood. That’s the shape of most robots.txt damage: a well-intentioned optimization with no visible symptom.
What people usually get wrong
“We blocked it, so it’s hidden.” Google says a blocked address can still appear in results if other sites link to it. Robots.txt controls walking, not talking.
“Belt and suspenders: block it and noindex it.” These cancel out. Pick one, and for keeping a page out of search results, pick noindex with crawling allowed.
“A sitemap gets pages indexed.” It helps Google find them. What happens next is indexing, the separate process of analyzing a page and deciding whether to store it.
“Every site needs a sitemap.” Google says well-linked sites are usually discovered without one. It’s a fix for a discovery problem, not a default.
What you can do about it
Open yourdomain.com/robots.txt in a browser right now. It takes ten seconds and occasionally explains a year of confusion.
After every launch, redesign, or hosting change, check that no Disallow: / line survived from the staging environment. This single line is responsible for more sudden disappearances than any other cause.
Leave your styles, scripts, and images crawlable unless you have a specific reason and have checked what the page looks like without them.
Use noindex, not robots.txt, for pages you want out of search results, and make sure those pages are still crawlable so the rule can be read.
Submit your sitemap in Google Search Console and read the sitemap report afterwards. A sitemap listing addresses that redirect, error, or are blocked is telling Google your own list is unreliable.
Generate the sitemap automatically. A hand-maintained one is accurate for about a month.
Words you will hear
Disallow. The robots.txt instruction not to request a given path.
Noindex. A rule inside a page telling search engines not to store it in their index.
Sitemap index. A file listing several sitemaps, used once one file is not enough.
User-agent. The name a crawler identifies itself by, used to target robots.txt rules.
For robots.txt, sitemap, crawler and index, see How search works. For SERP, query and search console, see What is SEO?
