# Robots.txt: what is this file for?

> The robots.txt file is a text file placed at the root of a site that tells crawlers (Googlebot, Bingbot…) which areas they're allowed to explore and which they aren't.

- Source canonique : [https://allaux.fr/en/glossaire/robots-txt](https://allaux.fr/en/glossaire/robots-txt)
- Langue : EN
- Dernière mise à jour : 2026-09-30

## Why this file matters for an online store

An e-commerce site often generates thousands of technical URLs: cart pages, checkout pages, customer account pages, or faceted filter results (colour, size, price). If crawlers explore all these URLs indiscriminately, they waste part of the "crawl budget" allocated to the site and can index duplicate content generated by the filters.

The robots.txt file directs crawlers to the pages that actually matter: categories, product pages, editorial content, while excluding the cart, the back office, or the parameters generated by filters.

## A concrete example

The file is accessible at https://example.com/robots.txt and contains directives such as Disallow: /cart or Disallow: /admin, usually followed by a Sitemap: line pointing to the XML sitemap, which helps search engines discover the important pages.

## Blocking isn't deindexing

> A Disallow directive prevents an URL from being crawled, but not necessarily from being indexed if external links point to it. To remove a page from search results, a noindex tag is needed, not just robots.txt.

## FAQ

### Is the robots.txt file enough on its own to stop a page being indexed?

No. It prevents crawling, but a blocked page can still appear in results if it's referenced elsewhere. For guaranteed deindexing, the meta noindex tag must be used.

### Where should the robots.txt file be placed?

At the root of the domain name, never in a subfolder, otherwise crawlers won't find it at the standard address they check automatically.

### Should the back office be blocked in robots.txt?

It's good practice to keep it out of the index, but robots.txt is a public file anyone can view: it never replaces genuine protection through authentication.