How to control crawling, indexing and snippets

Crawling and indexing are different

Use robots.txt to control what crawlers fetch, noindex to control what appears in search results, a canonical tag to say which duplicate URL should represent a group, and snippet rules to limit the text shown. Most mistakes come from using one of these to do another's job.

Crawling is a bot fetching a URL. Indexing is a search engine storing what it found and making the page eligible to appear in results. A page can be crawled and not indexed and, less obviously, a URL can be indexed without being crawled.

What robots.txt controls

The robots.txt file at the root of your host tells compliant crawlers which paths not to fetch. Its format is standardized in RFC 9309. Use it to keep crawlers out of areas with no search value, such as internal search results, endless filter combinations and admin paths.

User-agent: *
Disallow: /search
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Blocking a URL does not remove it from the index. If other pages link to it, Google can still index the URL without its content, usually with no description. robots.txt is not a security control either: the file is public and obeying it is voluntary. AI crawlers such as GPTBot, ClaudeBot and PerplexityBot document that they follow robots.txt, though fetches made in real time on a user's behalf may be handled differently.

What noindex controls

A noindex rule tells search engines not to show a page in results. Put it in a meta robots tag in the HTML, or send it as an X-Robots-Tag HTTP header, which also works for PDFs and other non-HTML files.

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The page must stay crawlable for this to work. A crawler only sees the noindex when it fetches the page, so if robots.txt blocks the URL, the rule is never read and the page can stay indexed. Removal also waits for the next crawl. For something urgent, the Removals tool in Search Console hides a URL from Google results for about six months while the noindex takes effect.

Controlling snippets

nosnippet

A robots rule that stops Google from showing a text snippet or video preview for the page.

max-snippet:[number]

A robots rule that limits the text snippet to that many characters.

data-nosnippet

An HTML attribute for span, div and section elements that keeps that part of the page out of snippets.

<meta name="robots" content="max-snippet:160">
<p>Member price: <span data-nosnippet>log in to see</span></p>

Google says these preview controls also limit how your content is shown in its AI features, such as AI Overviews. Use them with care, because a result with little or no snippet gives searchers less to go on.

Which to use when

Keep a page out of search results

noindex, with the page left crawlable.

Several URLs show the same content

A canonical tag, or a 301 redirect if the duplicate URL does not need to exist.

Stop crawlers wasting requests on endless URL variations

A Disallow rule in robots.txt.

A page has been removed for good

Return a 404 or 410 status.

Limit the text shown in results and AI features

nosnippet, max-snippet or data-nosnippet.

Keep private content private

Password protection or a login, not robots.txt or noindex.

Never block a page in robots.txt that you want noindexed. If an indexed URL has to go, add noindex, wait until it has dropped out of results, and only then block crawling if you still need to.