Faceted navigation SEO: block unneeded filters in robots.txt

When Googlebot spends most of its requests on filter and sort URLs, block those URLs in robots.txt unless you need some of them in search results. Google’s documentation on faceted navigation lists robots.txt and URL fragments as the ways to prevent crawling, and calls rel="canonical" and rel="nofollow" “generally less effective in the long term”.

Decide first whether any filter page belongs in search

1. List the parameters that create the waste. Group the filter requests in your server logs or crawl stats by parameter: color=, size=, sort= and so on. Every parameter that can change multiplies the number of possible URLs. A typical faceted URL looks like https://example.com/items.shtm?products=fish&color=radioactive_green&size=tiny. Crawlers cannot tell whether a new-looking URL is useful without fetching it, so they request a very large number of filter URLs before concluding the URLs are useless. Those requests are then not spent on discovering new, useful URLs, the problem described in the guide to crawl budget on large sites.
2. Sort the filter combinations into two groups. One group holds combinations that people search for and that deserve to rank, for example a category plus one attribute with real demand. The other holds everything else: sort orders, stacked attributes, price sliders. Combinations in the first group are a programmatic SEO question, and each one has to clear a quality bar before it should be eligible to rank.
3. Pick one of Google’s two paths for each group. If you do not need the filter URLs potentially indexed, prevent crawling of them. If you do, follow the best practices further down. Google adds that crawling faceted URLs “tends to cost sites large amounts of computing resources” because of the number of URLs and the work needed to render them.

Do not reach for noindex at this stage to fix the crawl problem. A noindex rule only works on a page that is not blocked by robots.txt and that the crawler can access, so the page still has to be fetched.

Block the filter URLs you don’t need indexed

4. Write a disallow rule per filter parameter. Leave the individual item pages crawlable, plus one listing page that shows all products without filters. Changing robots.txt takes write access to the root of the site host, and the canonical, link and 404 steps further down take access to the templates or the application code. This is Google’s own example:

   user-agent: Googlebot
   disallow: /*?*products=
   disallow: /*?*color=
   disallow: /*?*size=
   allow: /*?products=all$

   The three disallow lines stop crawling of URLs that carry those parameters, and the allow line keeps the unfiltered listing open. Google argues that there is often no good reason to allow crawling of filtered items, since it consumes server resources for no or negligible benefit.

5. Do not use robots.txt to remove pages that are already indexed. A URL disallowed in robots.txt can still be indexed if other places on the web link to it. The address and the anchor text of those links can then appear in Google Search results, without the page content. If indexed filter URLs have to leave the index, they need a noindex rule that Googlebot can read, which means they cannot be disallowed at the same time. Google also warns against using robots.txt for canonicalization for the same reason.
6. On a new build, put the filters in URL fragments instead. Google Search generally does not support fragments in crawling and indexing, so a filter mechanism based on # has no impact on crawling, positive or negative. The example URL from the first section becomes https://example.com/items.shtm#products=fish&color=radioactive_green&size=tiny. The trade-off is that no filtered state can rank on its own. Keep page numbers out of fragments, though: Google’s pagination rules require a real URL for every page in a series.

Canonical and nofollow reduce crawling more slowly, and nofollow must be on every link

7. Point canonicals at a version with fewer filters. If robots.txt and fragments are ruled out, a canonical tag on each filter URL can, in Google’s words, “over time, decrease the crawl volume of non-canonical versions of those URLs”. Google’s example points ?products=fish&color=radioactive_green&size=tiny to ?products=fish. Google treats rel="canonical" as a strong signal for which URL should become canonical, but it is not a crawl block, and the reduction comes gradually.
8. Add nofollow to every link that points at a filter URL, or to none. Google’s condition is strict: every anchor pointing to a specific URL must carry rel="nofollow" for the attribute to be effective. One plain link to the same URL, from a breadcrumb, a “popular filters” block or a template you forgot, undoes the rest.

Make the filter URLs you keep crawlable cheap to crawl

For the combinations you want potentially crawled and indexed, Google expects more server load and possibly slower discovery of new URLs. These steps keep the number of URLs down.

9. Separate parameters with &. Commas, semicolons and square brackets are hard for crawlers to recognize as separators, because they usually are not used as separators.
10. Fix the order of filters in the path. If filters live in the path, as in /products/fish/green/tiny, the logical order must always be the same, and no filter may appear twice. Otherwise the same result set gets several addresses.
11. Return a 404 when a filter combination has no results. If there are no green fish in stock, users and crawlers should get a “not found” error with the 404 status code. The same applies to duplicate filters, nonsensical combinations and nonexistent pagination URLs, such as page 40 of a filtered list that has three pages. Serve the 404 at the URL where it was requested, and do not redirect to a shared “not found” page. A single-page app may not be able to do this, and Google then refers to its best practices for single-page apps.

Filter URLs that stay crawlable but get no links from the rest of the site also tend to become orphaned pages, so give the combinations you keep a place in the internal linking.

Search Console no longer has a URL Parameters tool

Google announced on March 28, 2022 that the URL Parameters tool in Search Console would be deprecated a month later. Only about 1 percent of the parameter configurations in the tool were useful for crawling, according to the announcement, and Google’s crawlers now learn how to handle parameters on their own. For more control, Google points to robots.txt rules. The controls sit on your own site, in robots.txt, markup, URL structure and status codes, and none of the steps above requires anything from Google.

Not every search engine supports the same robots.txt rules, and crawlers can interpret the syntax differently, so check other engines against their own documentation before you rely on the same rules there.

12. Measure the effect in the same logs you started with. Compare requests per filter parameter before and after the change. Google describes the effect of canonicals as something that comes over time, so give that route longer than a robots.txt rule before you judge it.

Last fact-checked: 13 November 2026