Google Is Crawling Your Shop Filter URLs Instead of Your Products

You have probably been told that the more pages of your site Google has indexed, the better. More pages, more chances to be found. It sounds obviously true, and for a plain brochure site it roughly is.

On an online shop it is usually wrong.

Open Search Console on a shop that sells sixty products and you will often find Google reporting several thousand known addresses. Most owners read that number as progress. It is not progress. Almost none of those addresses are products, and the products that matter are queued behind the ones that are not.

Every filter your shop page offers is a link. Size, colour, brand, price range, sort by price, sort by popularity. Tick two boxes and the address in your browser changes. That changed address is a brand new page as far as a crawler is concerned, and crawlers follow links. Nothing is broken. Nobody made a mistake. The shop is doing exactly what it was built to do, and the side effect is a problem that only shows up in the two places most owner-operators never look: the indexing report and the server load graph.

Where the extra addresses actually come from

Open your own shop page and apply a single filter. Then look at the address bar. On most WordPress shops you will see something appended after a question mark, along the lines of ?filter_size=44 or ?min_price=20&max_price=50. Tick a second box and it grows. Change the sort order and it grows again.

Each of those variations is a separate address that returns a page. Not a redirect, not an error. A full page, built on request.

The arithmetic gets away from you fast. Take three filters offering six options each, where a shopper can tick more than one box at a time. That is 64 possible states per filter, and 64 cubed is more than 250,000 combinations across the three. Add two sort orders and you are past half a million addresses. That is arithmetic rather than an estimate, and it is why a catalogue of sixty products can present a crawler with a surface larger than a national newspaper.

You did not write those pages. Your shop generates them one at a time, on demand, for anyone who asks. Including software.

Crawlers walk straight into it, and Google says so itself

This is not a theory from the SEO industry. It is documented by Google in its own guidance on faceted navigation, which is the technical name for filters like these.

Google's position is that "because the URLs created for the faceted navigation seem to be novel and crawlers can't determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs" ^[1]. That is the part people expect. The next sentence is the one that costs money: "if crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs."

New and useful is your new stock.

Consider a typical case, and late September is the season for it. A farm and workwear shop in Longford loads a winter boot range into the shop ahead of the weather it was bought for. The product pages deploy correctly. They are live and they are accurate.

For the next few weeks the only people who see them are the customers already on the site, because the crawler that should have picked them up is working through boots filtered by size, then by width, then by size and width and brand together, in an order nobody chose. The range sells eventually. It sells at the end of its season instead of the start, which is the difference between full price and a markdown.

That is the quiet version of this failure. Nothing goes down. No alert fires. The stock simply arrives in search results after the customers have already bought elsewhere.

The half of this that has nothing to do with rankings

There is a second cost, and it lands on your server rather than your rankings.

A page cache works by storing the finished HTML for an address and handing it straight to the next person who asks for that same address. Page caches generally key on the full URL, query string included, and that is correct behaviour: ?filter_size=44 and ?filter_size=45 are genuinely different pages, so they cannot share a stored copy. The consequence for a filtered shop is unavoidable. A crawler working through filter combinations is requesting addresses that nobody has ever requested before, over and over. Almost every one is a cache miss.

A miss is not free. It goes through to PHP, runs a product query against the database, assembles a page, and returns it to a robot that then decides the page was not worth having. Do that a few thousand times an hour and your shop is doing real work for no revenue at all. Your customer browsing on a phone is queueing behind it.

This is where the infrastructure under the shop decides how bad the day gets. On a stack with Nginx and FastCGI page caching in front of PHP-FPM, with Redis holding the object cache, the pages real customers ask for still come off the cache at full speed while the crawl churns away underneath. That is the practical value of a caching layer that absorbs this kind of traffic before it ever reaches PHP. On a cheap shared plan with a hard ceiling on concurrent PHP processes and no page cache worth the name, the same crawl is indistinguishable from a small traffic spike, and it arrives without warning.

Google's own documentation closes the loop between these two halves. The Search Console status "Discovered - currently not indexed" is defined as a page that "was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl" ^[4]. Read that on a shop. If your server is already labouring under a crawl of filter combinations, the crawl that gets postponed is not the junk. It is whatever Google found most recently, which on a busy week is your newest product.

What Search Console is telling you, and the switch that no longer exists

The Page Indexing report is where this becomes visible, and the wording matters.

"Crawled - currently not indexed" means "the page was crawled by Google but not indexed. It may or may not be indexed in the future" ^[4]. "Duplicate without user-selected canonical" means Google decided the page was a duplicate of another and picked the other one ^[4]. On a shop with open filters, both of those buckets fill up with filtered views, and both are Google telling you plainly that it fetched something and got nothing for the trip.

There used to be a control for this. Search Console shipped a URL Parameters tool in 2009, back when it was still called Webmaster Tools, and it let you tell Google which parameters to ignore. Google retired it in 2022, and was blunt about why: only about 1% of the parameter configurations site owners had specified were useful for crawling, and the crawlers had become better at working it out unaided ^[3].

So there is no longer a switch inside Search Console to flip. Whatever you do about this now happens on your own site, which is the honest position anyway.

One thing to verify before you change anything: how your filters are built. Google "generally doesn't support URL fragments in crawling and indexing" ^[1], which is exactly why its own guidance suggests filters implemented as fragments after a hash character rather than as query parameters. WooCommerce's current Product Filters block updates the product list without reloading the page by default, and reloads the full page only when the surrounding block is configured to do so ^[6]. Classic filter widgets and many third-party filter plugins do write the selection into the address bar. Click a filter, watch the address bar, and you will know within five seconds which kind you are running.

Five Steps to See What Google Is Crawling on Your Shop

• Click. Apply one filter on your own shop page and check whether the address bar changes. If it does not, you can stop here and go back to work.
• Count. Open the Page Indexing report in Search Console and compare the number of known addresses against the number of products you actually sell. A gap of ten or twenty times is the signal.
• Inspect. Look at the sample URLs listed under "Crawled - currently not indexed" and "Duplicate without user-selected canonical". If they are full of question marks and filter names, you have found it.
• Decide. Work out which filtered views are genuine customer demand. Nobody searches for boots in size 44 in brown sorted by price, but plenty of people search for safety boots, so that one may deserve a real category page of its own.
• Disallow. Block the rest in robots.txt using the pattern style Google gives in its own faceted navigation guidance, then verify in the Crawl Stats report over the following weeks that the volume has dropped.

What blocking cannot do

Three honest limitations, because a half-understood fix here does more damage than leaving it alone.

Blocking in robots.txt stops the crawling, not necessarily every appearance. Google states it directly: "a page that's disallowed in robots.txt can still be indexed if linked to from other sites" ^[5]. The traffic cost goes away. The listing might not.

Worse, and this is the one that catches people, a blocked address is an address Google cannot read. If you disallow your filter URLs in robots.txt and also add a canonical tag or a noindex instruction to those pages, Google will never see the tag, because it is no longer fetching the page to look. Pick one mechanism. Blocking and tagging the same URLs at the same time is a common own goal.

Third, and most important: most shops do not need to do any of this. Google's crawl budget guide is written for sites with more than 10,000 pages updating daily, or more than a million pages in total, and it opens by telling almost everyone else to leave: "if your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide" ^[2].

That is the test. Publish a product, check in a day or two, and if it is in Google, your crawl is healthy and this article is not about you. Worth separating from a neighbouring problem, too: this is not the duplicate content story, which is a different failure with a different fix, covered in our guide to where WordPress quietly creates duplicate pages.

And one genuine concession. If you sell twenty products and never want to think about crawl paths again, a closed hosted store builder removes this problem entirely, because it does not let you create these addresses in the first place. You give up filter flexibility, plugin choice and the ability to move the shop elsewhere later, which for a very small catalogue can be a fair trade.

It stops being a fair trade the moment the catalogue grows, or the day you want to leave. Full WordPress hands you the rope and expects you to tie the knot, and that is the cost of actually owning the thing. If you are building a shop from scratch rather than repairing one, the platform decision is where this gets settled: a Web60 site is built in under a minute and runs on managed infrastructure for sixty euro a year, with the caching already configured and the server side already handled.

For the wider picture of how these structural decisions affect load times as well as indexing, our complete WordPress performance guide covers the same territory from the speed side.

Conclusion

The number of indexed pages was never the score. It is a count of doors, and most shops with a filter sidebar have built several hundred thousand doors into rooms nobody wants to stand in.

The work here is small and mostly diagnostic. Click a filter, read the address bar, compare two numbers in Search Console, and you will know inside ten minutes whether your crawl is being spent on your products or on combinations of your products. If new stock is appearing in search the same week you publish it, leave the whole thing alone. If it is not, you now know where to look first.

Frequently Asked Questions

Does having more pages indexed help my SEO?

Not by itself. Google indexes pages it considers useful, and a shop page filtered by three attributes is a near-duplicate of the unfiltered version with fewer products on it. A high indexed-page count on a small catalogue is usually a sign that filter combinations are being crawled, not a sign of growing visibility. The number worth watching is how quickly a newly published product page gets crawled and indexed.

How do I know if Google is crawling my shop filters?

Open the Page Indexing report in Search Console and look at the sample addresses under "Crawled - currently not indexed" and "Duplicate without user-selected canonical". If those samples contain question marks with filter or sort parameters, your filters are being crawled. The Crawl Stats report, under Settings, shows the volume of requests Googlebot is making and how it trends over time.

Should I block filter URLs in robots.txt or add a noindex tag?

Choose one, never both. Blocking in robots.txt stops the requests reaching your server, which is what you want if the goal is to protect crawl time and server capacity. A noindex tag keeps pages out of the index but only works if Google can still fetch the page to read the tag, so it cannot be combined with a robots.txt block on the same addresses. For filter combinations that nobody searches for, blocking is usually the right call.

Will blocking filter URLs stop customers using the filters?

No. Robots.txt is a set of instructions for crawlers and has no effect on visitors. Your customers keep filtering exactly as before. The only thing that changes is that automated crawlers stop fetching the millions of possible combinations, which frees capacity for the pages that real people and search engines actually need.

Do shop filters slow down my website for customers?

They can, indirectly. Filtered addresses are usually cache misses, so each request is built from scratch by PHP and the database rather than served from a stored copy. A crawler grinding through combinations generates a steady stream of those expensive requests, and on hosting with tight limits on concurrent processes, genuine customers end up queueing behind them. Proper page and object caching keeps customer-facing pages fast while that happens underneath.

My shop is small. Is this worth worrying about?

Probably not, and that is the honest answer. Google's own crawl budget guidance says sites whose pages are crawled the same day they are published do not need to think about this at all. Verify that first. If your new products appear in search within a few days of going live, your crawl is healthy and the filters are doing no measurable harm.

Sources