How do tiered and sharded posting lists impact ecommerce crawl rates?

Curious what other folks are seeing with Google's index server changes using tiered and sharded phrase posting lists. On a big ecommerce site (70k+ SKUs), crawl stats in Search Console got more volatile after Feb 2026. Anyone see patterns or have ideas how this setup might affect indexing for fresh product pages vs stale category pages?

Asked by alice.steeves

2 Answers

Yeah, tiered and sharded posting lists can really mess with crawl rates now, especially since Google started pushing more aggressive sharding by phrase group around Feb 2026. What I've seen: when your product URLs get stuck in deeper shards or lower tiers, Crawl Stats gets spikier and fresh pages sometimes take days longer to hit the index. If your category pages go stale and aren't getting enough internal links, they get dropped to colder tiers and see way fewer crawl requests, so any fresh product page hanging off those might not get discovered quickly either. I'd look at your internal linking strategy first. Make sure your fresh SKUs get cross-linked from high-value evergreen pages and recent blog posts, not just categories. Also, using dynamic sitemaps that highlight page recency can help nudge Googlebot into the right shard. Check crawl requests at the URL pattern level in GSC to see which shards or tiers are slowest right now. Definitely more volatile in 2026 than it was last year.

Answered by tanzeel

Yeah, with Google moving further into sharded and tiered posting lists, it can mess with crawl rates, especially for huge ecommerce sites like yours. I see a lot more spikes and dips in Crawl Stats after that Feb 2026 shift too. What seems to happen is fresh SKUs on product pages might actually get crawled faster since Google can target new stuff across shards, but old or low-performing category pages sometimes get pushed out to deeper tiers and don't see the crawler as often. I've found making sure your XML sitemaps only include live, legit product URLs helps signal to Google which pages should be in those higher-priority shards. Also, internal links and even minor updates to those stale categories can sometimes pull them back into more active crawl buckets. If you notice some sections go dead, try adding a few links from fresh content or tweak meta data just enough to get Googlebot's attention back.