We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

← All posts

Your SERP tracker is now a sample

Crawlmind Engineering··5 min read

A SERP tracker is a tool that sends automated queries to Google and records what comes back, and since mid-September 2026 Google has been blocking more of those queries, so the data many teams use for rankings and AI Overview visibility now has holes in it.

This matters for anyone who reports AI Overview presence, People Also Ask coverage or keyword positions from a third-party tool. Most of those numbers come from scraping. When the scraping degrades, the chart does not say "missing data." It often says "you dropped."

#What happened in September

On September 18, 2026, Search Engine Roundtable reported that Google appeared to be blocking AI scrapers, search scrapers and tracking tools more effectively from around September 13. Derek Perkins of Nozzle described an 80% drop in the data his company could retrieve from Google Search, and said DataForSEO was seeing similar trouble. The Sistrix status page said its data collection was "running at a reduced rate" after Google changed how results are delivered. Glenn Gabe observed that Ahrefs kept collecting normally, Semrush slowed down, and Sistrix fell behind in picking up ranking recoveries.

The timing was poor. Google was already in a period of heavy movement: Search Engine Roundtable logged ranking volatility on September 15 and September 23 to 24, plus a September 4 update that seemed to revert on September 13, followed by the September 2026 spam update. A tracker with a partial feed in the middle of real ranking changes produces charts where the two cannot be separated.

#This is the third step, not the first

The September blocking fits a pattern that has been building for almost two years.

In January 2025, Google launched SearchGuard, a system that sends a JavaScript challenge with search requests to check that a real browser is asking. Google described it in its December 2025 DMCA lawsuit against SerpApi, alleging that SerpApi sent "hundreds of millions" of automated queries a day and got around SearchGuard by misrepresenting the device, software or location of its requests. Google laid out its side in a post titled Why we're taking legal action against SerpApi's unlawful scraping.

In September 2025, Google removed the parameter that let tools load 100 results on one page. Search Engine Roundtable wrote at the time that most third-party tracking tools were broken. AccuRanker limited tracking to the top 20, and Semrush moved some refreshes from daily to every two days.

The policy side has not changed. Google's spam policies define machine-generated traffic as sending automated queries to Google, and name "scraping results for rank-checking purposes" as an example. Every rank tracker that does not have Google's express permission is operating against that rule, and Google is getting better at enforcing it.

#Why AI Overview data is hit hardest

Keyword positions have a first-party fallback in Search Console. AI Overview content mostly does not.

The Search Console AI performance report, available globally since September 1, 2026, shows impressions for your URLs in AI Overviews, AI Mode and AI Overviews in Discover, broken down by page, country, device and date. It does not include queries or clicks. It also says nothing about which competitors were cited next to you, what the answer said, or whether your brand was mentioned without a link.

That remaining detail is exactly what AEO teams buy third-party tools for, and it can only be collected by loading the results page and reading it. AI Overviews are also harder to capture than a list of blue links, because part of the answer can sit behind a "Show more" control that Google keeps testing changes to. A collector that gets challenged or throttled can return a page with the organic results intact and the AI Overview incomplete or absent. A failed capture recorded as "no AI Overview" or "not cited" looks identical to a real loss.

#How to read tracker data now

None of this means third-party data is useless. It means you should treat it as a sample with known failure modes and check it before you act on it.

Ask your vendor for collection health. Request the share of scheduled queries that returned a complete page, by day, for your project. If they cannot give you that, you cannot tell a real drop from a collection gap. Sistrix posting a public status note was the right behavior. Silent gaps are worse.

Separate "not found" from "not collected." A keyword that returned a page without your URL is a measurement. A keyword where the request failed is not. Reports that merge the two will show phantom losses whenever collection degrades.

Cross-check against Search Console before escalating. If a tracker shows a sharp loss of AI Overview citations, compare it with AI impressions for the same pages in the AI performance report over the same dates. If impressions are flat, the tracker probably lost coverage. If impressions fell too, the loss is likely real. Do the same for organic positions using the regular performance report.

Compare two tools during volatile weeks. In September, different vendors degraded by different amounts, as Gabe's comparison showed. When two independent collectors disagree sharply, the disagreement is information. Hold the decision until they converge or Search Console settles it.

Weight trends over single days. Track rolling averages across a stable keyword set rather than reacting to one refresh. A single bad crawl day should not trigger a content rewrite.

Keep non-Google engines in the mix. ChatGPT, Perplexity and Claude answers are collected through different channels and are not affected by Google's anti-scraping changes in the same way. If your Google AI data looks broken while other engines look stable, that pattern points to collection, not content.

#What to expect next

The direction is clear. Google is spending money on detection, it is willing to sue, and its policies already treat rank-checking scrapers as spam. Expect more periods like September, where some vendors keep collecting and others fall behind for days or weeks. The SerpApi case is still open, and SerpApi filed a motion to dismiss in February 2026 arguing that the DMCA protects copyrighted works rather than websites, and that it only reads publicly visible pages. Its outcome may decide how much of today's tracking market is legally safe to run.

Until then, report third-party Google data with a note on coverage, anchor every alarm to a first-party number, and treat a sudden cliff in an AI Overview chart as a question for your vendor before it becomes a question for your content team.

Related field notes

Share or discuss

Field notes in your inbox

New posts, no spam. Roughly monthly. Unsubscribe with one click.