Your ChatGPT tracker is a proxy
Crawlmind Engineering··5 min read
A ChatGPT citation tracker is a proxy: it measures what an API or a search index returns for your prompts, not what a person sees in the ChatGPT app, and a new study puts a number on how far apart those two are.
The paper is Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations, a preprint from researchers at Xi'an Jiaotong University and the National University of Singapore, submitted in October 2026. It has not been peer reviewed. It is still the most direct comparison we have seen between the consumer product and the two proxies most tools rely on.
We have a stake in this. Crawlmind's ChatGPT engine sends tracked prompts to OpenAI's Responses API with the built-in web search tool, the same kind of proxy the paper tests. This post is partly a note on how to read our own numbers.
#What the researchers measured
The team recorded the event stream that ChatGPT's web interface returns to a signed-in session. From it they pulled the search queries ChatGPT actually ran, the pages it retrieved, and the pages it finally cited. The prompts came from Search Arena, a public dataset of real user conversations with search-enabled models, split across factual lookups, synthesis, analysis, and recommendation questions, half in English and half in Chinese.
They ran each prompt repeatedly against ChatGPT Web, then ran the same prompts through two proxies in the same three-day window: Google organic results fetched through a scraping API, and OpenAI's Responses API web search tool. The main comparisons cover the Instant settings of two model generations, with a Thinking setting as a supplementary case.
#The headline result
Combining Google results with the Responses API's full candidate list, the proxies recovered 28.3% of cited URLs for one model generation and 26.2% for the next. The paper's abstract summarizes this as about one quarter of cited URLs.
Coverage improves as you zoom out. The same combined proxy recovered 50.3% and 48.7% of cited hostnames, and 50.7% and 45.1% of cited entities. Google alone did worst: at the URL level it recovered 9.4% and 6.4% of cited URLs.
The Responses API was the better proxy of the two. Its own citations matched ChatGPT Web more precisely than Google did, and that precision rose between the two generations, with hostname precision going from 27.7% to 40.5%. The proxy is getting closer to the product. It is not the product.
#Why the gap exists
The paper frames a ChatGPT answer as a pipeline: rewrite the question into search queries, retrieve pages, then choose which ones to cite. Each stage adds distance from a proxy.
Query rewriting is the first stage. ChatGPT never searched with the user's exact wording in the matched Instant settings; the exact-match rate was 0.0%. Language is another source of drift. For Chinese questions, 67.0% and 69.6% of runs issued English-only queries. A tracker that scrapes Google for the prompt text is checking a query ChatGPT did not run.
Retrieval is also unstable from run to run. Across 20 repeats of the same prompt, 40.8% and 44.5% of candidate URLs appeared in only one run, while only 9.4% and 7.7% showed up in at least 14 of the 20. The researchers found that a page retrieved often across repeats was more likely to be cited, and they are careful to call this an association rather than a lever.
Rank mattered less than many people assume. Candidates beyond retrieval position 15 were still cited 24.0% and 19.0% of the time. Publication recency showed no reliable effect in this data set.
#What this means for your dashboard
None of this makes tracking useless. It changes which numbers you should trust.
Read domains, not URLs. A proxy agrees with the real product about half the time on which sites get cited and about a quarter of the time on which pages. If a report says ChatGPT cited a specific blog post of yours, treat it as a sample. If it says ChatGPT keeps citing your domain across many prompts, that is a stronger signal.
Treat page-level wins as hypotheses. When a rewrite of one page appears to win a citation in the tracker, check whether the cited host was already showing up, and confirm by hand in the ChatGPT app before you roll the change out to a whole template.
Do not use Google rankings as a stand-in for ChatGPT. Google alone was the weakest proxy in the study. Ranking in the top results for your prompt text tells you little about whether ChatGPT retrieved you, because ChatGPT is searching for something else.
Run prompts more than once. With nearly half of retrieved URLs showing up in a single run, one pass per prompt mostly measures noise. Repeated runs at least let you separate the stable core from the one-offs. Our earlier post on diminishing returns from reruns covers how many repeats are worth paying for.
Record what you queried. A tracker result is only comparable over time if the model, the tool version, the prompt wording, and the language stay fixed. Log them with every run, and annotate the week a model changes so a proxy shift is not read as a content win or loss.
#Limits worth stating
The study used a small prompt set in two languages, observed signed-in sessions only, and the authors note that their citation definition may include supporting pages not shown to users. They also saw ChatGPT silently switch to a fallback model during long repeat sessions, which is a reminder that the product itself is not a fixed target. Brands are not separated from other entities, so the paper does not support a brand-specific conclusion.
The practical takeaway still holds. Every ChatGPT visibility tool, ours included, observes ChatGPT through a narrower window than the app. Report at the level where the window is widest, and say which window you used.
Related field notes
November 21, 2026 · 5 min
AI answers now hand out your coupon codes
ChatGPT and Google now show discount codes inside answers, and most codes chatbots return do not work. Make sure the codes they find are yours and current.
November 21, 2026 · 5 min
Shallow pages lose in AI search first
Google's head of Search says AI answers leave little room for pages that only restate facts. How to find your shallow pages and decide what to do with each.
November 20, 2026 · 5 min
The ten blue links are back as a tab
Google, Bing and now ChatGPT offer or test a view with only web links. That view ranks pages, not passages, and AI visibility reports often skip it.
Share or discuss
New posts, no spam. Roughly monthly. Unsubscribe with one click.