Tag
#Ai
74 posts tagged Ai.
Copilot is testing a judge, not a writer
Microsoft-Decision-1 scores options instead of writing text. If Copilot adopts scorers like it, pages get checked by more fast, cheap judges per answer.
Oct 13, 2026 · 5 min · Crawlmind Engineering
Data voids now get an AI answer anyway
Google now shows AI Overviews for typos and queries with no matching pages. Thin queries about your brand get an answer whether you wrote one or not.
Oct 12, 2026 · 5 min · Crawlmind Engineering
Gemini's new link tag is a step, not growth
Gemini began adding UTM tags to outbound links in late September 2026. Expect a jump in Gemini traffic that is mostly relabeled visits, not new demand.
Oct 12, 2026 · 5 min · Crawlmind Engineering
An AI SEO diagnosis needs an evidence trail
A new benchmark shows AI models name a cause for an SEO problem about half the time even when the evidence cannot support one. Ask for the trail.
Oct 11, 2026 · 5 min · Crawlmind Engineering
Your AI referral mix is not the market's
Published AI referral shares put Claude anywhere from under 1% to 18.5%. The panel decides the number. Measure your own mix before you set engine priorities.
Oct 11, 2026 · 5 min · Crawlmind Engineering
Forum threads now reach Google by push
Google's UGC Fresh Data Program lets approved forums push posts within minutes. Your own pages still wait for the crawler. Plan for that gap.
Oct 09, 2026 · 5 min · Crawlmind Engineering
Slowing Googlebot also slows your AI answers
Google's crawl-rate doc now spells out Retry-After. Every throttle you send Googlebot also delays what AI Overviews and AI Mode know about you.
Oct 09, 2026 · 5 min · Crawlmind Engineering
Your fix reaches AI answers on Google's clock
Google published typical times for crawling, indexing and serving. Since AI Overviews use the same pipeline, they set how long a corrected page takes to show up.
Oct 06, 2026 · 5 min · Crawlmind Engineering
Search traffic was never a contract
A US judge dismissed Penske's AI Overviews suit: expecting Google traffic is not an agreement. Plan visibility as terms Google can change, not a deal.
Oct 06, 2026 · 5 min · Crawlmind Engineering
Fact-check the fields AI engines quote
Google now says all AI-generated content needs a manual fact-check, including titles, meta descriptions, structured data and alt text. Start there.
Oct 04, 2026 · 5 min · Crawlmind Engineering
Check who wrote the tracking parameter
An "AI Overview" tag spotted on Google links came from a Chrome extension, not Google. Attribution built on unverified URL parameters breaks quietly.
Oct 03, 2026 · 5 min · Crawlmind Engineering
Discover is becoming an AI answer feed
Google Discover now mixes AI summaries into the feed and is testing a Dive deeper overview. Publishers need to measure it as an AI surface, not a referral pipe.
Oct 02, 2026 · 5 min · Crawlmind Engineering
Your camera traffic has pages, not queries
Search Console now reports Lens and Circle to Search traffic as Web multimodal. You get pages, clicks and AI impressions, but no queries and no API split yet.
Oct 01, 2026 · 5 min · Crawlmind Engineering
Brand mentions follow the retrieval path
New data on 34,960 unbranded prompts: when an AI engine never retrieves your domain or searches your name, it almost never recommends you.
Sep 30, 2026 · 5 min · Crawlmind Engineering
Find the stage where your brand drops out
A new 34,960-observation study splits AI brand visibility into match, exposure, fan-out and selection. Each failing stage needs a different fix.
Sep 30, 2026 · 6 min · Crawlmind Engineering
A robots.txt block does not unpublish you
A canary-token study found most AI chatbots kept returning a site's content after robots.txt blocked all crawlers or the site went offline.
Sep 29, 2026 · 5 min · Crawlmind Engineering
AI engines cite synthetic pages from the tail
Three studies find AI-written pages in AI answer citations, and the share rises the further engines reach past the ranked head of the web.
Sep 26, 2026 · 5 min · Crawlmind Engineering
Google's goto links tax every SERP copy
Google now wraps result links in google.com/goto redirects. The target is scrapers, which includes your rank tracker and some AI answer pipelines.
Sep 24, 2026 · 5 min · Crawlmind Engineering
A browser agent is not a crawler
Agentic browsing runs inside the user's own session, so robots.txt, bot allowlists and crawler analytics all miss it entirely.
Sep 23, 2026 · 5 min · Crawlmind Engineering
Your AI Overview loss depends on the baseline
One Wikipedia study revised its AI Overviews traffic loss by a factor of three. What changed was the baseline.
Sep 23, 2026 · 5 min · Crawlmind Engineering
Preferred Sources makes AI visibility personal
Google now badges a reader's chosen sites inside AI Overviews and AI Mode. A logged-out visibility audit cannot see that layer at all.
Sep 23, 2026 · 5 min · Crawlmind Engineering
IndexNow reaches Bing, not every AI engine
IndexNow pings reach a short list of engines. Google, OpenAI's crawler and Perplexity are not on it, so treat it as a Bing freshness tool, not an AI one.
Sep 17, 2026 · 5 min · Crawlmind Engineering
AI crawlers fetch your site from the US
GPTBot reaches your site from three US cities. If your pages adapt to visitor country, that is the only version an AI engine ever reads.
Sep 15, 2026 · 5 min · Crawlmind Engineering
Your crawler allowlist trusts a string
AI crawler identity is self-reported. Published IP ranges and Web Bot Auth signatures are the only two ways to actually check it.
Sep 15, 2026 · 5 min · Crawlmind Engineering
Read your AI crawler log by status code
Verified AI bots get a 200 on 73% of requests. The rest are 403s, 404s and redirect hops, and most of that is fixable at your edge.
Sep 14, 2026 · 5 min · Crawlmind Engineering
Your sentiment chart is mostly noise
Brand mention barely moves between runs. The sentiment label flips about seven times more often, so most weekly tone charts measure nothing.
Sep 13, 2026 · 6 min · Crawlmind Engineering
Your audit stops at turn one
GEO tracking scores the first answer. A fifth of real AI search sessions keep going, and the follow-up turn re-decides who gets named.
Sep 13, 2026 · 5 min · Crawlmind Engineering
Agents quit where users would retry
A 10 second delay collapsed one web-agent benchmark from 42% to 2%. Agents get one pass at your site, with no second try.
Sep 12, 2026 · 6 min · Crawlmind Engineering
Your video is read, not watched
AI answers lean hard on YouTube, but the attributes that track with citation are description, timestamps and recency, not views or subscribers.
Sep 12, 2026 · 5 min · Crawlmind Engineering
Incumbency breaks at 0.075 stars
AI assistants default to the brand they already know, but only until one comparable number separates the options.
Sep 11, 2026 · 5 min · Crawlmind Engineering
Your images reach AI answers as text
Alt text reaches only half of the major AI crawlers, and the multimodal GEO research shows caption text, not pixels, is what moves visibility.
Sep 10, 2026 · 5 min · Crawlmind Engineering
The fifth rerun buys almost nothing
A decomposition of 12,933 LLM brand answers shows language and model coverage cut measurement error far faster per query than extra reruns.
Sep 09, 2026 · 5 min · Crawlmind Engineering
AI agents fall for consensus, not hidden text
LLM search agents ignore hidden text but endorse fabricated products when sources appear to agree. Most of that source pool is not your site.
Sep 08, 2026 · 5 min · Crawlmind Engineering
Deep research cites you but gets you wrong
On deep research surfaces the link works and the page is relevant, yet the claim attached to it often is not supported. That gap is measurable.
Sep 08, 2026 · 6 min · Crawlmind Engineering
Google's AI opt-out is all or nothing
Search Console's generative AI control pulls your whole property out of AI Overviews and AI Mode, and the report behind it has no clicks.
Sep 07, 2026 · 5 min · Crawlmind Engineering
Your AI bot policy now needs three answers
Cloudflare split AI traffic into Search, Agent, and Training, with new defaults landing September 15. One block rule no longer covers it.
Sep 01, 2026 · 5 min · Crawlmind Engineering
Recognition is not recommendation
AI described 96% of tested brands accurately, yet 89% never showed up in category answers. Those are two different systems.
Aug 31, 2026 · 5 min · Crawlmind Engineering
AI conversion lift: read the fine print
AI traffic went from converting 38% worse to 60% better in sixteen months. Most of that swing is measurement, not shopper behavior.
Aug 28, 2026 · 5 min · Crawlmind Engineering
Your AI crawler rules run on the honor system
Allow and block rules for AI crawlers match a user-agent string anyone can type. Cryptographic signing is arriving, but not yet where citations are decided.
Aug 28, 2026 · 5 min · Crawlmind Engineering
AI commerce: discovery outlives checkout
Checkout protocols churned hard through 2026. The product feed and the crawlable product page are the parts that kept working.
Aug 21, 2026 · 5 min · Crawlmind Engineering
robots.txt now asks what for, not just who
Content Signals and RSL add a purpose layer to robots.txt. The ai-input signal is the one that decides whether you can be cited.
Aug 21, 2026 · 4 min · Crawlmind Engineering
Why your AI traffic still shows as direct
GA4 finally has an AI Assistant channel, but referrer mechanics still dump ChatGPT clicks into Direct. Why, and how to close the gap.
Aug 20, 2026 · 5 min · Crawlmind Engineering
E-E-A-T for AI: authorship, sourcing, dates
The three E-E-A-T signals AI engines can actually parse, and the exact markup and habits that make each one machine-legible.
Jul 03, 2026 · 5 min · Crawlmind Engineering
The cost of AI invisibility: a B2B framework
A qualitative way to reason about what your company loses when AI assistants answer buyer questions without ever mentioning you.
Jul 02, 2026 · 5 min · Crawlmind Engineering
Canonical and hreflang in an AI-answer world
Canonical and hreflang tags still shape which URL AI engines cite, but they reach the model through Bing's index and get honored unevenly.
Jul 01, 2026 · 5 min · Crawlmind Engineering
CDN caching gotchas that hide fresh content
Long TTLs, serve-stale, and Vary headers can keep your updated pages invisible to crawlers. Five CDN caching traps and how to avoid them.
Jul 01, 2026 · 5 min · Crawlmind Engineering
Most AI crawlers don't run your JavaScript
Most AI crawlers read raw HTML and never execute JavaScript, so client-rendered content can vanish from AI answers. Here is what survives.
Jun 30, 2026 · 5 min · Crawlmind Engineering
Block or allow AI crawlers: a decision guide
Whether to block or allow AI crawlers depends on one split: training bots versus search bots. A framework by business model.
Jun 30, 2026 · 5 min · Crawlmind Engineering
llms.txt explained: what belongs in it
What llms.txt is, what the spec says belongs inside it, and an honest look at whether AI engines actually read it in 2026.
Jun 30, 2026 · 4 min · Crawlmind Engineering
robots.txt for AI: partial-access patterns
How to grant some AI crawlers some paths in robots.txt, and how engines actually interpret allow, disallow, and wildcards.
Jun 30, 2026 · 5 min · Crawlmind Engineering
AI crawlers in 2026 and what each one fetches
GPTBot, ClaudeBot, PerplexityBot and Google-Extended do different jobs. Knowing which fetches what lets you allow citations and block training.
Jun 29, 2026 · 5 min · Crawlmind Engineering
FAQ schema in 2026: signal or noise
FAQ schema in 2026 no longer earns a Google SERP feature, but it still feeds AI answer engines when the Q-and-A is real.
Jun 22, 2026 · 4 min · Crawlmind Engineering
Writing for follow-up questions in AI search
AI assistants now generate the next question for the user. Content that answers the follow-up, not just the headline query, gets cited twice.
Jun 22, 2026 · 5 min · Crawlmind Engineering
How freshness signals shape AI answers
Freshness signals tell AI engines how recently a page changed. Here is what dateModified and changelog pages actually do for citations.
Jun 19, 2026 · 4 min · Crawlmind Engineering
What structured data AI engines actually read
Structured data still helps AI engines extract your facts. But the schema landscape narrowed in 2025 and 2026, so the old advice is stale.
Jun 18, 2026 · 5 min · Crawlmind Engineering
Document order beats heading level for AI
A page's heading outline is read in DOM order, not by level. Get the source order wrong and both crawlers and AI extractors misread your structure.
Jun 16, 2026 · 3 min · Crawlmind Engineering
Why your JS-rendered pages can be invisible to AI crawlers
A page can render perfectly in a browser, pass every on-page check, and still be silently de-indexed by an HTTP header your crawler never saw. Here's the X-Robots-Tag trap, why JS-rendered crawls are especially exposed to it, and how to test for it in two minutes.
Jun 15, 2026 · 4 min · Crawlmind Engineering
What "partial" AI-bot access in your robots.txt actually means
Run an AI-crawler check and you'll often see every bot marked "partial" instead of a reassuring green "allowed." That usually means your robots.txt is correct, not broken. Here's how the verdict is computed and when partial is actually a problem.
Jun 15, 2026 · 3 min · Crawlmind Engineering
Why ChatGPT cited nothing, and the one parameter that fixed it
We were tracking which domains ChatGPT cites for a query and getting zero citations back, even for questions that obviously needed fresh web data. The content wasn't the problem. One API parameter was. Here's the before/after and what it means for measuring AI visibility.
Jun 15, 2026 · 4 min · Crawlmind Engineering
Why we built Crawlmind
A short history of why a team of SEO and AI engineers ended up building an audit tool that grades sites for ChatGPT, Perplexity and Gemini instead of just Google.
May 28, 2026 · 2 min · Crawlmind Engineering
How to audit your site for AI visibility in 15 minutes
A focused 15 minute walkthrough that catches the five mistakes that quietly cost SaaS sites their AI citations. No paid tools required, runs against any live URL.
May 27, 2026 · 3 min · Crawlmind Engineering
Who actually gets cited by ChatGPT, and why
A look at the citation patterns we see when ChatGPT answers B2B SaaS queries with its browsing tool. The pattern is consistent, and the leaders are not who you'd expect from Google rankings.
May 15, 2026 · 2 min · Crawlmind Engineering
Should you block GPTBot? An honest breakdown
GPTBot has become the canary for "I take AI seriously" robots-txt edits. Almost every guide gets the analysis wrong. Here's what GPTBot actually does, and the four cases where blocking it makes sense.
May 02, 2026 · 3 min · Crawlmind Engineering
Schema markup that moves the needle for AI search
Most schema.org guides treat every type as equally important. They're not. Here's the short list of types and properties that meaningfully change whether an AI engine cites you, based on what we see across customer audits.
Apr 20, 2026 · 3 min · Crawlmind Engineering
How Perplexity decides what to cite
From watching which pages Perplexity actually cites, a few signals consistently drive the citation set. Here's what mattered, what didn't, and what changes when you optimize for it.
Apr 08, 2026 · 3 min · Crawlmind Engineering
From SEO to AEO. Rewriting your content for answer engines.
Answer Engine Optimization isn't a new acronym for the same SEO playbook. Six concrete differences in how you should write, structure, and measure content when the audience is an LLM that summarizes for a human.
Mar 25, 2026 · 3 min · Crawlmind Engineering
The new E-E-A-T. Signals AI engines look for in 2026.
Google's E-E-A-T framework was a decade old by the time LLMs adopted a version of it. The new signals overlap substantially with the old ones. Here's the rest, with the practical implications for SaaS and content sites.
Mar 12, 2026 · 3 min · Crawlmind Engineering
Why your llms.txt is probably broken (and how to tell)
Across the llms.txt files we parse in customer audits, most have at least one issue that materially reduces their usefulness to AI engines. Here are the seven most common, in order of frequency.
Feb 28, 2026 · 3 min · Crawlmind Engineering
The seven AI crawlers you should know about in 2026
A field guide to the seven crawlers that actually matter for AI search visibility right now. What each one does, the right robots-txt posture for most SaaS, and the common misconfigurations to avoid.
Feb 15, 2026 · 3 min · Crawlmind Engineering
The cost of being invisible to AI. A SaaS pricing case study.
A B2B SaaS noticed organic traffic was flat while pipeline was softer than usual. The cause was not Google. It was being absent from the AI answers their buyers were actually reading.
Feb 01, 2026 · 3 min · Crawlmind Engineering
dateModified. The most underrated SEO lever in 2026.
Most SaaS sites set dateModified once at publish time and never touch it again. The cost is real, the fix is cheap, and the trap of "fake date refresh" is worth understanding before you reach for it.
Jan 18, 2026 · 3 min · Crawlmind Engineering
Why SaaS teams need citation tracking, not rank
Rank tracking measures where Google puts you. Citation tracking measures whether ChatGPT, Perplexity and Gemini actually surface you when a buyer asks. The two metrics increasingly disagree, and only one of them predicts pipeline.
Jan 04, 2026 · 4 min · Crawlmind Engineering
Content strategy for the LLM era. Six principles that replace the old playbook.
The content playbook built for 2018 SEO doesn't survive contact with LLM answer engines. Six principles that replace the old assumptions, with the strategic and operational implications for content teams.
Dec 12, 2025 · 4 min · Crawlmind Engineering
FAQ pages still work in 2026 (if you build them right)
Google deprecated FAQ rich results in 2023. Many teams concluded FAQ markup was dead. The opposite turned out to be true for AI engines, but only if your FAQ is actually a FAQ. Here's the distinction that matters.
Nov 20, 2025 · 3 min · Crawlmind Engineering