Tag
#Ai
57 posts tagged Ai.
Google's goto links tax every SERP copy
Google now wraps result links in google.com/goto redirects. The target is scrapers, which includes your rank tracker and some AI answer pipelines.
Sep 24, 2026 · 5 min · Crawlmind Engineering
A browser agent is not a crawler
Agentic browsing runs inside the user's own session, so robots.txt, bot allowlists and crawler analytics all miss it entirely.
Sep 23, 2026 · 5 min · Crawlmind Engineering
Your AI Overview loss depends on the baseline
One Wikipedia study revised its AI Overviews traffic loss by a factor of three. What changed was the baseline.
Sep 23, 2026 · 5 min · Crawlmind Engineering
Preferred Sources makes AI visibility personal
Google now badges a reader's chosen sites inside AI Overviews and AI Mode. A logged-out visibility audit cannot see that layer at all.
Sep 23, 2026 · 5 min · Crawlmind Engineering
IndexNow reaches Bing, not every AI engine
IndexNow pings reach a short list of engines. Google, OpenAI's crawler and Perplexity are not on it, so treat it as a Bing freshness tool, not an AI one.
Sep 17, 2026 · 5 min · Crawlmind Engineering
AI crawlers fetch your site from the US
GPTBot reaches your site from three US cities. If your pages adapt to visitor country, that is the only version an AI engine ever reads.
Sep 15, 2026 · 5 min · Crawlmind Engineering
Your crawler allowlist trusts a string
AI crawler identity is self-reported. Published IP ranges and Web Bot Auth signatures are the only two ways to actually check it.
Sep 15, 2026 · 5 min · Crawlmind Engineering
Read your AI crawler log by status code
Verified AI bots get a 200 on 73% of requests. The rest are 403s, 404s and redirect hops, and most of that is fixable at your edge.
Sep 14, 2026 · 5 min · Crawlmind Engineering
Your sentiment chart is mostly noise
Brand mention barely moves between runs. The sentiment label flips about seven times more often, so most weekly tone charts measure nothing.
Sep 13, 2026 · 6 min · Crawlmind Engineering
Your audit stops at turn one
GEO tracking scores the first answer. A fifth of real AI search sessions keep going, and the follow-up turn re-decides who gets named.
Sep 13, 2026 · 5 min · Crawlmind Engineering
Agents quit where users would retry
A 10 second delay collapsed one web-agent benchmark from 42% to 2%. Agents get one pass at your site, with no second try.
Sep 12, 2026 · 6 min · Crawlmind Engineering
Your video is read, not watched
AI answers lean hard on YouTube, but the attributes that track with citation are description, timestamps and recency, not views or subscribers.
Sep 12, 2026 · 5 min · Crawlmind Engineering
Incumbency breaks at 0.075 stars
AI assistants default to the brand they already know, but only until one comparable number separates the options.
Sep 11, 2026 · 5 min · Crawlmind Engineering
Your images reach AI answers as text
Alt text reaches only half of the major AI crawlers, and the multimodal GEO research shows caption text, not pixels, is what moves visibility.
Sep 10, 2026 · 5 min · Crawlmind Engineering
The fifth rerun buys almost nothing
A decomposition of 12,933 LLM brand answers shows language and model coverage cut measurement error far faster per query than extra reruns.
Sep 09, 2026 · 5 min · Crawlmind Engineering
AI agents fall for consensus, not hidden text
LLM search agents ignore hidden text but endorse fabricated products when sources appear to agree. Most of that source pool is not your site.
Sep 08, 2026 · 5 min · Crawlmind Engineering
Deep research cites you but gets you wrong
On deep research surfaces the link works and the page is relevant, yet the claim attached to it often is not supported. That gap is measurable.
Sep 08, 2026 · 6 min · Crawlmind Engineering
Google's AI opt-out is all or nothing
Search Console's generative AI control pulls your whole property out of AI Overviews and AI Mode, and the report behind it has no clicks.
Sep 07, 2026 · 5 min · Crawlmind Engineering
Your AI bot policy now needs three answers
Cloudflare split AI traffic into Search, Agent, and Training, with new defaults landing September 15. One block rule no longer covers it.
Sep 01, 2026 · 5 min · Crawlmind Engineering
Recognition is not recommendation
AI described 96% of tested brands accurately, yet 89% never showed up in category answers. Those are two different systems.
Aug 31, 2026 · 5 min · Crawlmind Engineering
AI conversion lift: read the fine print
AI traffic went from converting 38% worse to 60% better in sixteen months. Most of that swing is measurement, not shopper behavior.
Aug 28, 2026 · 5 min · Crawlmind Engineering
Your AI crawler rules run on the honor system
Allow and block rules for AI crawlers match a user-agent string anyone can type. Cryptographic signing is arriving, but not yet where citations are decided.
Aug 28, 2026 · 5 min · Crawlmind Engineering
AI commerce: discovery outlives checkout
Checkout protocols churned hard through 2026. The product feed and the crawlable product page are the parts that kept working.
Aug 21, 2026 · 5 min · Crawlmind Engineering
robots.txt now asks what for, not just who
Content Signals and RSL add a purpose layer to robots.txt. The ai-input signal is the one that decides whether you can be cited.
Aug 21, 2026 · 4 min · Crawlmind Engineering
Why your AI traffic still shows as direct
GA4 finally has an AI Assistant channel, but referrer mechanics still dump ChatGPT clicks into Direct. Why, and how to close the gap.
Aug 20, 2026 · 5 min · Crawlmind Engineering
E-E-A-T for AI: authorship, sourcing, dates
The three E-E-A-T signals AI engines can actually parse, and the exact markup and habits that make each one machine-legible.
Jul 03, 2026 · 5 min · Crawlmind Engineering
The cost of AI invisibility: a B2B framework
A qualitative way to reason about what your company loses when AI assistants answer buyer questions without ever mentioning you.
Jul 02, 2026 · 5 min · Crawlmind Engineering
Canonical and hreflang in an AI-answer world
Canonical and hreflang tags still shape which URL AI engines cite, but they reach the model through Bing's index and get honored unevenly.
Jul 01, 2026 · 5 min · Crawlmind Engineering
CDN caching gotchas that hide fresh content
Long TTLs, serve-stale, and Vary headers can keep your updated pages invisible to crawlers. Five CDN caching traps and how to avoid them.
Jul 01, 2026 · 5 min · Crawlmind Engineering
Most AI crawlers don't run your JavaScript
Most AI crawlers read raw HTML and never execute JavaScript, so client-rendered content can vanish from AI answers. Here is what survives.
Jun 30, 2026 · 5 min · Crawlmind Engineering
Block or allow AI crawlers: a decision guide
Whether to block or allow AI crawlers depends on one split: training bots versus search bots. A framework by business model.
Jun 30, 2026 · 5 min · Crawlmind Engineering
llms.txt explained: what belongs in it
What llms.txt is, what the spec says belongs inside it, and an honest look at whether AI engines actually read it in 2026.
Jun 30, 2026 · 4 min · Crawlmind Engineering
robots.txt for AI: partial-access patterns
How to grant some AI crawlers some paths in robots.txt, and how engines actually interpret allow, disallow, and wildcards.
Jun 30, 2026 · 5 min · Crawlmind Engineering
AI crawlers in 2026 and what each one fetches
GPTBot, ClaudeBot, PerplexityBot and Google-Extended do different jobs. Knowing which fetches what lets you allow citations and block training.
Jun 29, 2026 · 5 min · Crawlmind Engineering
FAQ schema in 2026: signal or noise
FAQ schema in 2026 no longer earns a Google SERP feature, but it still feeds AI answer engines when the Q-and-A is real.
Jun 22, 2026 · 4 min · Crawlmind Engineering
Writing for follow-up questions in AI search
AI assistants now generate the next question for the user. Content that answers the follow-up, not just the headline query, gets cited twice.
Jun 22, 2026 · 5 min · Crawlmind Engineering
How freshness signals shape AI answers
Freshness signals tell AI engines how recently a page changed. Here is what dateModified and changelog pages actually do for citations.
Jun 19, 2026 · 4 min · Crawlmind Engineering
What structured data AI engines actually read
Structured data still helps AI engines extract your facts. But the schema landscape narrowed in 2025 and 2026, so the old advice is stale.
Jun 18, 2026 · 5 min · Crawlmind Engineering
Document order beats heading level for AI
A page's heading outline is read in DOM order, not by level. Get the source order wrong and both crawlers and AI extractors misread your structure.
Jun 16, 2026 · 3 min · Crawlmind Engineering
Why your JS-rendered pages can be invisible to AI crawlers
A page can render perfectly in a browser, pass every on-page check, and still be silently de-indexed by an HTTP header your crawler never saw. Here's the X-Robots-Tag trap, why JS-rendered crawls are especially exposed to it, and how to test for it in two minutes.
Jun 15, 2026 · 4 min · Crawlmind Engineering
What "partial" AI-bot access in your robots.txt actually means
Run an AI-crawler check and you'll often see every bot marked "partial" instead of a reassuring green "allowed." That usually means your robots.txt is correct, not broken. Here's how the verdict is computed and when partial is actually a problem.
Jun 15, 2026 · 3 min · Crawlmind Engineering
Why ChatGPT cited nothing, and the one parameter that fixed it
We were tracking which domains ChatGPT cites for a query and getting zero citations back, even for questions that obviously needed fresh web data. The content wasn't the problem. One API parameter was. Here's the before/after and what it means for measuring AI visibility.
Jun 15, 2026 · 4 min · Crawlmind Engineering
Why we built Crawlmind
A short history of why a team of SEO and AI engineers ended up building an audit tool that grades sites for ChatGPT, Perplexity and Gemini instead of just Google.
May 28, 2026 · 2 min · Crawlmind Engineering
How to audit your site for AI visibility in 15 minutes
A focused 15 minute walkthrough that catches the five mistakes that quietly cost SaaS sites their AI citations. No paid tools required, runs against any live URL.
May 27, 2026 · 3 min · Crawlmind Engineering
Who actually gets cited by ChatGPT, and why
A look at the citation patterns we see when ChatGPT answers B2B SaaS queries with its browsing tool. The pattern is consistent, and the leaders are not who you'd expect from Google rankings.
May 15, 2026 · 2 min · Crawlmind Engineering
Should you block GPTBot? An honest breakdown
GPTBot has become the canary for "I take AI seriously" robots-txt edits. Almost every guide gets the analysis wrong. Here's what GPTBot actually does, and the four cases where blocking it makes sense.
May 02, 2026 · 3 min · Crawlmind Engineering
Schema markup that moves the needle for AI search
Most schema.org guides treat every type as equally important. They're not. Here's the short list of types and properties that meaningfully change whether an AI engine cites you, based on what we see across customer audits.
Apr 20, 2026 · 3 min · Crawlmind Engineering
How Perplexity decides what to cite
From watching which pages Perplexity actually cites, a few signals consistently drive the citation set. Here's what mattered, what didn't, and what changes when you optimize for it.
Apr 08, 2026 · 3 min · Crawlmind Engineering
From SEO to AEO. Rewriting your content for answer engines.
Answer Engine Optimization isn't a new acronym for the same SEO playbook. Six concrete differences in how you should write, structure, and measure content when the audience is an LLM that summarizes for a human.
Mar 25, 2026 · 3 min · Crawlmind Engineering
The new E-E-A-T. Signals AI engines look for in 2026.
Google's E-E-A-T framework was a decade old by the time LLMs adopted a version of it. The new signals overlap substantially with the old ones. Here's the rest, with the practical implications for SaaS and content sites.
Mar 12, 2026 · 3 min · Crawlmind Engineering
Why your llms.txt is probably broken (and how to tell)
Across the llms.txt files we parse in customer audits, most have at least one issue that materially reduces their usefulness to AI engines. Here are the seven most common, in order of frequency.
Feb 28, 2026 · 3 min · Crawlmind Engineering
The seven AI crawlers you should know about in 2026
A field guide to the seven crawlers that actually matter for AI search visibility right now. What each one does, the right robots-txt posture for most SaaS, and the common misconfigurations to avoid.
Feb 15, 2026 · 3 min · Crawlmind Engineering
The cost of being invisible to AI. A SaaS pricing case study.
A B2B SaaS noticed organic traffic was flat while pipeline was softer than usual. The cause was not Google. It was being absent from the AI answers their buyers were actually reading.
Feb 01, 2026 · 3 min · Crawlmind Engineering
dateModified. The most underrated SEO lever in 2026.
Most SaaS sites set dateModified once at publish time and never touch it again. The cost is real, the fix is cheap, and the trap of "fake date refresh" is worth understanding before you reach for it.
Jan 18, 2026 · 3 min · Crawlmind Engineering
Why SaaS teams need citation tracking, not rank
Rank tracking measures where Google puts you. Citation tracking measures whether ChatGPT, Perplexity and Gemini actually surface you when a buyer asks. The two metrics increasingly disagree, and only one of them predicts pipeline.
Jan 04, 2026 · 4 min · Crawlmind Engineering
Content strategy for the LLM era. Six principles that replace the old playbook.
The content playbook built for 2018 SEO doesn't survive contact with LLM answer engines. Six principles that replace the old assumptions, with the strategic and operational implications for content teams.
Dec 12, 2025 · 4 min · Crawlmind Engineering
FAQ pages still work in 2026 (if you build them right)
Google deprecated FAQ rich results in 2023. Many teams concluded FAQ markup was dead. The opposite turned out to be true for AI engines, but only if your FAQ is actually a FAQ. Here's the distinction that matters.
Nov 20, 2025 · 3 min · Crawlmind Engineering