Why we built Crawlmind

We started Crawlmind because the old toolchain stopped matching the problem. Existing SEO audit tools were built for one customer: Google. They check titles, meta descriptions, schema, page speed. All things that still matter, but they only describe half the picture in 2026.

#What changed

Three quiet shifts happened in the last 18 months.

1. ChatGPT, Perplexity and Gemini browse the web in real time. When a user asks them a question, they fetch live pages, parse them, and cite the ones they found useful in the answer. Whether your page gets cited depends on signals these engines weight differently than Google does.
2. llms.txt is becoming the de-facto standard for telling AI engines which pages on your site are canonical, summary-ready, or off-limits. Almost no audit tools check for it. Even fewer grade the content of yours.
3. Schema markup is now table stakes for AI answers. Perplexity in particular leans heavily on @type: Article, Person, and BreadcrumbList to decide what to surface. A site without them loses to one that has them, every single time, in our testing.

#What we grade for

A Crawlmind audit checks the legacy SEO surface (titles, canonicals, schema, performance) and the AI-visibility surface:

• llms.txt presence, structure, and freshness
• AI-crawler robots-txt allow/deny posture (PerplexityBot, GPTBot, ClaudeBot, Applebot-Extended, CCBot, Diffbot, and friends)
• E-E-A-T signals (author, dateModified, organizational publisher)
• Citation readiness ("atomic-answer" coverage, statistic density, authoritative outbound links)
• Real-time engine probes. Does ChatGPT actually cite you for the queries you care about?

#Why this works

Because the AI-visibility scores aren't synthetic. The "does ChatGPT cite you" panel hits OpenAI's Responses API with web_search_preview and reads the citations off the structured response. Same for Perplexity Sonar. The numbers reflect what your actual customers see when they ask their actual chatbot a question about your space.

That's the bet. If it pays off you'll know, because your traffic mix will shift from "10 SEO plus 0 AI" toward "10 SEO plus 8 AI" over a quarter. If it doesn't, the audit still flags the legacy SEO issues that matter for the half of search that's still Google.

Either way, we'll publish what we learn here. Subscribe to the RSS feed and you won't miss a post.