We respect your privacy.

We use strictly necessary cookies to keep you signed in and to protect against CSRF. With your permission we also use a small amount of first-party analytics to improve the product. We do not sell your data and we do not use third-party advertising trackers. See our cookie policy and privacy policy .

← All posts

An MCP endpoint is not a discovery channel

Crawlmind Engineering··4 min read

An MCP endpoint is a machine-facing interface your site exposes so an AI client can query your content or call your tools directly, and it is a delivery channel rather than a discovery one. That distinction goes missing in most of the "make your site agent-ready" advice published this year. Standing up an /mcp route changes what an agent can do once it arrives. It does not change how many agents arrive.

#What NLWeb actually shipped

Microsoft introduced NLWeb on May 19, 2025 as "an open project developed by Microsoft that aims to make it simple to create a rich, natural language interface for websites using the model of their choice and their own data." The same announcement makes the protocol link explicit: "Every NLWeb instance is also a Model Context Protocol (MCP server), allowing websites to make their content discoverable and accessible to agents and other participants in the MCP ecosystem if they choose."

Read the end of that sentence literally. It offers discoverability to participants in the MCP ecosystem, which is not the same population as the retrieval layer behind an AI answer. Launch collaborators named in the announcement include Eventbrite, Shopify, Tripadvisor, O'Reilly Media and Hearst, so this is a real project with real adopters, not a thought experiment.

The useful detail for anyone weighing the work: NLWeb is built on semi-structured formats sites already publish, Schema.org and RSS among them. The inputs to an agent interface are the same inputs a crawler reads. Investment in clean markup and clean feeds pays into both columns, which is not true of the endpoint itself.

#Discovery is two competing drafts, and neither has landed

For an engine to find your MCP server without being told about it, there has to be an agreed place to look. Two proposals want to be that place, and they do not agree.

Inside the protocol, SEP-2127 proposes "MCP Server Cards": a JSON document at /.well-known/mcp/server-cards.json carrying the server's name, transports, supported protocol versions, tools and auth details. The proposal was opened against the modelcontextprotocol repository on January 21, 2026 and is still open and unmerged.

Outside it, an IETF Internet-Draft titled "The 'mcp' URI Scheme and MCP Server Discovery Mechanism" specifies a different path, /.well-known/mcp-server, with an optional DNS TXT fallback at _mcp.{domain} and an mcp:// URI scheme. That document is an individual submission with no working group sponsorship. It says of itself that it is "not endorsed by the IETF" and has "no formal standing in the IETF standards process", and the current revision expires on September 25, 2026.

Two incompatible well-known paths, one unmerged, one expiring this week. That is the present state of automatic MCP discovery on the public web. Nothing is crawling for either file at a scale that would change your visibility.

#The door into ChatGPT has a reviewer behind it

There is a real MCP-shaped route into a major assistant, and it is worth being precise about its shape. OpenAI's Apps SDK is built on MCP, and its published guidelines describe "the minimum standard a published plugin must meet to remain available in the universal directory shared by ChatGPT and Codex". The same page notes that "Previously approved plugins that are later found in violation may be removed."

That is an app store, not an index. Someone reviews the submission, approves or rejects it, and can pull it later. Building the MCP server is the tractable half of that project. The distribution half is an application, with a queue and an outcome you do not control. Treating it as a visibility channel means accepting a gatekeeper you did not have to clear for organic citation.

#Citations still come off the crawled page

Meanwhile the pipeline that decides whether you show up in an answer has not moved. OpenAI documents OAI-SearchBot as the agent "used to surface websites in search results in ChatGPT's search features", and its guidance for eligibility is unglamorous: "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges."

The same page draws a line that most teams miss. ChatGPT-User, the agent that fetches a page live while someone is mid-conversation, "is not used to determine whether content may appear in Search." Eligibility is decided by the crawl. The live fetch only serves a person who is already looking. An MCP server sits in neither path, which is the whole point: it is a third surface, reachable by clients that were configured to reach it.

#A defensible order of operations

Put the endpoint last, and be honest about what each step buys.

First, a crawlable HTML answer on the page itself. This is the eligibility gate for every engine that cites, and no protocol work substitutes for it.

Second, structured data and feeds. Schema.org markup, accurate dates and a clean feed serve the crawler that decides your citations and are also what NLWeb ingests if you later stand it up. One piece of work, two payoffs.

Third, the MCP endpoint, scoped as a product surface rather than a marketing one. It makes sense for logged-in users, named partners, internal agents and a directory listing you have decided to apply for. If you build it, publishing both well-known files costs an afternoon and hedges whichever draft wins. Do not book the traffic in advance.

Fourth, measure it apart from everything else. MCP traffic arrives as API calls, not page views, so it will not appear next to your citation data and should not be blended into it.

The agentic web may well end with most sites exposing a tool interface. That future does not arrive through a well-known path nobody queries. For now the sentence that gets you cited is still a sentence on a page a crawler can fetch.

Related field notes

Share or discuss

Field notes in your inbox

New posts, no spam. Roughly monthly. Unsubscribe with one click.