Your domain and AI: llms.txt, crawlers, MCP

1 Introduction

Your website is no longer read only by people and search engines. AI companies crawl it to train models and to answer questions, assistants fetch pages when a user asks about you, and AI agents look for tools they can call on your behalf. A handful of files and endpoints decide what they may read and what they may do.

The AI & agents section of a domain scan collects them in four blocks: llms.txt (a guide you offer to language models), the AI crawling policy (what your robots.txt allows each AI crawler to do), agent discovery files, and any MCP server your domain announces. Most of this is a matter of choice. One part is a real security issue: an MCP server that answers anyone, without authentication.

Known AI crawlers the scan checks against your robots.txt

2024

Anthropic publishes the Model Context Protocol (MCP)

Art. 4

EU Directive 2019/790: the text and data mining opt-out must be machine-readable

2 How It Works

Each block answers one question. The scan reads public files over HTTPS and, only when your domain announces an MCP server, talks to it with the protocol itself.

What do you tell AI?

llms.txt and agent discovery files: what you chose to present to models and agents, and whether it leaks internal URLs.

What may AI read?

robots.txt rules per crawler, Content-Signal, and the EU text and data mining reservation in tdmrep.json.

What may AI do?

MCP servers: which tools an agent can call, whether a token is needed, and whether those tools change data.

Policy versus security

Blocking or allowing AI crawlers is a business decision, not a vulnerability: the scan reports it as info and it does not change your score. Two things are different. Internal URLs advertised in llms.txt (medium), and an MCP server that lets anyone call tools that change state or that carries hidden instructions for the model (high). Those are the findings to act on first.

3 Detailed Analysis

llms.txt: a map for language models

/llms.txt is a markdown file that tells language models which pages of your site matter, with a short summary. It is a community proposal (llmstxt.org), not an IETF or W3C standard, and no AI company is obliged to read it. A companion /llms-full.txt sometimes holds the full content in one file.

# Example Corp

> Invoicing software for small businesses.

## Docs
- [Getting started](https://example.com/docs/start): set up an account
- [API reference](https://example.com/docs/api)

No llms.txt is fine

Its absence is not a weakness and the scan does not flag it. Publish one only if you want to guide how models present your site.

What the scan reports

Title, summary, sections, the number of links (and how many are external), and whether llms-full.txt exists. It raises a medium finding when a link points to an administrative or internal URL: paths like /admin, /wp-admin, /.env, /.git, phpmyadmin, grafana, jenkins, or hosts like staging. and internal.. Generated files pick these up easily; remove them.

AI crawling policy: robots.txt, Content-Signal and TDMRep

AI crawlers identify themselves with a user-agent name and, for the major operators, follow robots.txt. They do three different jobs: training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and fetch on user request (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User). You can treat them differently. To refuse training but stay visible in AI search:

# Refuse model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including AI search and user-requested fetches
User-agent: *
Disallow: /cart/

How the scan reads it

For each crawler: Blocked (Disallow: / with no Allow: /), Partly (some paths disallowed) or Allowed, and whether the crawler has its own rule or falls under User-agent: *. No robots.txt means every crawler is allowed by default.

Google-Extended and Applebot-Extended are tokens

They do not crawl by themselves. They tell Google and Apple whether content fetched by their normal crawlers may be used for AI. Blocking them does not remove you from Google Search or Apple search features.

Content-Signal

A newer robots.txt line, proposed by Cloudflare, that states what content may be used for: Content-Signal: search=yes, ai-input=yes, ai-train=no. The scan shows each signal as yes, no or unset. Like the rest of robots.txt, it is a declaration that crawlers choose to honour.

TDMRep: the EU opt-out

In the EU, text and data mining of public content is allowed unless the rights holder reserves it "in a machine-readable way" (Directive 2019/790, article 4). The TDM Reservation Protocol (a W3C community group specification) does this with /.well-known/tdmrep.json. The scan reports whether the file exists and whether it reserves rights (tdm-reservation set to 1).

[
  { "location": "/", "tdm-reservation": 1, "tdm-policy": "https://example.com/tdm-policy.json" }
]

Agent discovery files

Agents look for machine-readable descriptions of what your domain offers. The scan checks:

A2A agent card

/.well-known/agent-card.json (or the older agent.json): an agent you run, its endpoint, its skills and its security schemes. The report shows "no security scheme declared" when the card lists none: check that the endpoint really requires authentication.

MCP server cards and the _mcp TXT record

/.well-known/mcp/server-card.json, /.well-known/mcp.json, /.well-known/mcp-server, or a _mcp TXT record in DNS. These conventions are still emerging. A card announcing an endpoint that does not answer as an MCP server is flagged low: remove stale cards.

ChatGPT plugin manifest

/.well-known/ai-plugin.json belongs to a plugin platform OpenAI retired in 2024. If it is still there, it is flagged low: delete it, along with the API spec it points to if nothing else uses it.

Exposed MCP servers

The Model Context Protocol lets an AI agent connect to a server and call its tools (functions such as "search orders" or "create ticket"), read its resources and use its prompts. An MCP server reachable from the internet is an API for agents, and it needs the same protection as any API.

The scan only connects where your domain announces a server (a server card, an _mcp record, an llms.txt link, or a host named mcp.*), never by guessing paths. It sends initialize, lists tools, resources and prompts, and never calls a tool.

Authentication required: the good case

The server refuses to list anything without a token. The report then shows what it publishes about getting one: authorization server, scopes, client registration (dynamic registration per RFC 7591, a client metadata document, or pre-registered clients only). The MCP specification builds this on OAuth.

Open, with tools that change state (high)

Anyone on the internet, human or agent, can call tools that write, send, delete or deploy. The scan classifies tools from their annotations and names (delete, create, send, exec, transfer...). Treat it like an unauthenticated admin API.

Tool poisoning (high)

Tool descriptions are read by the model as part of its context. When a description carries instructions aimed at the model, any agent that loads the server can be steered by them. Check that nobody altered your descriptions, and that they come from your own code, not from user-supplied data.

Open and read-only (info), or over plain HTTP (medium)

A public, read-only server can be intended (public documentation search). Make sure it really only returns public data. A server on http:// sends tokens and answers in clear text: move it to HTTPS.

Fix order for an open server: require authentication (or take it offline) today, then review every tool that changes state, scope tokens to the minimum, and log every call. Exposed services in general are covered in exposed vulnerabilities.

Check your domain on mlab.sh

A domain scan reads your llms.txt, your AI crawler rules and TDMRep file, your agent discovery files, and inspects any MCP server you announce without calling a single tool.

Scan your domain on mlab.sh

4 Red Flags

An MCP server open to anyone, with tools that write or delete

The most urgent finding in this section. Require authentication or take the server offline before anything else.

Tool descriptions that talk to the model

"Ignore previous instructions", a fake <system> tag, "do not tell the user", an order to read ~/.ssh or .env, or invisible Unicode characters: the scan flags all of these as tool poisoning. Find out how they got there.

Admin, staging or internal links in llms.txt

You are handing every model a list of internal entry points. Regenerate the file from public pages only.

An agent card with no security scheme

Either the agent endpoint is open, or the card is incomplete. Check which, and declare the authentication it uses.

A crawler policy nobody chose

Not a vulnerability, but if the scan shows every AI crawler allowed (or blocked) and nobody decided that, take the decision with your legal and marketing teams and write it in robots.txt.

Related Modules

security.txt, robots.txt and well-known endpoints

Publish a security contact with RFC 9116, understand what /.well-known/ exposes, and why robots.txt is not a security control.

Third-party scripts and browser supply chain

Every external script your site loads runs with your users' trust. Supply-chain risks, Subresource Integrity and self-hosting.

How to read a domain scan report

Score, grade, severities and report sections: how to read an mlab.sh domain scan and decide what to fix first.