llms.txt: Does It Work, and What to Do Instead

What is llms.txt, and what is it not?

llms.txt is a curated reading list for machines: who you are, which pages are canonical, where the authoritative answers live. Jeremy Howard of Answer.AI proposed the spec on September 3, 2024. The premise is practical: HTML built for humans is noisy (navigation, scripts, consent banners) and model context windows are finite, so hand the machine a clean map.

The spec has two tiers. /llms.txt is the short index: a summary plus curated links. /llms-full.txt inlines the full content into one document, so an agent can load everything about you in a single request.

What it is not matters as much. It is not robots.txt: no enforcement exists behind it. It is not a ranking signal: no search engine has said it reads the file for ranking. And it is not access control: a crawler that ignores it loses nothing. robots.txt says "here is what you may fetch"; llms.txt says "here is what is worth reading". Conflating the two produces most of the bad takes.

Does Google use llms.txt?

Google Search does not; Google's Chrome tooling does. Within ten days in May 2026, Google made two moves that point in opposite directions, which is why both camps can quote it.

On May 5, 2026, Google added an llms.txt audit to Lighthouse under a new agentic-browsing category. The audit flags a site if fetching /llms.txt returns a server error, and the documentation says that without the file "agents may spend more time crawling the site to understand its high-level structure" (ppc.land). On May 15, 2026, Google published guidance on optimizing for generative AI features in Search (AI Overviews and AI Mode). llms.txt is absent from it, consistent with what Google's search representatives have said since the spec appeared: standard technical SEO is what counts.

These are two layers. Google Search ranks and cites from its existing HTML index. Agentic browsing, where an AI agent visits your site to complete a task for a user, has different needs, and that is what the Chrome team started auditing. Anyone saying "Google requires it" or "Google killed it" is quoting one layer and ignoring the other.

Do AI crawlers actually read llms.txt?

Rarely. An aggregate analysis of 515 million bot events found that requests for /llms.txt are a negligible share of AI-crawler traffic (published at aeo.press). GPTBot, ClaudeBot and PerplexityBot overwhelmingly request HTML pages, the way search crawlers always have.

Publishing is moving faster than reading. By March 31, 2026, 7.4% of Fortune 500 companies (37 of 500) had shipped an llms.txt (ppc.land). Developer-documentation sites adopted it much faster, because coding agents are the one consumer that demonstrably reads such files today.

The honest summary: shipping llms.txt does not measurably change how the major AI crawlers read your site in 2026. Anyone selling it as an AI-traffic unlock is selling ahead of the evidence.

What controls AI crawlers in 2026?

Five layers, only two of which have teeth. llms.txt is one instrument in the stack.

The layers people argue about (llms.txt, Content Signals) are advisory. The layer that changed crawler behavior overnight is the CDN edge, and many site owners never configured it consciously.

Why do the crawl-to-click numbers matter?

Because AI systems consume content at industrial scale and send back almost no visits. Cloudflare Radar's Q1 2026 data put the crawl-to-refer ratio, pages fetched per referred human click, at roughly 1,276:1 for OpenAI's GPTBot and roughly 23,951:1 for Anthropic's ClaudeBot (Cloudflare). If your business monetizes pageviews, that asymmetry explains the publisher revolt and the pay-per-crawl experiment.

But the ratio measures clicks, and clicks are not the only return. A crawl that never sends a visitor still decides whether a model knows you exist, describes you accurately and names you when a buyer asks for a shortlist. For a B2B company, the AI answer is often the touchpoint, and your analytics never register it. We unpacked the shift in AEO vs GEO vs SEO: the goal moves from winning the click to being the accurately cited answer.

Should you block or allow AI crawlers?

It depends on what your content is for. The same crawl-to-click math cuts differently by business type.

For most B2B companies the answer is open. The marketing site exists so the market knows what you do; an AI system that reads it and repeats it accurately is doing the site's job. Blocking GPTBot to protect content whose purpose is to be known inverts the strategy. The asset worth protecting is accuracy, not access, and keeping accuracy across platforms is the core of AI visibility work.

What happens to your brand when your own site is blocked?

Open, high-authority sources weigh more. Cloudflare blocks AI crawlers by default for new domains, publishers block or meter access, and pay-per-crawl prices what used to be free. As the open web shrinks from a crawler's point of view, retrieval concentrates on corpora that stay open by design: Wikipedia, Wikidata, public registries, academic repositories.

If your site is dark to crawlers, by choice or by CDN default, then your Wikipedia article (if one is warranted), your Wikidata entity and the other open sources become the de facto record AI reads about you. That is why we treat encyclopedic presence as infrastructure rather than vanity. A page exists only if independent sources support notability, and no one can promise what an AI will say. The mechanics are in Wikipedia AEO, the Wikidata and knowledge-graph service and Wikipedia SEO tactics.

How do you write a good llms.txt?

Write it like a dictionary entry, not a slogan, and keep every fact one you can stand behind. Four rules cover most of the value:

1. Define the entity in the first line. If a model reads only twenty tokens, they should say what you are.
2. Point to the full profile. Keep the index skimmable and link llms-full.txt for agents that want everything in one fetch.
3. Pair canonical URLs with facts that carry numbers and dates. "Founded: 2010" is retrievable and repeatable; adjectives are not. Put the price next to the URL when the page states one.
4. State what you do not claim. Models echo their sources. If your file overclaims, the AI answer overclaims, and the prospect's first call begins with a correction. A short "what we do not claim" section is accuracy insurance.

A minimal skeleton:

# Company name: one-line definition

> Two-sentence summary of what you do and for whom.

## Key pages
- [Services](https://example.com/services): what is offered and prices
- [About](https://example.com/about): founding date, location, leadership

## What we do not claim
- No guaranteed outcomes; limits of the service in one line.

Regenerate the file when facts change. A stale file that misstates your prices is worse than no file. We publish our own at the domain root, generated from the site's canonical data, and its closing section states what we do not guarantee.

How WikiBusines helps

llms.txt is the cheapest and least consequential layer of AI readability. We work on the consequential ones. Paid editing on Wikipedia is always disclosed in WikiBusines' name, and we never guarantee what an AI will say.

• AI visibility audit, €490. How assistants currently describe you and which sources they draw on.
• LLM Hub. A machine-readable knowledge hub for your company, built from your own canonical facts.
• Wikidata, from €680. Entity records that engines use to tell you apart from similarly named companies.
• Notability audit, from €490, credited toward the page if you order it within one month. Page creation from €2,400 for English, 4-5 weeks, with 90 days of monitoring and guarantee after publication.

To start, contact us.