What is llms.txt actually proposing?
llms.txt, proposed by Jeremy Howard on 3 September 2024, is a plain markdown file at /llms.txt that gives a large language model a compact, clean map of a site's content. The spec's own reasoning is that "context windows are too small to handle most websites in their entirety," so it proposes an H1 title, an optional blockquote summary, some context sections, and H2-delimited lists of links to markdown versions of pages (each reachable by appending .md to a URL).
That is a sensible answer to a real problem. Documentation sites are noisy—navigation, scripts, marketing chrome—and a model trying to answer a technical question at inference time benefits from a clean, linkable markdown alternative. Nothing about the format itself is oversold.
What have the platforms actually said about it?
Google has said plainly that you do not need it. Its "AI features and your website" guidance states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features," and adds "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary." The page does not mention llms.txt at all—it is simply not part of how Google describes qualifying for AI features.
Documented: the llms.txt spec itself, and Google's explicit statement that no new machine-readable file is required. Unverified: any claim that an AI provider's crawler fetches and uses third-party llms.txt files when deciding what to cite. No such claim appears in the spec, and none is made here.
Does Anthropic publishing one change anything?
Anthropic publishes an llms.txt for its own documentation at docs.claude.com/llms.txt—and that is a different fact from any AI provider consuming llms.txt files published by other people's websites. One is a company organising its own docs for easier reference. The other is a claim about ranking or citation behaviour across the open web. Conflating the two is the most common error in coverage of this topic, and it is worth being precise about it: a vendor using the convention on its own site says nothing about whether that vendor's model treats a third-party site's /llms.txt as a signal.
Publishing an llms.txt for your own docs is not the same claim as an AI provider consuming llms.txt from your website. No source here says the second thing.
So is it worth publishing one?
A defensible yes—treated as documentation hygiene, not a ranking lever. It costs little, it does no harm, and if any model or tool does happen to reference it, that is a bonus rather than the plan. It should not be sold, bought, or reported on as an AI SEO tactic.
One clear title line naming the site or project.
A short blockquote stating what the site or product does.
Brief prose sections giving orientation, not marketing copy.
Markdown links to real pages, grouped under H2 headings, with a short note per link.