llms.txt
llms.txt is a plain-text file at the root of a website that gives large language models a curated, machine-readable map of the site: what the site is, who stands behind it, and which URLs carry the canonical content. It plays the role robots.txt plays for crawlers, but for retrieval and comprehension rather than permission.
What belongs in the file
A short description of the organization and its expertise, the named entities behind it, and an annotated list of the pages that represent the site's best answers. Annotation matters more than completeness: a sitemap lists everything, while llms.txt should say what each thing is for.
How it differs from its neighbors
robots.txt governs access. sitemap.xml governs discovery. llms.txt governs interpretation — it tells a model which of your thousand pages actually answer questions, and in whose voice.
A structure that works
Open with an H1 naming the site, follow with a blockquote-style summary of what the organization does and who stands behind it, then group annotated links under headings such as Core Definitions, Research, People, and Methods. Each link gets one clause explaining what question it answers. Close with contact and licensing notes so a model knows how the material may be used.
- —Keep it under roughly 3,000 words; curate rather than dump.
- —Annotate every URL with the question it answers.
- —Name the human entities and link to their author hubs.
- —Refresh it whenever a cornerstone page ships, not on a calendar.
- —Serve it as text/plain at /llms.txt with no redirect chain.
The companion files
llms.txt is one of a small family. sitemap.xml enumerates everything for crawlers. robots.txt sets permissions, including whether specific AI user agents may fetch. Some publishers add an llms-full.txt containing the concatenated text of cornerstone pages so a retrieval system can ingest the corpus in one request. Publishing all four costs an afternoon and removes ambiguity about what your site is.
How to tell whether it is being used
Check server logs for requests to /llms.txt by AI user agents, and watch whether answers about your organization start echoing the phrasing you used in the file's summary. Adoption is uneven across engines, so treat the file as cheap insurance rather than a guaranteed channel.
Frequently asked
- Is llms.txt an official standard?
- It is a community convention rather than a formal specification, and support varies by engine. The cost of publishing one is trivial and it doubles as a clean site map for humans auditing your corpus.
- Does llms.txt block AI training?
- No. It is descriptive, not restrictive. Access controls belong in robots.txt and your terms.
Related definitions
Answer Engine Optimization
Answer Engine Optimization (AEO) is the practice of structuring content so answer engines — ChatGPT, Google AI Overviews, Perplexity, Claude, and Copilot — can extract, cite, and reuse it as the answer to a question. Where SEO competes for a click on a ranked link, AEO competes to be the sentence the machine says out loud.
Generative Engine Optimization
Generative Engine Optimization (GEO) is the practice of influencing what a generative model says about a topic, brand, or person — through the sources it retrieves and the training corpus it absorbed. AEO targets a single extractable answer; GEO targets the model's overall representation of you.
Entity SEO
Entity SEO is the practice of making the people, organizations, and concepts on a site unambiguously identifiable to machines — through consistent naming, structured data, and links to authoritative profiles. Search and answer engines reason over entities, not keywords, so an unresolved entity cannot be credited, cited, or trusted.