/llms.txt
The llms.txt File, Explained
A single Markdown file at your site root that hands AI tools a clean, curated map of your most important pages. Here is the exact format from the spec, a copy-ready example, and a free generator.
The big idea
A robots.txt for the age of AI — but curated, and in Markdown
It is just Markdown
A plain /llms.txt file at your site root: an H1 name, a one-line summary, then H2 sections of curated links. No new syntax to learn.
It is read at inference time
When a chatbot or AI search tool answers a question about your site, it can pull llms.txt for a clean, low-noise map — not your nav, ads, or scripts.
It is curated, not exhaustive
Unlike a sitemap, llms.txt lists only the pages that genuinely help a model understand you, small enough to fit in a context window.
What llms.txt Is — and Why It Exists
Large language models increasingly rely on website content to answer questions, but they hit a wall: a context window is finite, and a real web page is mostly navigation, ads, and scripts wrapped around the actual information. Converting all of that to clean text on the fly is slow and lossy.
llms.txt solves this by letting you publish the clean version. It is a Markdown file at /llms.txt that names your project, summarizes it in a sentence, and links to your most useful pages — exactly the high-signal context a model needs, and nothing else. The proposal comes from Jeremy Howard at Answer.AI (September 2024), and it deliberately mirrors the well-known /robots.txt and /sitemap.xml conventions.
It pairs naturally with the broader idea of Markdown as the language of AI — Markdown is token-efficient and preserves structure, which is why it is the format the spec chose.
The Format, Part by Part
The spec is precise but short. The sections must appear in this order — and only the first one is required.
| Part | Status | What it is |
|---|---|---|
| H1 — project name | Required | A single # heading with the name of your site or project. This is the only required line in the entire file. |
| Blockquote summary | Optional | A > blockquote with a short description carrying the key information a model needs to understand the rest of the file. |
| Body sections | Optional | Zero or more Markdown sections (paragraphs, lists) of any type except headings — extra context on how to interpret the files below. |
| H2 file lists | Optional | Zero or more ## sections, each a list of links. Each item is a required [name](url) hyperlink, then optionally a colon and a note. |
| ## Optional section | Special | A section literally named “Optional” has special meaning: its links can be skipped when a shorter context is needed. |
A Complete Example
Here is a full llms.txt with every part of the spec. Because it is plain Markdown, it renders cleanly too — the source is on the left, the way a model reads its structure is on the right.
# Acme Docs
> Acme is an open-source toolkit for building
> data pipelines in Python.
Important: Acme is not a hosted service. Everything
runs locally or in your own infrastructure.
## Docs
- [Quick start](https://acme.dev/start.md): Install and run your first pipeline
- [API reference](https://acme.dev/api.md): Every public class and method
## Examples
- [ETL walkthrough](https://acme.dev/etl.md): A complete end-to-end pipeline
## Optional
- [Changelog](https://acme.dev/changelog.md): Full release historyAcme Docs
Acme is an open-source toolkit for building
data pipelines in Python.
Important: Acme is not a hosted service. Everything
runs locally or in your own infrastructure.
Docs
- Quick start: Install and run your first pipeline
- API reference: Every public class and method
Examples
- ETL walkthrough: A complete end-to-end pipeline
Optional
- Changelog: Full release history
Free llms.txt Generator
Fill in your project name, a one-line summary, and the links you want to expose. The valid, spec-formatted file builds live on the right — copy it or download it as llms.txt. Everything runs in your browser.
# Acme Docs
> Acme is an open-source toolkit for building
> data pipelines in Python.
## Docs
- [Quick start](https://acme.dev/start.md): Install and run your first pipeline
- [API reference](https://acme.dev/api.md): Every public class and method
## Optional
- [Changelog](https://acme.dev/changelog.md): Full release history
Generated entirely in your browser — nothing is uploaded. Save the result as llms.txt at the root of your site so it resolves at /llms.txt.
llms.txt vs sitemap.xml vs robots.txt
These live at similar well-known paths, so they get confused. They solve different problems — llms.txt is the only one written for a model to read and small enough to fit in context.
| File | Read by | Scope | Format |
|---|---|---|---|
| llms.txt | LLMs & AI search, at inference time | Curated — only your most useful pages | Markdown |
| llms-full.txt | LLMs that want everything in one file | The full text of those pages, inlined | Markdown (large) |
| sitemap.xml | Search engine crawlers | Exhaustive — every indexable page | XML |
| robots.txt | All crawlers & bots | Access rules — what may be crawled | Plain text directives |
The Companion: .md Versions of Your Pages
The proposal has a second half that is easy to miss. Wherever a page would help an LLM, serve a clean Markdown copy at the same URL with .md appended. The links inside a good llms.txt usually point at these.
| Human page | Markdown version |
|---|---|
| /docs/intro.html | /docs/intro.html.md |
| /guide/ | /guide/index.html.md |
index.html.md instead.Ways to Generate It
Pick whatever matches your stack. For most small sites, writing it by hand is the right call — the curation is the whole point.
By hand (recommended for small sites)
Use the generator above or just open a text editor. You know which pages matter most; a crawler does not.
Docs framework plugins
VitePress (vitepress-plugin-llms), Docusaurus (docusaurus-plugin-llms), and Drupal can emit llms.txt — and often the .md pages — at build time.
Crawl an existing site
Tools like Firecrawl and Mintlify can crawl your site and produce a starter file. Treat the output as a draft and trim it down to the high-value pages.
Expand to llms-full.txt
The llms_txt2ctx CLI turns an llms.txt into a single context file that inlines the linked content — handy when a tool wants everything at once.
Common Mistakes
The errors that turn a useful llms.txt into noise.
Avoid
Listing every URL like a sitemap
Do
Hand-pick the pages that explain your site
Avoid
Putting it at /docs/llms.txt only
Do
Serve it at the root: /llms.txt
Avoid
Linking to HTML pages full of nav and ads
Do
Link to clean .md versions where you can
Avoid
Skipping the H1 or the summary
Do
Always include the # name; add the > summary
Frequently Asked Questions
What is llms.txt?
llms.txt is a Markdown file placed at the root of a website (yourdomain.com/llms.txt) that gives AI tools a curated, low-noise map of your most important pages. It contains an H1 with the project name, an optional one-line summary in a blockquote, and H2 sections listing links to key resources. It was proposed by Jeremy Howard of Answer.AI in September 2024 to help large language models use a website's content at inference time, since full HTML pages are noisy and often too large for a context window.
What is the exact format of an llms.txt file?
The spec defines a specific order: (1) an H1 with the project or site name — the only required section; (2) an optional blockquote summary with the key information needed to understand the file; (3) zero or more Markdown sections of any type except headings, for extra detail; and (4) zero or more H2 sections containing 'file lists' — Markdown lists where each item is a required [name](url) hyperlink optionally followed by a colon and a note. A section named 'Optional' has special meaning: its links can be skipped when a shorter context is needed.
Where do I put the llms.txt file?
At the root of your domain so it resolves at https://yourdomain.com/llms.txt, the same convention used by robots.txt and sitemap.xml. The spec allows it in a subpath too, but the root is the canonical location tools look for first. It is a static text file — just upload it alongside your other root files or have your framework generate it at build time.
What is the difference between llms.txt and sitemap.xml?
A sitemap.xml lists every indexable page on your site for search engine crawlers, in XML, and is often far too large to fit in an AI context window. llms.txt is the opposite: a short, hand-curated Markdown overview of only your most useful pages, meant to be read at inference time when an AI tool is answering a question about your site. A sitemap aims for completeness; llms.txt aims for a high-signal summary. They serve different purposes and can coexist.
What is llms-full.txt?
llms-full.txt (sometimes generated as llms-ctx.txt or llms-ctx-full.txt) is a companion file that inlines the actual content of the linked pages into a single large Markdown document, rather than just listing links. llms.txt is the lightweight index; llms-full.txt is the full corpus for tools that want to load everything at once. The FastHTML project generates both from its llms.txt using the llms_txt2ctx command-line tool.
Is llms.txt an official standard?
No. It is a community proposal introduced by Answer.AI in September 2024, not an official web standard, and not all major AI crawlers consume it yet. That said, it is cheap to add, it coexists with robots.txt and sitemap.xml, and it is widely supported by documentation frameworks through plugins for VitePress, Docusaurus, Drupal, and others. Adoption has grown to hundreds of sites and is tracked in public directories like llmstxt.site and directory.llmstxt.cloud.
Do AI tools like ChatGPT and Claude actually read llms.txt?
Support is uneven and evolving. The file is designed for inference-time use — for example, when you point an AI coding assistant at a library's documentation, or when an AI search tool gathers context about your site. Some tools and crawlers read it today; others ignore it. Because it is a small, low-cost file that does no harm, the common recommendation is to add it now and benefit as support broadens, rather than wait for universal adoption.
How do I generate an llms.txt file?
You can write it by hand — it is just Markdown — or use the free generator on this page to fill in your project name, summary, and links and copy the result. If your docs run on VitePress, Docusaurus, or Drupal, plugins can generate it automatically at build time. Tools like Firecrawl and Mintlify can also crawl an existing site and produce a starter file you then edit down to the highest-value pages.
What are the .md page versions the spec mentions?
Alongside llms.txt, the proposal suggests serving a clean Markdown version of any page that would help an LLM, at the same URL with .md appended (for example, /docs/intro.html.md). For URLs without a filename, append index.html.md. This gives crawlers and chatbots a parse-friendly copy of the page without changing the human-facing HTML, and it is what the links inside a good llms.txt usually point to.
Related Guides
llms.txt is one piece of the bigger Markdown-for-AI picture. Keep going: