What Is llms.txt? Who Reads It and Whether It Helps SEO

llms.txt is a plain-text markdown file that lives at the root of a website (https://example.com/llms.txt) and tells AI systems what the site is about and which pages are worth reading. Think of it as a curated table of contents written for language models rather than for people or for crawlers.

The idea is simple, the format is simple, and the debate about whether it matters is not. This article explains what the file is, who reads it today, what the evidence says about SEO and AI visibility, and how to write one in ten minutes if you decide it is worth having.

Where llms.txt came from

The proposal was published on 3 September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, at llmstxt.org. His argument was practical: language models have limited context windows and struggle with the navigation, ads, scripts and boilerplate on ordinary HTML pages. A site can help by publishing one small file that says, in markdown, "here is what we are, and here are the pages that matter, each with a one-line summary."

It deliberately borrows from two files every site owner already knows:

• robots.txt tells crawlers what they may fetch.
• sitemap.xml lists everything that exists.
• llms.txt says what is worth reading, and why.

Unlike the other two, llms.txt is not an official standard. No standards body owns it, and no search engine has committed to it. It spread because it was easy to adopt and because documentation platforms started generating it automatically.

What the file looks like

The format is intentionally minimal so a model can consume it directly. A complete example:

# Acme Analytics

> Privacy-first web analytics for small teams. No cookies, one script tag, EU hosting.

Acme replaces Google Analytics for sites that need GDPR compliance without a consent banner.

## Docs

- [Quick start](https://acme.example/docs/quick-start): Install the script and see your first pageview in two minutes.
- [Event tracking](https://acme.example/docs/events): Track clicks, form submits and custom events.

## Pricing

- [Plans](https://acme.example/pricing): Free up to 10k pageviews, then flat monthly tiers.

## Optional

- [Changelog](https://acme.example/changelog): Release notes, updated weekly.

The rules from the spec:

1. An H1 with the site or project name. This is the only required element.
2. A blockquote with a one-paragraph summary. Strongly recommended.
3. Zero or more paragraphs of context.
4. H2 sections, each containing a markdown list of links in the form [title](url): description.
5. A section named exactly ## Optional for links that can be skipped when context is tight.

Nothing else. No nested headings, no HTML, no front matter.

A companion file, llms-full.txt, is allowed to contain the complete text of the important pages so a model can load the whole documentation set at once. It makes sense for API references and developer docs with a few hundred pages; it is pointless for a marketing site or a blog.

Who actually reads llms.txt today

This is where honesty matters, because most articles on the topic overstate adoption.

Documentation platforms generate it. Mintlify began producing llms.txt and llms-full.txt automatically for every hosted docs site in late 2024, and Fern, GitBook and others followed. That is why you find the file on the docs of Anthropic, Cloudflare, Stripe and many developer tools: their hosting produces it, not a deliberate SEO decision.

Coding tools consume it. Editors and agents that index documentation on request, such as Cursor's docs feature, will use an llms.txt when it exists because it is cleaner than scraping HTML. If your customers are developers pasting your docs URL into an AI editor, the file genuinely improves their experience.

Search engines and AI assistants have not committed. Google's John Mueller said in April 2025 that no AI service had told Google it uses llms.txt, and compared it to the keywords meta tag. OpenAI, Anthropic and Perplexity crawl the web with their own bots (GPTBot, ClaudeBot, PerplexityBot) and none has published a statement that those crawlers prioritise or even read the file. Server logs from sites that publish one typically show few or no fetches from those bots.

So the accurate summary in 2026 is: widely published, read by developer tooling, ignored by the systems most people hope it will influence.

Does llms.txt help SEO or AI visibility?

Traditional SEO: no. It is not a ranking signal, it does not affect crawling or indexing, and Google has said as much. It also cannot hurt: it is a static text file that does not touch your HTML, your Core Web Vitals or your crawl budget.

AI citations: not directly, not yet. Being cited by ChatGPT, Perplexity or Google's AI Overviews comes from the same things that drive search visibility: crawlable pages, clear writing, direct answers near the top of the page, and links from sources those systems already trust. There is no evidence that a site with llms.txt is cited more often than an equivalent site without one. If you want to check whether AI systems can actually read your pages, that is a question about rendering, robots rules and content structure; an AI website audit tests those directly.

The indirect benefit is real. Writing a one-line description for each of your twenty most important pages is exactly the exercise that fixes vague titles, missing meta descriptions and orphaned pages. Several teams have told us the biggest outcome of creating the file was discovering how many of their key pages had no description at all.

Insurance is cheap. If answer engines do start reading it, sites that already publish one benefit on day one. The cost is ten minutes.

How to write one in ten minutes

1. Start from your sitemap, not your navigation. Pull the list of indexable URLs and sort by importance to a first-time reader: product and pricing pages, then docs and guides, then blog posts.
2. Cut hard. Twenty to forty pages with real descriptions beats two hundred with none. Login, checkout, legal and tag pages do not belong.
3. Write the summary line for a stranger. The blockquote should tell a model what you do, for whom, and what makes you different, in one paragraph.
4. Use your real titles and descriptions. If a page's meta description is good, reuse it. If it is missing, this is the moment to write it and put it on the page as well.
5. Group by intent. Sections like Product, Docs, Pricing, Guides map to what a model will be asked about.
6. Move the long tail to Optional. Old blog posts and changelogs go there.
7. Serve it as plain text at the root. In most frameworks that means dropping the file into the public folder. Open the URL in a browser: you should see raw markdown with a text/plain content type, not your app shell.

Our free llms.txt generator does steps 1 to 6 automatically: it reads your sitemap, fetches your pages, pulls the real titles and descriptions, groups them by section and gives you an editable draft to download.

Should you block AI crawlers instead?

Some site owners are weighing llms.txt against the opposite move: blocking GPTBot, ClaudeBot and similar crawlers in robots.txt. The two are independent. llms.txt is a guide for assistants that are allowed to read your site; if you block a bot in robots.txt it will not read your llms.txt either.

The decision comes down to your business. Publishers whose product is the content have a case for blocking. Companies that sell something else and want to be recommended when someone asks an assistant for options generally want to be as readable as possible, which means allowing the crawlers, publishing clear pages, and treating llms.txt as a small, free extra.

The bottom line

llms.txt is a reasonable ten-minute task with a small guaranteed benefit (a forced review of your key pages) and a speculative upside (early adoption if answer engines start reading it). It is not a growth lever and it does not replace fixing your titles, descriptions, structure and speed. Do it after those, not instead of them. If you are not sure where your site stands on the fundamentals, a free audit will tell you in about a minute.