llms.txt generator

What is llms.txt?

llms.txt is a plain markdown file served at the root of a website (https://example.com/llms.txt) that tells AI assistants what the site is about and which pages are worth reading. It was proposed by Jeremy Howard in September 2024 as a lighter-weight cousin of robots.txt and sitemap.xml: robots.txt says what a crawler may read, the sitemap lists everything, and llms.txt says what is worth reading, with a one-line description of each page.

The format is deliberately simple so that a language model can consume it directly inside its context window:

# Site name

> One sentence describing the site.

Optional paragraphs with important context.

## Section

- [Page title](https://example.com/page): One-line description.

## Optional

- [Less important page](https://example.com/other): Skippable if context is tight.

Only the H1 is required. The blockquote summary, the H2 sections and the special ## Optional section are conventions from the spec.

Does llms.txt help SEO?

Honest answer: not directly, and not yet. As of 2026 none of Google, Bing, OpenAI, Anthropic or Perplexity has publicly committed to reading llms.txt as a ranking or citation signal, and Google's John Mueller has compared it to the keywords meta tag. What it does do:

• Costs nothing and cannot hurt. It is a static text file with no effect on your HTML, crawl budget or Core Web Vitals.
• Is read by developer tooling today. Documentation hosts such as Mintlify, Fern and GitBook generate it automatically, and code editors that index documentation (Cursor, for example) consume it, so if you sell to developers it is genuinely used.
• Forces a useful exercise. Writing a one-line description for each important page is the same work as fixing your meta descriptions and internal linking, which do affect search.
• Positions you for adoption. If answer engines start reading it, sites that have one get the benefit on day one.

Treat it as cheap insurance and a documentation aid, not as a growth lever.

How this generator builds your file

1. Reads /robots.txt and /sitemap.xml (including sitemap indexes) to find your public URLs. If there is no sitemap it falls back to the internal links on your homepage.
2. Prioritises shallow URLs (/pricing before /blog/2023/05/post) and fetches up to 30 of them.
3. Pulls each page's title and meta description, falling back to the first heading and paragraph when those are missing.
4. Skips pages marked noindex, non-HTML responses and anything that errors.
5. Groups pages by their first path segment (/docs/* becomes "Docs") and moves deep pages into ## Optional.

The result is a solid first draft. You should still read it once: delete login, checkout and legal pages, and rewrite the summary line in your own words.

Where to put the file

The file must be served at the site root as plain text. Where that lives depends on your stack:

• Static sites, Vite, Next.js, Astro, TanStack Start: drop llms.txt into the public/ folder.
• WordPress: upload to the web root via SFTP, or use a dedicated llms.txt plugin; recent versions of Yoast SEO and Rank Math can generate one for you.
• Webflow, Framer, Squarespace, Wix: no native file hosting at the root; use a redirect rule to a hosted text file, or a 301 to a page that returns text/plain.
• Shopify and other hosted commerce platforms: check the platform's documentation; where the root is locked, a theme-level redirect is usually the only option.

Verify by opening https://yourdomain.com/llms.txt in a browser. You should see raw markdown with a text/plain content type, not your app shell or a 404.

llms.txt vs llms-full.txt

The spec also allows llms-full.txt: a single file containing the full text of every important page, so an assistant can load your whole documentation at once. It is useful for docs sites with under a few hundred pages and worthless for blogs and marketing sites. Start with llms.txt; add llms-full.txt only if you have a real docs set and someone asks for it.

Frequently asked questions

Is llms.txt an official standard?

No. It is a community proposal published at llmstxt.org in 2024. It has meaningful adoption among developer documentation tools but no search engine or major AI lab has formally committed to reading it.

Will adding llms.txt get my site cited by ChatGPT or Perplexity?

There is no evidence that it does today. Citations come from being crawlable, clearly written and linked from pages those systems already trust. llms.txt is a free complement to that work, not a substitute for it.

How many pages should I list?

Fewer than you think. The file is meant to fit in a model's context window alongside your actual content. Twenty to forty well-described pages beats a dump of five hundred. Put long-tail pages under the Optional heading.

Should I block AI crawlers in robots.txt if I have llms.txt?

The two are independent. llms.txt is a guide for assistants that are allowed to read your site. If you block GPTBot, ClaudeBot or PerplexityBot in robots.txt, they will not read llms.txt either.

Can I include pages behind login?

You can list them, but it is pointless: assistants cannot authenticate, and a URL that returns a login page wastes their context. List public pages only.

How often should I update it?

Whenever you add or remove an important page. Most teams regenerate it monthly or wire it into their build so it stays in sync with the sitemap.