What Is llms.txt, and Does Your Site Need One?

Every few months a new file promises to get you into AI answers.

Right now, that file is llms.txt.

Agencies sell it as an "AI SEO" fix. Clients ask me whether they are falling behind without one.

Probably not.

Below: what llms.txt is, what Google has said about it in writing, and when I think it is worth adding.

What Is llms.txt?

llms.txt is a plain-text Markdown file that summarises a website for large language models and lists links to its most useful pages, so an AI tool can understand the site without crawling all of it.

That's it.

No code to install. No tags on your pages. Just a text file at an address like yourdomain.com/llms.txt.

A minimal one for a consultancy site might look like this:

  # Example Consulting

  > Independent bookkeeping and payroll for UK construction firms with 2 to 50 staff.

  ## Services

  - [Bookkeeping](https://example.com/bookkeeping.md): monthly bookkeeping in Xero, fixed fee
  - [Payroll](https://example.com/payroll.md): weekly and monthly payroll, CIS returns

  ## Optional

  - [Blog](https://example.com/blog.md): guides on CIS, VAT and cash flow

A human can read it in twenty seconds. That is the point.

Where the idea came from

The proposal was published by Jeremy Howard on llmstxt.org in September 2024.

The current version, which the site describes as an update based on two years of adoption, was last revised in August 2026.

The site is open about what it is: a specification open for community input. Not a standard, not a protocol any search engine has signed up to.

What llms.txt is not

Three confusions come up in almost every conversation I have about it:

• It is not an access control. It cannot block or allow a crawler. That is still robots.txt and your firewall.
• It is not a sitemap. A sitemap lists everything. llms.txt is a short, curated shortlist with notes.
• It is not GEO. Getting named by chat assistants depends on your reputation across the web, which is what generative engine optimisation deals with.
• It is not a ranking signal. Google has said so plainly, as you'll see below.

How the llms.txt Format Works

The format is deliberately loose.

According to the proposal, the parts appear in this order:

That last section is clever.

AI models can only hold so much text at once. Marking some links as optional lets a tool drop them first.

The file can sit at the root of the domain or inside a folder, where it covers the pages beneath that path.

The Markdown copies of your pages

The proposal goes one step further.

It suggests that useful pages also exist as clean Markdown at a matching address, for example page.md or page.html.md.

Pages can point to those copies with a link element of type text/markdown in the HTML head.

This part asks a lot more of a site than one text file does. For most business sites I would not bother.

llms.txt vs robots.txt vs sitemap.xml

The three files get lumped together, but each does a different job:

The llms.txt proposal itself frames it this way. It says the file can complement robots.txt by giving context for content you already allow, and that sitemaps are no substitute because they are too large and carry no explanation.

If you have not looked at your robots.txt in a year, do that before anything here. A blanket AI-bot block does far more damage than a missing llms.txt.

What Google Says About llms.txt

This is where most of the hype falls apart.

In its guide to optimising for generative AI features on Google Search, Google says you do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, because Google Search does not use them.

It goes further.

The guide says it is fine to maintain llms.txt for other services, but doing so will neither harm nor help your visibility or rankings, because Google Search ignores the files.

That covers AI Overviews and AI Mode too.

So if someone sells you llms.txt as a way into Google's AI answers, you now have Google's own words to send back.

Chrome takes a softer line

There is a wrinkle.

Lighthouse, the auditing tool built into Chrome, now has an llms.txt audit in its agentic browsing checks.

Read the details, though. The audit flags a server error when the file is requested. If the file simply returns a 404, the audit is marked not applicable, because providing it is optional.

So one Google team checks the file is not broken. Another says Search ignores it.

Neither says you need one.

Who Actually Reads llms.txt?

Honest answer: it is hard to know.

Google Search has said it does not. I have not found documentation from the other major AI search products confirming they use it to choose sources, so I treat any claim that they do as unproven until the vendor says so.

The original use case was narrower and more convincing.

Coding assistants and AI agents often need to read software documentation on demand. A tidy index of the docs, with Markdown versions of each page, saves them wading through navigation menus and scripts.

You can check your own site in ten minutes:

1. Pull your server or CDN access logs for the last 30 days.
2. Filter for requests to /llms.txt.
3. Look at the user agents making those requests.

If almost nothing requests it, you have your answer for your site.

Note: Audit tools, Lighthouse included, now request llms.txt just to check whether it loads. A request in your logs is not proof that an AI assistant used it.

Does Your Site Need One?

For most sites, no. For a few, it is a sensible extra.

Here is how I decide:

My rule is simple.

If it takes an hour or less and your foundations are in order, add it. If someone wants to charge you a project fee for it, spend that money on the things that do get pages cited.

Those things are dull. Crawler access. Pages that are indexed. Sections that answer a question in their opening line. Mentions on the sites AI answers already lean on. I compare them with classic ranking work in AI search vs traditional SEO.

That is the work inside proper AI search optimisation, and none of it depends on a text file.

How to Create an llms.txt File in 5 Steps

If you decide to add one, keep it short and keep it accurate.

Step 1: Pick the pages that explain your business

Choose 10 to 30 URLs a newcomer would need: core services or products, pricing, key guides, about and contact.

Leave out tag pages, thin posts and anything you would not show a new client.

Step 2: Write the H1 and summary

Use your real business name in the H1.

Then write one blockquote sentence that says what you do, for whom and where. Use the same wording as your About page, so the description matches everywhere.

Step 3: Group links under clear H2s

Use headings like "Services", "Guides" and "Company".

Give each link a short note after a colon. Write it as a fact, not a slogan. "Fixed-fee payroll for firms under 50 staff" beats "award-winning payroll".

Step 4: Move secondary links into "Optional"

Archives, older posts and nice-to-have pages go under an H2 called Optional, so a tool short on space can skip them.

Step 5: Publish it and keep it in step with the site

Upload the file to your root, check it returns a 200 status as plain text, and add it to the checklist you run when pages change.

A file full of redirected or deleted URLs is worse than none.

If you run a technical SEO review each quarter, add llms.txt to it alongside robots.txt and the sitemap. My technical SEO checklist covers the rest of that review.

Mistakes to Avoid With llms.txt

Most of the problems I see are about expectations, not syntax:

• Paying for it as an AI ranking fix. Google says Search ignores it.
• Treating it as a block list. Writing "do not use my content" in llms.txt does nothing. Use robots.txt and each vendor's crawler controls.
• Dumping every URL into it. That turns it into a worse sitemap.
• Letting it go stale. Old prices and dead links give any tool that reads it the wrong picture.
• Forgetting the real blockers. A firewall that challenges AI crawlers hides your pages no matter what the file says. So does a stray indexing setting, which is what happened in this language services case study.

On that last point, I explain crawler access step by step in my guides to getting cited by Perplexity and appearing in Google AI Overviews.

Spend the Hour Where It Counts

llms.txt is a reasonable idea with a narrow use.

It helps AI tools that choose to read it, mainly around documentation. It does nothing for Google Search, and Google has said so in writing.

So here is your first action: open your robots.txt and your firewall settings, and confirm the search crawlers you care about are allowed. Then decide on llms.txt.

If you want to see what does move AI visibility, my guide to Google AI Mode SEO is a good next read. And if you want the whole job looked at, my approach to being cited in AI answers starts with access and indexing, not a text file.