AI search
What is llms.txt and do you need one?
llms.txt is a proposed Markdown file that gives language models a short map of your site. Here is how it is structured and what it can and cannot do.
What llms.txt is
llms.txt is a proposal for a Markdown file at the root of a site, at /llms.txt, that gives language models a concise, curated map of the site: what it is and where the most useful pages are. Jeremy Howard of Answer.AI published the proposal in September 2024. The reasoning is that full web pages, with navigation, scripts and layout, are awkward for a model to read within a limited context, while a short text summary with links is easy.
The proposal also suggests offering clean Markdown versions of important pages at the same URL with .md appended.
How the file is structured
The format is ordinary Markdown in a fixed order:
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary of what the site is.
- Optional paragraphs or lists with more detail, without headings.
- H2 sections, each containing a list of links, with an optional note after a colon.
- An optional section titled Optional, for secondary links a model can skip when it needs a shorter context.
# Example Dental Software
> Scheduling and reminder software for independent dental clinics.
All plans include online booking and text reminders.
## Product
- [Features](https://www.example.com/features): What the software does
- [Pricing](https://www.example.com/pricing): Plans and what each includes
## Help
- [Getting started](https://www.example.com/help/start): Setup steps in order
## Optional
- [Blog](https://www.example.com/blog): Articles and announcementsWhat it does and does not do
- It is not a standard. It is a community proposal, not a specification from a standards body such as the IETF or W3C.
- No major AI search engine has publicly confirmed using it for ranking or citations. Google says AI Overviews need no special optimizations beyond standard SEO.
- It does not control access. robots.txt still decides which crawlers may fetch your pages.
- It does not replace an XML sitemap, which remains the way to list your URLs for search engines.
Should you add one?
It costs little to add, so publishing one is reasonable as long as you keep it accurate. It is most common on software documentation sites, where developers sometimes hand the file to coding assistants as context. Do not expect it to change how often AI search engines cite you.
- List only canonical, indexable pages that answer real questions.
- Use the same names, prices and facts as the pages it links to.
- Update it when you add, move or remove important pages, and check for broken links.
- Keep it short. It is a map, not a copy of the site, and not a place for keyword lists.
Put it to work
- AI crawler access and llms.txtAI engines can only cite pages they are allowed to read. Here is what Nalza checks and fixes.
- Free auditCheck your homepage in about a minute and see a preliminary score with your top issues.
More in AI search
- What is generative engine optimization (GEO)?GEO is the practice of making your site easy for AI answer engines to find, quote and cite. Here is how it relates to SEO and where to start.
- How do AI answer engines choose their sources?AI answer engines retrieve pages from a search index or a live fetch, then write an answer from them. Here is what that means for getting cited.
- What is Share of Answer in AI search?Share of Answer measures how often AI engines cite or mention you across a fixed set of prompts. Here is how to define it, sample it and compare it.
- How do Google AI Overviews work?Google AI Overviews summarize answers from Google's index and link to supporting pages. Here is what makes a page eligible and how to control it.
- AI crawlers and robots.txt: what to allowTraining crawlers, search crawlers and user-triggered fetchers do different jobs. Here is how to allow AI search bots in robots.txt and block training.
- How to write content AI engines can citeAI engines quote passages that answer a question clearly and stand on their own. Here is how to structure, source and maintain content they can cite.