llms.txt — A Map for Language Models

The llms.txt proposal was introduced by Jeremy Howard in September 2024. Rather than controlling access, it gives a website a place to present a concise, curated map of information that may be useful to language models.

The file uses Markdown and is normally published at /llms.txt. It can describe the site and link to authoritative pages or Markdown versions of important documents. The current llms.txt proposal also recommends standard HTML link relations that help machines discover alternative Markdown representations. It complements ordinary pages and robots.txt; it does not replace either.

Some implementations additionally publish llms-full.txt: a much larger export containing substantial or complete documentation in one file. Perplexity’s documentation provides an example. In this experiment, llms-full.txt is treated as an optional companion to llms.txt, not as a sixth messenger.

Publisher-side adoption is visible. Both OpenAI and Anthropic publish an llms.txt file for their own documentation, while documentation and website platforms increasingly generate it automatically.

That does not establish receiver-side adoption. Publishing such a file is different from promising that an AI crawler or retrieval system will automatically request and use files published by other websites. Whether identifiable bots actually look for llms.txt is therefore one of the questions examined by this experiment.