llms.txt
llms.txt is a markdown-formatted text file placed at a website's root path (`/llms.txt`) that serves as a guide to help large language models (LLMs) such as ChatGPT, Claude, and Gemini quickly and accurately understand the site's key content.
llms.txt is a markdown-formatted text file placed at a website's root path (/llms.txt) that serves as a guide to help large language models (LLMs) such as ChatGPT, Claude, and Gemini quickly and accurately understand the site's key content.
Why It Matters
Traditional search engines have crawlers that traverse an entire site and index its pages. In LLM-based AI search, however, the critical process is selecting which content to reference when the model generates an answer to a user's question. llms.txt acts as a content guide that directly tells AI, "The most important information on our site is here."
From a Generative Engine Optimization (GEO) perspective, even if users never visit your website directly, your content can be reflected in the answers AI generates. Providing accurate information to AI models through llms.txt increases the likelihood that your brand will be correctly represented in AI responses. Since Jeremy Howard of Answer.AI proposed the standard in 2024, it has been rapidly adopted by major companies including Anthropic and Cursor. As of February 2026, approximately 10% of domains have implemented llms.txt.
Structure and Format of llms.txt
llms.txt uses Markdown syntax because LLMs can naturally interpret Markdown. The structure according to the official specification is as follows:
- H1 header (required): The project or site name
- Blockquote (optional): A one-line summary of the project
- Free-form body (optional): Additional description of the project (any Markdown except H2)
- H2 sections + file lists (optional): Key URL lists grouped by topic
## Optionalsection (optional): Supplementary links that can be omitted when context length is limited
Below is a practical example:
# inblog
> SEO-optimized blog platform. Provides blog creation, management, and publishing.
## Guides
- [Getting Started](https://docs.inblog.ai/start): From blog creation to publishing your first post
- [SEO Settings](https://docs.inblog.ai/seo): Search engine optimization guide
## API Reference
- [REST API](https://docs.inblog.ai/api): Endpoint list and usage
## Optional
- [Changelog](https://docs.inblog.ai/changelog): Recent update history
Each link follows the format [Name](URL): Description, where the description after the colon is optional.
Differences from robots.txt
While robots.txt and llms.txt have similar names, their purposes are entirely different.
| Aspect | robots.txt | llms.txt |
|---|---|---|
| Purpose | Allows or blocks crawler access | Guides AI models to key content |
| Format | Directive-based (User-agent, Disallow) | Markdown-based (H1, H2, link lists) |
| Target | Search engine crawlers (Googlebot, etc.) | LLMs and AI agents (GPTBot, ClaudeBot, etc.) |
| Function | Crawl control (block/allow) | Content recommendation (guide/navigation) |
| Relationship | Not a replacement | Used alongside robots.txt |
In other words, if robots.txt is an access control file saying "Don't come here," llms.txt is a content map saying "Important information is here." The two files are complementary. To block AI crawling, you must set GPTBot, ClaudeBot, Google-Extended, and similar User-Agents to Disallow in robots.txt.
Setup Guide
The procedure for implementing llms.txt on your website is as follows:
- Select key content: Identify the pages you want AI to reference. Product introductions, documentation, guides, and FAQs—pages that represent your brand—should be prioritized.
- Write the file: Create a Markdown file following the structure above. Place the site name in H1, a one-line summary in a blockquote, and URL lists under H2 sections.
- Deploy to the root path: Deploy the file so it is accessible at
https://yourdomain.com/llms.txt. - Add llms-full.txt (optional): In addition to the summary version llms.txt, providing an
llms-full.txtthat includes full document content allows AI to gain deeper context. - Update regularly: Refresh llms.txt whenever content is added or URLs change. Outdated links can degrade the quality of AI responses.
llms.txt is not yet an official web standard from IETF or W3C, but it is rapidly being adopted as a de facto standard as AI search becomes mainstream. Websites preparing a GEO strategy would benefit from early adoption.
Related inblog Posts
How inblog Helps
inblog automatically generates llms.txt for each blog, helping AI models quickly understand your site structure.