In recent years, the search landscape has changed dramatically, driven by the rise of generative AI search services such as ChatGPT and Google's AI Overview. As AI increasingly provides direct answers in search results, key website content often struggles to get properly picked up and presented by these systems.
To address this issue, a new format called llms.txt has been proposed. It's essentially a simplified, AI-friendly site index and summary file designed specifically to help AI efficiently discover, read, and understand a website's most important content.
The llms.txt file is a simple, markdown-based file located in the root directory (“/”) of a website, designed specifically to guide AI-based search engines (LLMs) toward the URLs they should prioritize during crawling and indexing. Put simply, it clearly highlights key pages that an AI needs to understand, helping these systems more accurately learn from and cite your content.
In a way, you can think of llms.txt as an evolution of the traditional sitemap, a more AI-friendly format optimized for the modern search environment.
# FastHTML
> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.
Important notes:
- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.
## Docs
- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options
## Examples
- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.
## Optional
- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development. Traditional website content often posed challenges for AI chatbots and search engines to accurately understand and learn from. Particularly, HTML-based web pages frequently mixed core content with advertisements, navigation elements, and JavaScript, making it difficult for AI to reliably extract essential information.
In response to these issues, Jeremy Howard, co-founder of Answer.AI, proposed the llms.txt file format in September 2024. The goal was simple: to improve AI learning efficiency and ensure precise content citations in an increasingly AI-driven search landscape.
Since its introduction, llms.txt has quickly gained traction among various companies and platforms.
For instance, Mintlify, a SaaS provider specializing in AI-powered documentation tools for enterprises, reported a 27% increase in AI citation accuracy and an 18% reduction in incorrect citations after implementing llms.txt.
Cursor and Bolt, two leading tech platforms, have also adopted llms.txt, enabling AI systems to more accurately learn and utilize their content.
These examples clearly highlight the practical effectiveness and tangible benefits of llms.txt in real-world use cases.
Although llms.txt is still an emerging technology, it has the potential to become a key standard in the evolving AI search landscape.
Currently, there's ongoing competition with similar file formats such as ai.txt, but the critical point is clear: websites that proactively adapt and prepare now will reap significant benefits in AI-driven search environments moving forward.
Therefore, website owners should strategically structure and highlight core content through llms.txt, making it easily accessible and understandable for AI crawlers.
In an AI-driven search landscape, adopting llms.txt is a crucial strategy to ensure your website’s content is accurately and effectively communicated.
Proactively implement llms.txt today, and position your site ahead of the curve in the age of AI-powered search.
To take your site further and prepare for AI-powered search, read: GEO vs SEO: Know the difference and get ahead.
1️⃣
Q. Could robots.txt and llms.txt conflict with each other?
A. Not at all, the two files serve distinct purposes. robots.txt controls which content bots can or cannot access, while llms.txt acts as an AI-friendly directory, highlighting content that AI should prioritize for learning and indexing.
2️⃣
Q. Should I prepare an llms.txt file even if AI crawlers don’t officially support it yet?
A. Yes, it’s wise to start early. By implementing llms.txt now, your website will be ahead of the curve and gain a significant advantage once AI crawlers adopt it as a standard.
3️⃣
Q. Can I set up llms.txt in Inblog?
A. While this feature isn’t currently available, our team is actively researching it and positively considering its implementation. We plan to provide an update in the near future, stay tuned !
4️⃣
Q. What's the difference between llms.txt and llms-full.txt?
A. The llms.txt file is a concise markdown-based index located at your website’s root directory, guiding AI (LLMs) directly to your most essential URLs and summaries of key content.
In contrast, llms-full.txt includes the full, detailed content (articles, examples, code snippets, etc.) from those URLs within a single file, allowing AI to learn everything it needs immediately, without further navigation.
Using both together provides optimal results, llms.txt gives AI an instant overview, while llms-full.txt supplies comprehensive details, enabling deeper and more effective AI understanding of your site.
[References]
