Imagine a Malaysian digital tools company with public setup guides and API documentation. Someone suggests adding /llms.txt so AI assistants can find the right instructions. The useful question is whether a named agent actually needs that file—or whether the guides themselves first need repair.
What Is llms.txt?
llms.txt is a proposed, Markdown-formatted file that gives an AI agent a short map of a website and links to important material. A site can place it at the root, such as https://example.com/llms.txt, or in a subpath covering a documentation section. The current proposal asks for a project name as a top-level heading and suggests a brief description and grouped links to more detailed pages.
Think of it as a curated reading guide, not a place to paste the whole website. The idea is that an agent which chooses to consult the file can follow the most relevant links. Whether a particular product does so is a separate question.
What It Cannot Do
An llms.txt file cannot tell a search engine which pages to index, grant a crawler permission, or force an AI answer to cite your site. robots.txt is for crawler access rules; a sitemap helps search engines discover URLs; the proposed llms.txt format offers context and selected links. One does not replace the others.
Google explicitly says Google Search does not use llms.txt for visibility or ranking, including its generative AI features. Google may still crawl or index a text file; finding it in an index does not mean Search treats it as a special AI signal. Chrome's Lighthouse has an agentic-browsing audit for the file, but its own documentation marks a missing file as not applicable because the file is optional. That audit is not evidence of a Google Search ranking benefit.
When Might It Be Worthwhile?
A documentation-heavy site may have a practical use case. If customers or coding agents need to distinguish setup instructions, API references, versioned examples and policy pages, a maintained index might help an agent that chooses to read it navigate. A small local service site with only a few clear pages is unlikely to gain much from another file. First make those pages accurate, accessible and linked together.
Do not assume that because one documentation platform or agent reads the file, every AI search system will. The decision should depend on a real use case you can test, not a promise of extra citations. Our AI search readiness guide covers the more basic checks for public pages.
How to Make a Small, Useful File
Start with an H1 naming the site or documentation project. Add a concise summary and a few H2 groups with descriptive Markdown links. The proposal requires the H1; the description and link groups make the file useful. For the digital tools company, the groups might be “Getting started,” “API reference” and “Current policies.” Link to public pages that actually answer those tasks. If you offer accurate Markdown versions, you can link them; do not create stale copies just to fill the file.
# Example Tools
> Public documentation for a fictional digital product.
## Getting started
- [Setup guide](https://example.com/docs/setup): First installation steps
## Reference
- [API reference](https://example.com/docs/api): Current endpointsThis is a format example, not a file for a real company. Keep descriptions factual. Do not include private dashboards, unreleased features, credentials or pages you do not want public. If the linked guides change, update the file.
Check It Before Publishing
Open the exact /llms.txt URL in a browser and confirm it returns the intended plain-text content rather than a server error or HTML homepage. Follow each link, check that it reaches the intended current page, and compare the summary with the published content. If you provide Markdown versions of pages, verify those separately; they are an additional part of the proposal, not a requirement for every site.
Record which agent or workflow is supposed to use the file. Test whether that workflow actually retrieves the relevant source; a public URL alone does not prove anyone reads it. One test answer is not evidence of a stable citation pattern.
Prioritize the Basics First
For Google Search, Google recommends accessible, indexable pages, useful content, internal links and accurate information. For ChatGPT search, OpenAI's published crawler guidance discusses robots.txt controls for OAI-SearchBot; it does not state that llms.txt is required. Those are different product policies, so do not turn either statement into a claim about every AI system.
If a customer cannot find the right setup guide on your site, fix navigation and the guide before adding a new machine-readable index. The site architecture guide explains how to connect important pages for people and crawlers.
Frequently Asked Questions
Is llms.txt Mandatory?
No. It is an emerging, optional convention. Google Search says it does not use the file for rankings or AI Search visibility, and Chrome's Lighthouse treats a missing file as not applicable.
How Do You Create an llms.txt File?
Create a plain-text Markdown file named llms.txt at the site root or an applicable subpath. Begin with an H1 for the project, then add a short summary and a curated list of valid public links. Publish it at the corresponding URL and maintain it when pages change.
What Is the Purpose of llms.txt?
Its proposed purpose is to help agents that choose to read it find a site's important documentation or other public pages. It does not itself control crawling, indexing, ranking or citations.