A practical guide for SEO specialists wondering whether this new file format is worth their time.
The Short Version
llms.txt is a Markdown file that tells AI systems which pages on your site matter most. Think of it as a curated table of contents for ChatGPT and similar tools.
| File | Purpose | Used by |
|---|---|---|
| robots.txt | Controls crawler access | Search engines |
| sitemap.xml | Lists all pages | Search engines |
| llms.txt | Highlights key content | AI systems (theoretically) |
Quick verdict: Low-risk experiment. Takes under 30 minutes to set up. But no major AI platform has confirmed systematic use for ranking or citations. Research on 300,000 websites found zero correlation between having llms.txt and being cited by AI.
Note: Information current as of December 2025. This space evolves quickly.
What is llms.txt?
llms.txt is a plain text file written in Markdown. You place it at yourdomain.com/llms.txt. The file lists your most important pages with brief descriptions.
Jeremy Howard proposed the standard in September 2024. Howard co-founded fast.ai and Answer.AI. He’s a respected figure in AI research, which gave the proposal initial credibility.
His vision is ambitious. In March 2025, Howard stated: “99.9% of attention is about to be LLM attention, not human attention.” Whether this proves prescient or premature remains to be seen.
The file structure is simple. Start with an H1 heading containing your site name. Add a one-sentence summary in a blockquote. Then list important pages under H2 sections, each with a brief description. An “Optional” section signals content AI can skip if context is limited.
Why Does This Exist?
Traditional search engines crawl your entire website ahead of time. AI systems work differently.
Some AI experiences use retrieval — search, browsing, or indexes that pull fresh context when you ask a question. Others answer purely from training data. llms.txt targets the retrieval scenario: when an AI agent actually visits your site in real time.
Here’s the problem. Someone asks: “How do I set up Product X?” The AI fetches your homepage. Maybe your pricing page. But your setup guide is buried three clicks deep. The AI never finds it.
Even modern models with huge context windows face constraints. Processing large amounts of text is expensive. Cleaner inputs are preferred. Your site might have thousands of pages. The AI still has to decide which ones matter.
llms.txt offers a shortcut. Instead of guessing, AI could check your llms.txt first. The file points directly to valuable content.
That’s the theory. Reality is more complicated.
llms.txt vs robots.txt vs sitemap.xml
| File | What it does | Who reads it | Format |
|---|---|---|---|
| robots.txt | Tells crawlers what NOT to access | Search engine bots | Plain text |
| sitemap.xml | Lists ALL indexable pages | Search engine bots | XML |
| llms.txt | Highlights IMPORTANT pages | AI systems | Markdown |
robots.txt controls access. sitemap.xml ensures discovery. llms.txt provides prioritization.
They complement each other. llms.txt doesn’t replace the other two.
The Reality Check: Does Anyone Actually Use It?
The concept sounds reasonable. The data tells a different story.
What The Research Shows
SE Ranking analyzed 300,000 domains in November 2025. They found llms.txt on about 10% of sites. The critical finding: zero correlation between having llms.txt and being cited by AI systems. Their machine learning model actually improved when they removed llms.txt as a factor.
Rankability scanned the top 1,000 most-visited websites. They found llms.txt on just 3 sites — 0.3% adoption among major platforms.
What Server Logs Show
The research measured citations. But are AI crawlers even fetching these files?
Search Engine Land tested their own llms.txt from August to October 2025. GPTBot, ClaudeBot, and PerplexityBot made zero requests to the file. The major AI crawlers didn’t check if it existed.
A Reddit user managing 20,000 domains reported similar findings. Only niche bots like BuiltWith fetched llms.txt files. The major players ignored them.
Google’s Position
Google’s John Mueller has been consistently skeptical. In April 2025, he compared llms.txt to the keywords meta tag — the outdated HTML element search engines abandoned because it was too easy to manipulate.
In June 2025, Mueller wrote on Bluesky: “FWIW no AI system currently uses llms.txt.”
His logic: if AI already downloads your full page, why trust a separate file you control? You could write anything there. No verification exists.
But Then Google Did Something Interesting
In December 2025, Google quietly added llms.txt to their own Search Central documentation. The same team whose spokesperson spent months dismissing the standard.
When asked about the contradiction, Mueller responded with a cryptic: “hmmn :-/”
This doesn’t signal official adoption. But it suggests internal experimentation — or at minimum, hedging bets. When Google publicly dismisses something and then quietly implements it, pay attention.
Who Actually Uses It?
Adoption clusters in tech and documentation. Anthropic publishes llms.txt for Claude documentation. So do Cloudflare, Stripe, and Mintlify.
The pattern is clear: developer-focused documentation sees potential value. Mainstream websites don’t bother.
How to Create Your llms.txt File
Two approaches, depending on your setup.
Option A: WordPress With Yoast
Yoast SEO added llms.txt support in June 2025. Available in free and premium versions.
- Go to Yoast SEO → Settings
- Click Site features
- Find llms.txt under APIs
- Toggle on
- Save changes
Yoast generates the file automatically and refreshes it weekly.
Option B: Manual Creation
Create a file named llms.txt with this structure:
—
# Your Site Name
> One sentence describing your business
## Products
– [Main Product](/product): What it does and who it’s for
– [Pricing](/pricing): Plans and features
## Resources
– [Documentation](/docs): How to get started
– [FAQ](/faq): Common questions answered
## Optional
– [Blog](/blog): Company news and updates
—
Upload to your root directory. Test by visiting yourdomain.com/llms.txt.
Mistakes to Avoid
Too many links. Stick to 5-15 pages. This is curation, not a sitemap.
Missing descriptions. Each link should explain what the page contains.
Never updating. Review quarterly. Remove outdated pages.
Should You Implement It?
| Your Situation | Recommendation |
|---|---|
| Documentation site | Yes — this is where llms.txt sees actual use |
| Blog with Yoast | Yes — one toggle, under 5 minutes |
| E-commerce | Low priority — product schema matters more |
| Limited resources | Skip — focus on content quality first |
| Expecting quick results | No — current data shows no impact |
The pattern: if implementation takes under 10 minutes, do it. If it requires significant effort, prioritize proven tactics instead.
The Bottom Line
llms.txt might matter someday. The evidence says it doesn’t matter today.
No major AI platform has confirmed they use it for ranking or citations. Independent research found zero correlation with AI visibility. Server logs show major crawlers aren’t even checking for the file.
But implementation is trivial with modern tools. Yoast handles everything with one toggle. There’s no downside to trying.
What to actually do:
If you use Yoast: enable llms.txt. One click. No risk.
If you don’t use Yoast: only bother if you have 30 spare minutes.
If you’re choosing between llms.txt and improving your content: improve your content. Structured data, clear writing, authoritative information. Those fundamentals work regardless of which standards AI platforms eventually adopt.
Getting Started
WordPress: Yoast SEO → Settings → Site features → llms.txt → Enable.
Manual: Create Markdown file. List 5-10 key pages. Upload to root directory.
Check your server logs monthly for AI crawler activity. Adjust based on what you learn.
