If you care about ai website optimization, your website can’t just be readable by humans—it also needs to be easy for AI assistants to understand quickly and accurately. The challenge is that most websites are too large (and too messy) to fit cleanly into an AI model’s limited context window, and converting full HTML pages into useful plain text is often imprecise (Howard, 2024).
That’s where /llms.txt comes in: it’s a proposed standard for a single, curated file that gives LLMs a clear overview of your site and direct links to the most important supporting documents (Howard, 2024).
What is llms.txt (and what it’s for)
The proposal is simple: publish a Markdown file at /llms.txt that provides concise background info and a structured list of links to deeper resources (Howard, 2024). Think of it as a “smart table of contents” for AI.
The goal isn’t to replace sitemap.xml or robots.txt. A sitemap is usually too big and uncurated for an LLM context window, and robots.txt is about access—not understanding (Howard, 2024).
The llms.txt format (the spec you should follow)
According to the llms.txt proposal, the file is Markdown and follows a specific order (Howard, 2024):
- H1 title with your site/project name (this is the only required part).
- A blockquote summary with the short “what is this site?” description.
- Optional supporting text (paragraphs/lists are okay, but no headings in this section).
- Zero or more H2 sections that contain “file lists” (curated URLs to key docs).
- Optionally include an H2 section titled “Optional” for secondary links that can be skipped if a shorter context is needed (Howard, 2024).
What makes an llms.txt “perfect” for AI website optimization
1) It’s curated (not a dump of every URL)
LLMs benefit most from a small set of “highest-signal” pages—like your core services, pricing approach, about page, contact, and policies—rather than hundreds of blog posts and archive pages. The point is to make it easy for an assistant to grab the best context fast (Howard, 2024).
2) It uses clean Markdown versions of key pages
The proposal also recommends offering a clean Markdown version of important pages at the same URL with .md appended (and for URLs without filenames, using index.html.md) (Howard, 2024).
For example:
- https://example.com/services/web-design/ → https://example.com/services/web-design/index.html.md
- https://example.com/pricing → https://example.com/pricing.md
This supports ai website optimization because the AI gets the same content without the distractions (navigation, scripts, ads, layout noise) that make HTML harder to interpret (Howard, 2024).
3) It explains your “rules of interpretation”
The “supporting text” section (the part after the blockquote and before any H2 headings) is where you can spell out things that prevent wrong answers—especially for service businesses.
Useful examples:
- Which geographic areas you serve (and which you don’t)
- Preferred contact method
- How your pricing works (fixed, hourly, starting ranges, by scope)
- What you mean by key terms (e.g., “custom” vs. “template”)
- Any constraints (minimum project size, lead time, industries you specialize in)
Keep it clear, concise, and free of unexplained jargon (Howard, 2024).
A step-by-step recipe you can copy
Step 1: Choose your “core set” of pages
Pick 5–15 URLs that represent how you want your business explained:
- Services (1 page per primary service)
- Locations / service area
- About / team / story
- Work / case studies (select highlights only)
- Contact
- Policies (returns, shipping, refunds, privacy—whatever applies)
- FAQs
Step 2: Create the .md versions (clean, readable, accurate)
Your Markdown pages should be easy to skim and “answer-ready.” Avoid huge navigation blocks, repeated CTAs, or auto-inserted related-post grids. The goal is clarity and signal density (Howard, 2024).
Step 3: Write /llms.txt in the right structure
Use the exact section order the spec describes (Howard, 2024). A practical structure for most businesses:
- ## Start Here (best overview pages)
- ## Services (your main money pages)
- ## Proof (a few case studies / testimonials pages)
- ## Policies (the pages that prevent mistakes)
- ## Optional (secondary reading)
Step 4: Add short notes to every link
In the spec, each list item is a Markdown link with an optional “: description” after it (Howard, 2024). Those descriptions matter—write them like you’re helping an assistant pick the right source instantly.
Step 5: Test it like an AI would use it
The proposal recommends expanding your llms.txt into an “LLM context file” and testing multiple models to see if they can answer questions correctly (Howard, 2024). In plain English: try real prompts your customers ask, and see if the assistant answers using the correct page.
If answers are wrong, your fixes are usually:
- Make your descriptions clearer
- Promote the right page into the non-Optional section
- Rewrite the Markdown pages to be more direct and less marketing-fluffy
Common mistakes to avoid
- Turning llms.txt into a full sitemap: too many links defeats the purpose.
- Skipping the blockquote summary: you lose the “one-glance” meaning of the site (Howard, 2024).
- Linking only to HTML: if you can provide .md versions, do it (Howard, 2024).
- Hiding critical facts in Optional: if it’s essential (service area, what you do, pricing rules), keep it in the main sections.
References
Howard, J. (2024, September 3). The /llms.txt file. llms-txt. Retrieved December 31, 2025, from https://llmstxt.org/
