llms.txt Setup Guide — Help AI Systems Understand Your Site

The llms.txt convention proposes a machine-readable index at your site root—similar in spirit to robots.txt and sitemap.xml—that tells AI systems which pages represent your canonical documentation, policies, and product information. It reduces noise from navigation chrome, tag archives, and duplicate URLs when models select sources to summarize or cite.

What llms.txt Contains

A typical llms.txt file includes a brief site description, optional contact or licensing notes, and a curated list of URLs grouped by purpose. Each entry may have a title and short annotation explaining why the page matters. The format is plain Markdown-like text, human-editable and easy to generate from a CMS or build script.

Example structure:

# Your Product Name
> Enterprise SEO crawling and GEO analytics platform.

## Docs
- [Getting Started](https://example.com/docs/start): Account setup and first crawl.
- [API Reference](https://example.com/docs/api): REST endpoints and authentication.

## Policies
- [Privacy Policy](https://example.com/privacy)

Prioritize stable, evergreen URLs over campaign landing pages that expire quarterly.

Where and How to Publish

Place the file at https://yourdomain.com/llms.txt with Content-Type: text/plain; charset=utf-8. Serve it over HTTPS with the same cache headers as other small static assets—short TTL if you update frequently, longer if the list changes rarely. Link to it from your footer or /ai information page so human visitors and vendor documentation can discover it.

Some teams also publish an expanded llms-full.txt with richer context or chunked documentation exports for offline ingestion—keep the root llms.txt concise and point to the full variant if needed.

Maintenance Checklist

• Update within one week of major product or docs restructures.
• Remove URLs that 404 or redirect to unrelated destinations.
• Align listed pages with your public sitemap priorities.
• Version significant changes in your changelog for internal tracking.

Relationship to robots.txt and AI Crawlers

llms.txt is advisory, not enforceable. Crawler access still depends on robots.txt, terms of service, and individual bot behavior. OpenAI, Anthropic, Google, and others operate distinct user-agents—review your robots policy consciously rather than blocking all AI crawlers by default without a strategy.

If you block GPTBot but publish llms.txt, you are signaling intent to human partners while limiting automated ingestion— a valid choice for sites concerned about training use vs retrieval use. Document the policy internally so marketing and legal align.

Automating Generation

In Next.js or similar frameworks, add a Route Handler or static file generated at build time from your docs manifest. Pull titles and descriptions from frontmatter. Exclude authenticated routes, A/B test variants, and localized duplicates unless each locale maintains its own llms.txt under language prefixes.

Validate the output in CI: every URL must return 200, titles must be non-empty, and total file size should stay under practical limits (many teams target under 50 KB) to encourage full ingestion in context-limited fetches.

Measuring Effectiveness

Direct attribution from llms.txt is still emerging. Proxy metrics include increased AI citation of listed URLs, reduced hallucinated product capabilities after publication, and inbound requests from AI vendors verifying site structure. Pair llms.txt with strong on-page content and schema for best results—it is a signpost, not a substitute for quality.