llms.txt vs Robots.txt: The New Standard for AI Crawler Control

Robots.txt has guarded the web’s front door since 1994. The bots knocking in 2026 include GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google’s own crawlers, each with a different job, and many site owners now ask whether they also need an llms.txt file. The honest answer has changed since this guide first went up in June: Google has since said its Search ignores llms.txt. It’s also one of the first questions I get when a business books a technical SEO service. So this guide is now built around what actually controls AI crawlers and AI features: robots.txt user agents, Google-Extended, snippet controls and Search Console’s AI opt-out, with llms.txt where it genuinely helps.

What is robots.txt built for?

Robots.txt tells crawlers which URLs they may fetch. It’s the Robots Exclusion Protocol, first defined in 1994 and published by the IETF as RFC 9309 in 2022. It controls access and nothing more: it can’t tell a crawler what your site is about, and it can’t force a badly behaved bot to obey.

That minimal interface was enough for Googlebot, and it’s still the main lever for AI crawlers, because every major AI company reads it. But it’s non-binding. The standard itself says “These rules are not a form of access authorization.” Polite crawlers obey; less polite ones don’t.

It’s also not a way to hide a page from Google. Google’s own guide warns that if other pages link to a blocked URL, Google “could still index the URL without visiting the page”. To keep a page out of results, use noindex or password protection.

Why was llms.txt created?

The llms.txt format is a proposal from Jeremy Howard, first published in September 2024, for a markdown file that gives AI tools a short summary of a site and links to its most useful pages. It was designed to help language models and agents read a site at the moment they need it, not to influence search rankings.

Think of it as a curated table of contents written for an LLM. It isn’t a replacement for robots.txt, and it doesn’t block anything. Its author released a second version in August 2026, saying thousands of sites now publish one and documentation platforms generate it automatically. It can point AI tools to the handful of pages that best explain your brand, but treat it as optional.

Does Google use llms.txt?

No. Google’s official guide to its generative AI features says Google Search doesn’t use llms.txt or other “special” AI files, and that creating one will neither help nor harm your visibility. You can keep one for other tools if you like, but it has no effect on Google rankings, AI Overviews or AI Mode.

It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.

Google Search Central, “Optimizing your website for generative AI features on Google Search”

Independent data points the same way. SE Ranking analysed nearly 300,000 domains and found an llms.txt on just 10.13% of them, with no correlation between having one and how often a domain is cited by AI tools. That fits Google’s wider message that AI search optimisation is still SEO, which we unpack in what generative engine optimisation (GEO) really involves.

There is one wrinkle. Chrome’s Lighthouse has an experimental Agentic Browsing category that checks for an llms.txt file. If you don’t have one, the audit is simply marked not applicable because, in Chrome’s words, “providing the file is optional at the moment”. That’s about helping browser agents use a site, not about rankings or citations.

What actually controls AI crawlers?

Robots.txt rules written for each bot by name. OpenAI and Anthropic each publish separate crawlers for model training, for their search products and for fetches a user triggers; Perplexity has a search crawler and a user fetcher. Each can be allowed or blocked on its own. Google adds the Google-Extended token and a Search Console setting for its AI features.

• OpenAI: GPTBot crawls for training, OAI-SearchBot surfaces sites in ChatGPT search and ChatGPT-User fetches pages when someone asks. OpenAI says “Each setting is independent of the others”, so you can block GPTBot and still appear in ChatGPT search. Because ChatGPT-User acts on a user’s request, robots.txt rules may not apply to it.
• Anthropic: ClaudeBot collects content that may be used for training, Claude-SearchBot indexes content for Claude’s search results and Claude-User fetches pages for users. Anthropic says its bots honour robots.txt.
• Perplexity: PerplexityBot surfaces and links sites in Perplexity and isn’t used to train foundation models, while Perplexity-User handles user requests and “generally ignores robots.txt rules”. A firewall may need to allow its bots explicitly.
• Google: Google-Extended is a robots.txt token, not a separate crawler. It controls whether your content trains Gemini models and grounds answers in Gemini Apps and Vertex AI, and it doesn’t affect inclusion or ranking in Google Search.

Here’s an example for a site that wants to stay visible in AI search while opting out of model training. A rule for one bot doesn’t apply to the others.

# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

llms.txt vs robots.txt at a glance

How do you control Google’s AI Overviews and AI Mode?

Not with AI-specific robots.txt tokens. Google’s AI features use the normal Search index, so the page-level controls are the ones you already know: noindex, nosnippet, data-nosnippet and max-snippet. Since 31 August 2026 there’s also a Search Console setting that removes your whole site from AI Overviews and AI Mode.

Google’s documentation on AI features says robots.txt rules for Googlebot manage crawling for Search, while nosnippet, data-nosnippet, max-snippet and noindex limit what’s shown from your pages. Those controls affect regular results too, so use them on specific pages or sections.

The site-wide option sits in Search Console under Settings, then Search generative AI. Google’s help page explains that “Include” is the default, that excluded sites’ links and content won’t appear in these features, that the setting doesn’t affect AI training (Google-Extended does that) and that exclusion usually takes effect within one to two days.

Think hard before opting out. Google says the setting isn’t used as a ranking signal elsewhere, but AI Overviews reach over 2.5 billion monthly users, and other sites’ content fills the gap you leave. Most businesses are better off staying in and watching the Generative AI performance report; our monthly Search Console routine shows where to find it.

What are the most common AI crawler mistakes on Indian business sites?

Most problems are self-inflicted: a staging site’s “Disallow: /” that went live, a firewall or CDN rule that quietly blocks AI search bots, and blanket blocks on every AI crawler when the owner only meant to stop training. Any of these can make a business invisible in ChatGPT or Perplexity without anyone noticing.

Start with the obvious one. If a website design company in India built or rebuilt your site, check that the staging rule didn’t go live with it; it’s one of the most common faults I find in audits. Then check your CDN and hosting firewall, because bot-protection settings can block AI crawlers even when robots.txt allows them.

The other classic mistake is confusing training with search. Blocking GPTBot only stops training; OAI-SearchBot decides whether you appear in ChatGPT search. A good AI SEO service checks this access layer before touching any content, because no amount of good writing helps if the right bots can’t reach your pages.

Is an llms.txt file worth adding?

Only as a low-priority extra. It won’t help Google rankings or AI Overviews, and there’s no evidence it lifts citations elsewhere. It can help if you publish developer documentation or a product that AI agents use. For most business sites, getting robots.txt right and publishing genuinely useful pages matters far more.

If you do publish one, follow the format: an H1 with your name, a short summary in a blockquote, then H2 sections listing your key pages with a one-line note each, and an optional section for secondary links. Keep marketing copy out. Here’s a minimal example:

# Example Physio Clinic
> Physiotherapy clinic in Faridabad offering sports injury rehab, back pain treatment and home visits across Delhi NCR.

## Key pages
- [Services](https://example.com/services/): treatments, prices and session lengths
- [Location](https://example.com/contact/): address, hours and areas served
- [Team](https://example.com/team/): physiotherapists and qualifications

## Optional
- [Blog](https://example.com/blog/): recovery guides and exercise videos

For an Indian business site, an llms.txt won’t help you stand out in Google, so keep it short: who you are, where you’re based, your services and your key pages. We keep one on our own site, as a small experiment rather than a ranking tactic.

How do you deploy and test your setup?

Put robots.txt at the root of every host you run, check it with Search Console’s robots.txt report, then watch your server logs for the bots you care about. Rules apply only to the host, protocol and port where the file sits, so each subdomain needs its own file.

Google’s robots.txt specification is explicit that the rules “apply only to the host, protocol, and port number where the robots.txt file is hosted”, so blog.example.com and shop.example.com each need their own. After a change, allow about a day (OpenAI and Perplexity both say changes can take around 24 hours), then look for the AI bots in your logs. Google-Extended won’t show up there, because it’s a token, not a crawler. Our technical SEO audit checklist covers the rest of the crawl checks.

What should you do this week?

Spend an hour on access before you spend a day on content. Many of the AI visibility problems I see start with a crawler that can’t reach the site, not with the writing. These steps cover robots.txt, Google’s controls and the optional llms.txt in the right order.

1. Open yourdomain.com/robots.txt and list every AI user agent it mentions, plus any wildcard rules that catch them.
2. Decide your policy by purpose: training (GPTBot, ClaudeBot, Google-Extended), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user requests.
3. Fix accidental blocks: leftover staging rules, CDN or firewall bot settings and blanket “block all AI” rules.
4. Check that important pages don’t carry noindex or nosnippet unless you mean it.
5. In Search Console, confirm the Search generative AI setting is on Include, unless you’ve chosen otherwise, and review the Generative AI performance report.
6. Add an llms.txt only if you publish documentation or expect AI agents to use your site.
7. Check your server logs a few days later to confirm the bots you want are getting through.

If you’d rather have someone look at this with you, a free SEO audit is a quick way to spot crawl and indexing problems.

The verdict: is llms.txt the new standard for AI crawler control?

No. Robots.txt is still the standard that controls AI crawlers, and in 2026 it does more work than ever, because the major AI companies now publish separate bots for search and for user requests, and most add a separate one for training. Google adds Google-Extended, snippet controls and the Search Console opt-out.

llms.txt is a harmless, optional extra for documentation and agents, not a ranking lever. Get robots.txt right first: it’s one of the checks I run every week as an SEO consultant in India, and it takes far less time than recovering the visibility you lose by getting it wrong. For how Google’s AI features choose their sources, read how Google AI Mode picks the sources it cites.

Frequently Asked Questions