Is your website ready for AI search? robots.txt AI crawlers, llms.txt and structured data

To be cited by AI assistants, your site needs three things: the AI search crawlers must be allowed to read it, its pages must be indexed by ordinary search engines, and its key facts must be clear in the text and in structured data. Blocking AI training crawlers such as GPTBot or Google-Extended does not, by the vendors' own documentation, stop you being shown in ChatGPT search or Google's AI Overviews. llms.txt is an optional proposal, not a standard, and Google says you do not need it.

Training bots and search bots are different

The most common mistake is treating "AI bots" as one group. Each major vendor now separates crawling for model training from fetching for search and answers. You can block one and allow the other.

• Training crawlers collect pages that may be used to train future models. Blocking them is a choice about your content's use. It does not, on its own, remove you from AI answers.
• Search crawlers build the index an assistant searches when it answers. Block these and you are much less likely to be cited or linked.
• User-initiated fetchers visit a page because a person asked the assistant about it or pasted the link. Some vendors say robots.txt may not apply to these.
• Control tokens such as Google-Extended and Applebot-Extended are not separate crawlers. They are names you use in robots.txt to say how content collected by the main crawler may be used.

The main AI crawlers and what they do

Checked against each vendor's documentation on 3 November 2026. Vendors add and rename bots, so check the linked pages before you rely on this list.

OpenAI says that if you allow both OAI-SearchBot and GPTBot it may use one crawl for both purposes. OpenAI and Perplexity both say robots.txt changes can take around 24 hours to take effect.

Microsoft Copilot and other assistants that use Bing's index depend on Bingbot, so do not block it if you want to appear there.

Example robots.txt

robots.txt sits at the root of your domain, for example https://example.co.uk/robots.txt. Rules are grouped by User-agent. A crawler follows the most specific group that names it, and ignores the general * group if a specific one exists. The format is standardised in RFC 9309.

Option A: appear in AI answers, opt out of training

# AI search and answer bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.co.uk/sitemap.xml

Option B: allow everything (the default)

User-agent: *
Allow: /

Sitemap: https://example.co.uk/sitemap.xml

Which option to choose is a business decision. A local plumber or accountant who wants to be recommended will usually gain more from being visible than they lose from training use. A publisher whose content is the product may decide differently.

How to check your own site in five minutes

1. Open your robots.txt. Visit yourdomain/robots.txt. Look for Disallow: / under User-agent: *. That blocks every compliant crawler, including Googlebot. It is often left over from when the site was being built.
2. Search for AI bot names in the file and check each one matches your intent, using the table above.
3. Check your CDN or host. Some security services and hosting plans offer a setting to block AI crawlers at the firewall. That blocks them whatever robots.txt says, and robots.txt will not show it. Look in your CDN's bot or security settings.
4. Check page-level tags. View the source of a key page and search for noindex or nosnippet in a robots meta tag. Either will keep that page out of Google's AI features.
5. Check indexing. In Google Search Console, use URL Inspection on your homepage and main service pages. Pages Google has not indexed cannot be supporting links in AI Overviews. Bing Webmaster Tools does the same for Bing.
6. Run a SiteVerdict check. SiteVerdict reads your robots.txt, reports which of the main AI crawlers are allowed or blocked, checks for an llms.txt file and for structured data, and flags basic SEO gaps.

What llms.txt is, honestly

llms.txt was proposed by Jeremy Howard in September 2024. The site describes it as "a proposal to standardise on using an /llms.txt file" that gives AI tools a short, curated overview of a site in Markdown, so they do not have to read full web pages. The specification is open for community input.

The format, per llmstxt.org:

• A Markdown file at /llms.txt (or under a subpath, covering pages below it).
• An H1 with the name of the site or project. This is the only required part.
• A blockquote with a short summary.
• Optional paragraphs or lists with more detail.
• H2 sections listing links in the form [name](url): optional note.
• An optional section titled "Optional" for links that can be skipped when space is short.
• The proposal also suggests offering clean Markdown versions of pages at the same URL with .md added.

A minimal example for a local business:

# Smith & Sons Plumbing

> Gas Safe registered plumbers in Plymouth, Devon. Boiler servicing, repairs and bathroom installations. Open Monday to Saturday.

## Services
- [Boiler servicing](https://example.co.uk/boiler-servicing): annual service, prices and booking
- [Emergency repairs](https://example.co.uk/emergency): same-day call-outs in Plymouth

## Contact
- [Contact and opening hours](https://example.co.uk/contact)

## Optional
- [About us](https://example.co.uk/about)

Its real status: it is not a web standard, and no major search engine has said it uses llms.txt for ranking or citations. Google's guidance on AI features says you do not need "new machine readable files, AI text files, or markup" to appear in them. Some AI coding tools and developer documentation sites use llms.txt, which is where it is most useful today. For a small-business site it is a low-cost extra, not a priority. Clear pages, indexing and correct robots.txt matter far more.

Structured data that helps

Structured data is code, usually JSON-LD, that labels facts on your page using the shared schema.org vocabulary. Google recommends JSON-LD, says markup must describe content visible on the page, and says it does not guarantee rich results. It also says no special schema is required for its AI features. What structured data does is remove ambiguity: your business name, address, phone number, opening hours and services become machine-readable facts.

A minimal LocalBusiness example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Plumber",
  "name": "Smith & Sons Plumbing",
  "url": "https://example.co.uk/",
  "telephone": "+44 1752 000000",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "1 Example Street",
    "addressLocality": "Plymouth",
    "postalCode": "PL1 1AA",
    "addressCountry": "GB"
  },
  "openingHours": "Mo-Sa 08:00-18:00"
}
</script>

Test it with Google's Rich Results Test, and keep it in step with the visible page. If the phone number on the page changes, change it in the markup too.

The basics still matter most

AI assistants that search the web mostly cite pages that already do well in ordinary search. The same fundamentals apply:

• Clear, specific text. Say what you do, where, for whom and at what price range, in plain sentences near the top of the page. Assistants quote text; they cannot quote what is only in an image.
• A search description on every key page. In our checks of 32,933 UK small-business websites (October to November 2026), 29.4% had no meta description on the homepage. See how to fix a missing meta description.
• Fast, mobile-ready pages. 21.2% of the UK sites we checked were slow on mobile in Google's test. See why your website is slow on mobile.
• HTTPS. 6.2% of those UK sites had no HTTPS at all. See fixing the "Not secure" warning.
• Consistent business details across your site, Google Business Profile and directories, so the facts an assistant finds agree with each other.

Full method and figures: UK small business websites 2026.

What is likely to change

• Vendors add, rename and split crawlers. Re-check their documentation at least twice a year.
• Google says it plans to support a shared machine-readable AI training preference signal in early 2027. That may give one way to state training preferences to many vendors at once.
• llms.txt may gain or lose support. Watch for an official statement from a major search or AI provider before investing time in it.

Developers and agencies can run the same AI-readiness checks across many sites through the SiteVerdict API and MCP server, or from the Chrome extension and WordPress plugin.

Related guides

• Missing meta description: how to fix it
• Why your website is slow on mobile
• "Not secure" warning: how to fix it