The AI-Readable Web: A Practical Checkup
What actually makes a website readable by crawlers, LLMs, and AI agents? and which parts are just conventions pretending to be standards?
For most of the web’s history, designing a website meant designing for two audiences: Humans saw the interface, search engines saw the document structure underneath it.
AI systems crawl websites, retrieve pages on behalf of users, extract information from them, summarize them, cite them, and increasingly attempt to interact with their interfaces. This has produced a new category of advice: add an llms.txt file, publish a Markdown version of every page, insert special instructions for language models, optimize your copy for AI, expose new metadata, and make sure every crawler can get in.
Some of these ideas are useful, some are experiments, some are simply old web standards with a new AI label attached on them, and some solve a problem we are not sure exists.
So instead of asking “How do I optimize my website for AI?”, there is a more useful question: What does an AI system actually need from a website? As of September 2026, the answer is becoming surprisingly clear.
Start with the visitor, not the AI
The phrase AI crawler has become almost meaningless. An automated request from an AI company can represent several completely different intentions. OpenAI, for example, currently distinguishes between OAI-SearchBot, which is used to surface websites in ChatGPT search results; GPTBot, which crawls content that may be used to improve its generative models; and ChatGPT-User, which can visit a webpage because a person explicitly asked ChatGPT to access it.
Anthropic makes a similar distinction between Claude-SearchBot, ClaudeBot, and Claude-User. That distinction matters.
- Search Crawler: Discover and index information (OAI-SearchBot, Claude-SearchBot).
- Training Crawler: Collect material that may be used for model development (GPTBot, ClaudeBot).
- User-triggered Agent: Retrieve something because a person asked for it (ChatGPT-User, Claude-User).
Allowing one does not necessarily mean allowing the others. You might want your articles to appear in an AI search product without permitting the same content to be collected for model training. Or you might block automated crawling while still allowing an assistant to open a page when a user explicitly asks it to.
Before making a website “AI-readable”, the first check is therefore almost boring:
Check your robots.txt
Not because robots.txt is new, but because the new ecosystem makes its old job more important.
A website should have an intentional policy rather than accidentally blocking everything or blindly allowing everything. AI visibility and AI training are not the same thing.
For example, this robots.txt configuration blocks crawlers used for model training by OpenAI while still allowing search crawlers to index the site and surface its content in AI-powered search results:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Can a machine understand the page you already have?
A lot of AI optimization advice starts by basically creating a second version of a website. That might be unnecessary.
Before generating Markdown, special manifests or agent-specific endpoints, look at the document you are already serving.
- Is the content present in the HTML?
- Does the page have a meaningful heading structure?
- Are links actually links?
- Are buttons labelled (or contain some sort of text)?
- Can navigation, article content and supplementary information be distinguished from each other?
- Does important information exist only after a complicated sequence of client-side interactions?
For years, these questions belonged to accessibility, progressive enhancement, SEO and basic frontend engineering. They increasingly matter to agents too.
Chrome’s experimental agentic browsing audits in Lighthouse explicitly describe the accessibility tree as a primary data model used by agents. Its checks include programmatic names, valid roles, relationships between elements and whether interactive content is actually represented in that tree.
This produces an interesting convergence. A well-structured page is easier for a screen reader to navigate. It is easier for a search engine to interpret. And it is increasingly easier for an agent to operate.
There is a temptation to think that AI needs a special version of the web. In many cases, it may simply need the web to be implemented properly.
The first layer of AI readability is still semantic HTML.
What about JSON-LD?
If semantic HTML describes the structure of a document, structured data can make some of its meaning more explicit. This is where JSON-LD comes in.
A page can say, in machine-readable terms, that it is an Article, who wrote it, when it was published, which organization published it, and what its canonical URL is. For example:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "The AI-Readable Web: A Practical Checkup",
"description": "What actually makes a website readable by crawlers, LLMs, and AI agents.",
"datePublished": "2026-09-23",
"dateModified": "2026-09-23",
"author": {
"@type": "Person",
"name": "Lorenzo Fiori"
},
"publisher": {
"@type": "Organization",
"name": "Typable Studio"
},
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://typable.studio/..."
}
}
</script>
Unlike llms.txt, JSON-LD is not an emerging AI convention. It is an established way of expressing structured data on the web, commonly using the Schema.org vocabulary. Google supports JSON-LD for structured data and recommends it as a format for eligible search features.
But that does not mean adding more JSON-LD makes a website more visible to LLMs. Google explicitly says that structured data is not required for its generative AI search experiences and that there is no special Schema.org markup that needs to be added for them.
That distinction is useful. JSON-LD can make entities and relationships explicit:
This page
↓
is an Article
↓
written by a Person
↓
published by an Organization
↓
about a particular subject
An AI system may be able to make use of that information, just as search engines and other software can. But there is currently little reason to treat Schema.org as an “LLM optimization layer”, and no need to invent properties purely for AI consumption.
The best use of structured data is much less exciting: Describe things that are actually true about the page, using a shared vocabulary that machines already understand.
For an editorial site, that might mean Article or BlogPosting, Person, Organization, BreadcrumbList, images, publication dates, and canonical relationships. It should also stay consistent with the content a person can actually see.
JSON-LD should clarify the page, not tell machines a different story.
This makes it another good example of the broader pattern we keep finding: the most useful “AI optimizations” are often technologies that were useful before generative AI appeared.
Google supports JSON-LD, Microdata and RDFa for structured data and currently recommends JSON-LD where possible. Its Article documentation uses structured data to communicate information such as the headline, images, dates and authorship.
Google’s current guidance for generative AI search says structured data is not required for those experiences and that no special Schema.org markup is needed specifically for them. Structured data remains useful as part of normal Search optimization and eligibility for supported search features.
Look at the page through four different lenses
A useful check is to stop thinking of a webpage as one thing. The same URL can effectively have several representations:
- The rendered interface: This is what a person sees: typography, spacing, images, motion, color, hierarchy.
- The DOM: This is the structural document underneath the presentation.
- The accessibility tree: This removes much of the implementation detail and exposes roles, names, states and relationships that assistive technologies, and increasingly agents, can reason about.
- A text representation: This might be extracted article text, Markdown, reader mode or another simplified representation designed primarily around content.
These representations overlap, but they are not identical. A beautifully designed page can have an incomprehensible DOM. A complicated DOM can produce an excellent accessibility tree. A Markdown conversion can remove thousands of tokens of navigation and interface chrome, but it can also discard relationships that existed in the original document.
So the goal should not be to find the representation that AI prefers. It should be to make sure the important information survives across them. That is a much more useful standard for quality.
Do you need a Markdown version?
Maybe. Markdown has obvious advantages for language-model clients. It is compact, its structural syntax is explicit, and it generally contains less presentation-related noise than production HTML.
This can mean fewer tokens to process and a cleaner separation between content and interface. The important development in 2026 is that serving Markdown no longer has to mean inventing an AI-specific URL scheme.
HTTP already has a mechanism for asking a server which representation a client prefers. Content negotiation.
A client can send:
GET /articles/example
Accept: text/markdown, text/html;q=0.8
and the server can respond with:
HTTP/2 200
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Same resource. Same information. Different representation.
This is not an AI protocol. It is simply HTTP.
Cloudflare’s Markdown for Agents, launched in 2026, now implements exactly this pattern: when enabled, a client requesting text/markdown can receive an automatically converted Markdown representation of an HTML page.
That is interesting for a reason that has very little to do with Markdown itself. The client does not need to know that your site has a special /ai/ endpoint. It does not have to guess whether adding .md to a URL will work.
It can express a preference using a standard HTTP request header and let the server decide what it can provide. That feels considerably more durable than detecting a bot’s User-Agent and serving it something different.
Markdown should be a representation, not a second website
There is one easy way to make this architecture bad: maintain the Markdown manually.
Imagine an article existing in HTML and Markdown as two independent documents. You correct a number in the HTML version. You forget the Markdown version. An agent retrieves the old information. You have successfully made the website more machine-readable and less reliable.
The better model is:
┌── HTML
Content source ─────┼── Markdown
└── other representations
One source of truth. Multiple outputs.
If your CMS stores structured content, this is relatively straightforward. The HTML template can remain expressive and visual while the Markdown renderer focuses on text hierarchy, links, code and media descriptions.
In our case, we use Payload CMS, which gives us the option to render rich text as either HTML or Markdown. However, we do not consider this necessary, since the HTML page markup is already very clean and well structured by default.
A static .md URL can still be useful, especially for documentation and developer-oriented content. But the architecture matters more than the suffix.
The point is not to create a secret AI website. The point is to offer another representation of the same resource.
What about llms.txt?
Few files have generated as much discussion relative to their size.
The original llms.txt proposal appeared in 2024 as a way for websites to provide language models with a concise, Markdown-formatted overview of their content.
In August 2026, the proposal reached version 2.
One of its biggest improvements addresses discoverability. The updated proposal recommends established link relations so pages can advertise their Markdown representation and the relevant llms.txt document:
<link
rel="alternate"
type="text/markdown"
href="/articles/example.md"
/>
<link
rel="describedby"
href="/llms.txt"
/>
This makes the system more coherent. But there is an important distinction to make: llms.txt is a convention, not a web standard.
Chrome’s own Lighthouse documentation currently calls it an emerging convention.
Google is more explicit. Its guidance for generative AI search says Google Search does not use llms.txt, and that having one will neither improve nor reduce a site’s visibility or ranking in Google Search, including its generative AI experiences.
That does not make the file useless. Documentation tools, coding agents and other AI clients may use it. It can act as a compact orientation document for a complex site. And at very low implementation cost, there is little harm in providing a good one.
But it should be understood for what it is: a convenient interface, not an AI ranking signal. A website with excellent content, crawlable HTML and no llms.txt is not suddenly invisible to AI. And a bad website does not become good because someone added one.
llms.txt should probably be boring
If you decide to add one, resist the urge to turn it into an encyclopedia. A useful llms.txt can describe the site, identify its important areas and link to canonical resources. It does not need to reproduce everything.
# Typable Studio
> Independent digital design and development studio.
## Main sections
- [Services](https://...)
- [Projects](https://...)
- [Articles](https://...)
## Documentation / resources
- [AI-readable web](https://...)
There is also a related convention called llms-full.txt, which puts much larger amounts of content into a single file. It has seen use in documentation ecosystems, but it is not part of the llms.txt proposal itself.
For a very large API reference, packaging documentation for direct ingestion can make sense. For a design studio, magazine, portfolio or typical marketing website, concatenating the entire site into one enormous text file is harder to justify.
The more interesting question is not “How much can we give the model?” It is: Can the model reliably find the right resource when it needs it? Good information architecture still beats information dumping.
Avoid invisible messages written only for AI
One emerging pattern has been to place hidden text inside a page telling an AI where to find a cleaner version:
<div class="visually-hidden" aria-hidden="true">
A Markdown version of this page is available at /article.md.
</div>
It is clever. It is also a good example of why AI-specific hacks can age quickly. If an agent extracts raw page text, the message might work.
If it reasons through the accessibility tree, aria-hidden=“true” explicitly removes that information from the representation the agent may be using.
Chrome’s current agentic browsing work makes this tension especially visible: accessibility semantics are becoming part of the machine-facing interface itself. Instead of hiding instructions inside content, use explicit relationships where possible:
<link
rel="alternate"
type="text/markdown"
href="/article.md"
/>
or the equivalent HTTP Link header. Machines should not need easter eggs.
Do not confuse content negotiation with cloaking
Serving Markdown and HTML from the same URL may initially sound suspicious. Are we showing one thing to humans and another thing to machines?
The distinction is in the content and the negotiation mechanism. If a server detects SomeAIBot/1.0 and silently sends that crawler a rewritten, keyword-stuffed article that humans never see, that is one thing.
If a client sends:
Accept: text/markdown
and receives the same article represented as Markdown, that is normal HTTP content negotiation.
The response should also include:
Vary: Accept
so caches know that the representation changes depending on the request’s Accept header.
There is another small detail worth getting right. If a client asks only for formats you cannot provide, returning:
406 Not Acceptable
is valid. But it is not mandatory.
RFC 9110 allows an origin server either to honor the preference and return 406, or to disregard the negotiation preference and provide a default representation.
This is a useful reminder that implementation details are often more nuanced than emerging “AI readiness” checklists suggest. The web already has rules for this. We should use them rather than invent stricter ones and then call them standards.
Search visibility and agent usability are becoming different problems
So far, everything has been about reading. But agents increasingly want to do things. Search a catalog. Submit a form. Add an item to a cart. Book an appointment. Change a reservation.
A human can infer these possibilities from visual interfaces. An agent has to identify the available actions, understand their parameters and execute them reliably.
This is where newer proposals such as WebMCP become interesting. Chrome is already experimenting with WebMCP integration in its agentic browsing tooling. A page can expose actions declaratively through HTML or imperatively through JavaScript, allowing an agent to discover capabilities such as book_appointment or add_to_cart rather than reverse-engineering every interaction from pixels and DOM events.
This changes the question. An AI-readable website exposes understandable information. An agent-ready website exposes understandable capabilities.
Those are related problems, but they are not the same one.
And if bots can act, identity becomes important
Once automated clients can perform meaningful actions, another old assumption starts to break.
A User-Agent string says:
I am SomeBot.
It does not prove it. Anyone can send the same string.
The emerging Web Bot Auth work at the IETF is trying to address that problem using HTTP Message Signatures.
The current working-group draft describes a protocol where automated HTTP clients can cryptographically sign their requests, giving servers a stronger mechanism for verifying the identity of automated traffic.
This is worth watching, but not deploying as though it were finished infrastructure.
As of September 2026, it is still an active Internet-Draft — explicitly work in progress, not a finalized RFC.
That distinction is important throughout this entire subject.
There are mature web standards.
There are conventions with growing adoption.
And there are experiments pointing toward what the web may become.
Treating all three as equivalent is how short-lived best practices get created.
The practical checkup
So what should a normal website actually do today?
Foundation
- Make the content crawlable.
- Review robots.txt intentionally.
- Distinguish search crawling from training crawling where providers support it.
- Use semantic HTML.
- Maintain a useful heading hierarchy.
- Give interactive elements proper names and labels.
- Make the accessibility tree meaningful.
- Avoid hiding essential content behind unnecessary client-side complexity.
- Keep your existing SEO foundations: canonical URLs, useful metadata, sitemaps and structured data where they genuinely apply.
Useful enhancements
- Provide Markdown representations when your content benefits from them.
- Generate HTML and Markdown from the same source.
- Advertise alternate representations with rel=“alternate”.
- Consider an llms.txt file as a lightweight site guide.
- Use rel=“describedby” where the llms.txt v2 convention makes sense.
- Consider Accept: text/markdown content negotiation if your infrastructure can support it cleanly.
- Log server-side traffic so you can see what automated clients actually request.
Probably not worth optimizing around
- Hidden messages written exclusively for LLMs.
- Invented AI tags.
- Rewriting otherwise good content into awkward “AI-friendly” prose.
- Treating llms.txt as a ranking hack.
- Maintaining separate AI content that can drift away from the human version.
- Creating enormous text dumps simply because an agent might ingest them.
Worth watching
- WebMCP and other mechanisms for exposing actions to agents.
- Web Bot Auth.
- Browser-level agent APIs.
- How major AI providers standardize discovery and content negotiation.
The difference between these categories matters more than any individual technique.