What actually makes a website readable by crawlers, LLMs, and AI agents? and which parts are just conventions pretending to be standards?
For most of the web’s history, designing a website meant designing for two audiences: Humans saw the interface, search engines saw the document underneath it. Now there is a third audience.
AI systems crawl websites, retrieve pages on behalf of users, extract information from them, summarize them, cite them, and increasingly attempt to interact with their interfaces. This has produced a new category of advice: add an llms.txt file, publish a Markdown version of every page, insert special instructions for language models, optimize your copy for AI, expose new metadata, and make sure every crawler can get in.
Some of these ideas are useful. Some are experiments. Some are simply old web standards with a new AI label attached. And some solve a problem we are not sure exists.
So instead of asking “How do I optimize my website for AI?”, there is a more useful question: What does an AI system actually need from a website? As of September 2026, the answer is becoming surprisingly clear.
The phrase AI crawler has become almost meaningless. An automated request from an AI company can represent several completely different intentions. OpenAI, for example, currently distinguishes between OAI-SearchBot, which is used to surface websites in ChatGPT search results; GPTBot, which crawls content that may be used to improve its generative models; and ChatGPT-User, which can visit a webpage because a person explicitly asked ChatGPT to access it.
Anthropic makes a similar distinction between Claude-SearchBot, ClaudeBot, and Claude-User. That distinction matters.
Allowing one does not necessarily mean allowing the others. You might want your articles to appear in an AI search product without permitting the same content to be collected for model training. Or you might block automated crawling while still allowing an assistant to open a page when a user explicitly asks it to.
Before making a website “AI-readable”, the first check is therefore almost boring:
Not because robots.txt is new, but because the new ecosystem makes its old job more important.
A website should have an intentional policy rather than accidentally blocking everything or blindly allowing everything. AI visibility and AI training are not the same thing.
A lot of AI optimization advice starts by basically creating a second version of a website. That might be premature.
Before generating Markdown, special manifests or agent-specific endpoints, look at the document you are already serving.
For years, these questions belonged to accessibility, progressive enhancement, SEO and basic frontend engineering. They increasingly matter to agents too.
Chrome’s experimental agentic browsing audits in Lighthouse explicitly describe the accessibility tree as a primary data model used by agents. Its checks include programmatic names, valid roles, relationships between elements and whether interactive content is actually represented in that tree.
This produces an interesting convergence. A well-structured page is easier for a screen reader to navigate. It is easier for a search engine to interpret. And it is increasingly easier for an agent to operate.
There is a temptation to think that AI needs a special version of the web. In many cases, it may simply need the web to be implemented properly.
The first layer of AI readability is still semantic HTML.
If semantic HTML describes the structure of a document, structured data can make some of its meaning more explicit. This is where JSON-LD comes in.
A page can say, in machine-readable terms, that it is an Article, who wrote it, when it was published, which organization published it, and what its canonical URL is. For example:
Unlike llms.txt, JSON-LD is not an emerging AI convention. It is an established way of expressing structured data on the web, commonly using the Schema.org vocabulary. Google supports JSON-LD for structured data and recommends it as a format for eligible search features.
But that does not mean adding more JSON-LD makes a website more visible to LLMs. Google explicitly says that structured data is not required for its generative AI search experiences and that there is no special Schema.org markup that needs to be added for them.
That distinction is useful. JSON-LD can make entities and relationships explicit:
An AI system may be able to make use of that information, just as search engines and other software can. But there is currently little reason to treat Schema.org as an “LLM optimization layer”, and no need to invent properties purely for AI consumption.
The best use of structured data is much less exciting: Describe things that are actually true about the page, using a shared vocabulary that machines already understand.
For an editorial site, that might mean Article or BlogPosting, Person, Organization, BreadcrumbList, images, publication dates, and canonical relationships. It should also stay consistent with the content a person can actually see.
JSON-LD should clarify the page, not tell machines a different story.
This makes it another good example of the broader pattern we keep finding: the most useful “AI optimizations” are often technologies that were useful before generative AI appeared.
Google supports JSON-LD, Microdata and RDFa for structured data and currently recommends JSON-LD where possible. Its Article documentation uses structured data to communicate information such as the headline, images, dates and authorship.
Google’s current guidance for generative AI search says structured data is not required for those experiences and that no special Schema.org markup is needed specifically for them. Structured data remains useful as part of normal Search optimization and eligibility for supported search features.
A useful check is to stop thinking of a webpage as one thing. The same URL can effectively have several representations:
This is what a person sees: typography, spacing, images, motion, color, hierarchy.
This is the structural document underneath the presentation.
This removes much of the implementation detail and exposes roles, names, states and relationships that assistive technologies — and increasingly agents — can reason about.
This might be extracted article text, Markdown, reader mode or another simplified representation designed primarily around content.
These representations overlap, but they are not identical. A beautifully designed page can have an incomprehensible DOM. A complicated DOM can produce an excellent accessibility tree. A Markdown conversion can remove thousands of tokens of navigation and interface chrome, but it can also discard relationships that existed in the original document.
So the goal should not be to find the representation that AI prefers. It should be to make sure the important information survives across them. That is a much more useful standard for quality.
Maybe. Markdown has obvious advantages for language-model clients. It is compact, its structural syntax is explicit, and it generally contains less presentation-related noise than production HTML.
This can mean fewer tokens to process and a cleaner separation between content and interface. The important development in 2026 is that serving Markdown no longer has to mean inventing an AI-specific URL scheme.
HTTP already has a mechanism for asking a server which representation a client prefers. Content negotiation.
A client can send:
and the server can respond with:
Same resource. Same information. Different representation.
This is not an AI protocol. It is simply HTTP.
Cloudflare’s Markdown for Agents, launched in 2026, now implements exactly this pattern: when enabled, a client requesting text/markdown can receive an automatically converted Markdown representation of an HTML page.
That is interesting for a reason that has very little to do with Markdown itself. The client does not need to know that your site has a special /ai/ endpoint. It does not have to guess whether adding .md to a URL will work.
It can express a preference using a standard HTTP request header and let the server decide what it can provide. That feels considerably more durable than detecting a bot’s User-Agent and serving it something special.
There is one easy way to make this architecture bad: maintain the Markdown manually.
Imagine an article existing in HTML and Markdown as two independent documents. You correct a number in the HTML version. You forget the Markdown version. An agent retrieves the old information. You have successfully made the website more machine-readable and less reliable.
The better model is:
One source of truth. Multiple outputs.
If your CMS stores structured content, this is relatively straightforward. The HTML template can remain expressive and visual while the Markdown renderer focuses on text hierarchy, links, code and media descriptions.
A static .md URL can still be useful, especially for documentation and developer-oriented content. But the architecture matters more than the suffix.
The point is not to create a secret AI website. The point is to offer another representation of the same resource.
Few files have generated as much discussion relative to their size.
The original llms.txt proposal appeared in 2024 as a way for websites to provide language models with a concise, Markdown-formatted overview of their content.
In August 2026, the proposal reached version 2.
One of its biggest improvements addresses discoverability. The updated proposal recommends established link relations so pages can advertise their Markdown representation and the relevant llms.txt document:
This makes the system more coherent. But there is an important distinction to make: llms.txt is a convention, not a web standard.
Chrome’s own Lighthouse documentation currently calls it an emerging convention.
Google is more explicit. Its guidance for generative AI search says Google Search does not use llms.txt, and that having one will neither improve nor reduce a site’s visibility or ranking in Google Search, including its generative AI experiences.
That does not make the file useless. Documentation tools, coding agents and other AI clients may use it. It can act as a compact orientation document for a complex site. And at very low implementation cost, there is little harm in providing a good one.
But it should be understood for what it is: a convenient interface, not an AI ranking signal. A website with excellent content, crawlable HTML and no llms.txt is not suddenly invisible to AI. And a bad website does not become good because someone added one.
If you decide to add one, resist the urge to turn it into an encyclopedia. A useful llms.txt can describe the site, identify its important areas and link to canonical resources. It does not need to reproduce everything.
There is also a related convention called llms-full.txt, which puts much larger amounts of content into a single file. It has seen use in documentation ecosystems, but it is not part of the llms.txt proposal itself.
For a very large API reference, packaging documentation for direct ingestion can make sense. For a design studio, magazine, portfolio or typical marketing website, concatenating the entire site into one enormous text file is harder to justify.
The more interesting question is not “How much can we give the model?” It is: Can the model reliably find the right resource when it needs it? Good information architecture still beats information dumping.
One emerging pattern has been to place hidden text inside a page telling an AI where to find a cleaner version:
It is clever. It is also a good example of why AI-specific hacks can age quickly. If an agent extracts raw page text, the message might work.
If it reasons through the accessibility tree, aria-hidden="true" explicitly removes that information from the representation the agent may be using.
Chrome’s current agentic browsing work makes this tension especially visible: accessibility semantics are becoming part of the machine-facing interface itself. Instead of hiding instructions inside content, use explicit relationships where possible:
or the equivalent HTTP Link header. Machines should not need easter eggs.
Serving Markdown and HTML from the same URL may initially sound suspicious. Are we showing one thing to humans and another thing to machines?
The distinction is in the content and the negotiation mechanism. If a server detects SomeAIBot/1.0 and silently sends that crawler a rewritten, keyword-stuffed article that humans never see, that is one thing.
If a client sends:
and receives the same article represented as Markdown, that is normal HTTP content negotiation.
The response should also include:
so caches know that the representation changes depending on the request’s Accept header.
There is another small detail worth getting right. If a client asks only for formats you cannot provide, returning:
is valid. But it is not mandatory.
RFC 9110 allows an origin server either to honor the preference and return 406, or to disregard the negotiation preference and provide a default representation.
This is a useful reminder that implementation details are often more nuanced than emerging “AI readiness” checklists suggest. The web already has rules for this. We should use them rather than invent stricter ones and then call them standards.
So far, everything has been about reading. But agents increasingly want to do things. Search a catalog. Submit a form. Add an item to a cart. Book an appointment. Change a reservation.
A human can infer these possibilities from visual interfaces. An agent has to identify the available actions, understand their parameters and execute them reliably.
This is where newer proposals such as WebMCP become interesting. Chrome is already experimenting with WebMCP integration in its agentic browsing tooling. A page can expose actions declaratively through HTML or imperatively through JavaScript, allowing an agent to discover capabilities such as book_appointment or add_to_cart rather than reverse-engineering every interaction from pixels and DOM events.
This changes the question. An AI-readable website exposes understandable information. An agent-ready website exposes understandable capabilities.
Those are related problems, but they are not the same one.
Once automated clients can perform meaningful actions, another old assumption starts to break.
A User-Agent string says:
It does not prove it. Anyone can send the same string.
The emerging Web Bot Auth work at the IETF is trying to address that problem using HTTP Message Signatures.
The current working-group draft describes a protocol where automated HTTP clients can cryptographically sign their requests, giving servers a stronger mechanism for verifying the identity of automated traffic.
This is worth watching, but not deploying as though it were finished infrastructure.
As of September 2026, it is still an active Internet-Draft — explicitly work in progress, not a finalized RFC.
That distinction is important throughout this entire subject.
There are mature web standards.
There are conventions with growing adoption.
And there are experiments pointing toward what the web may become.
Treating all three as equivalent is how short-lived best practices get created.
So what should a normal website actually do today?
The difference between these categories matters more than any individual technique.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "The AI-Readable Web: A Practical Checkup",
"description": "What actually makes a website readable by crawlers, LLMs, and AI agents.",
"datePublished": "2026-09-23",
"dateModified": "2026-09-23",
"author": {
"@type": "Person",
"name": "Lorenzo Fiori"
},
"publisher": {
"@type": "Organization",
"name": "Typable Studio"
},
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://typable.studio/..."
}
}
</script>This page
↓
is an Article
↓
written by a Person
↓
published by an Organization
↓
about a particular subjectGET /articles/example
Accept: text/markdown, text/html;q=0.8HTTP/2 200
Content-Type: text/markdown; charset=utf-8
Vary: Accept ┌── HTML
Content source ─────┼── Markdown
└── other representations<link
rel="alternate"
type="text/markdown"
href="/articles/example.md"
/>
<link
rel="describedby"
href="/llms.txt"
/><div class="visually-hidden" aria-hidden="true">
A Markdown version of this page is available at /article.md.
</div><link
rel="alternate"
type="text/markdown"
href="/article.md"
/>Accept: text/markdownVary: Accept406 Not AcceptableI am SomeBot.