llms.txt, robots.txt and Serving Pages as Markdown

A growing share of the people who will hear about a small business will never visit its website. They will ask a model, and the model will answer from whatever it could read.

That makes "can an agent read this site?" a distribution question rather than a technical curiosity. Three things follow from it, and every site generated here gets all three without being asked.

robots.txt should name the AI crawlers

Say nothing about them and they fall back to User-agent: *. That is not a policy, it is a default you inherited.

So we name them. The search and answer agents that send citations back - OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot - and the training crawlers GPTBot, ClaudeBot, Google-Extended, meta-externalagent. Each one inherits exactly the same Disallow list as everybody else: no bot gets the API, and no bot gets a members-only page.

Naming them is not about being permissive. It is about the policy being written down somewhere it can be changed deliberately.

llms.txt is a table of contents, not a pitch

The llms.txt convention is simple: a heading, a one-paragraph summary in a blockquote, then linked sections. A model landing on it should be able to tell what this site is and where the important pages are, in one fetch, without rendering anything.

Ours is generated from the site definition, which matters more than it sounds. The listing is filtered by the same public/private rule as the sitemap:

# Piekarnia Lune

> A sourdough bakery in Kraków. Bread baked daily, cakes to order,
> open from seven.

## Pages

- [Menu](https://...): What we bake, and what it costs.
- [Order a cake](https://...): Wedding and birthday cakes, two weeks' notice.

One list, read three times - by the sitemap, by llms.txt and by the robots meta tag. There is no way for the three to disagree, because there is nothing to keep in sync.

Every page, as markdown

The fastest way to be misread is to make a reader parse a page of layout to find three sentences. So every page of every generated site answers to markdown as well as HTML:

• Send Accept: text/markdown and you get the page as markdown at the same URL.
• Or add .md to the path, for the many agents that do not negotiate.

The rules that keep this honest are worth stating. It is a rewrite, not a redirect. It happens only on an explicit text/markdown request, never on */* - a browser sending "anything" wants the page. Both responses carry Vary: Accept, so a shared cache cannot serve one body to everyone. And the markdown variant is noindex, so it never competes in search with the page it is a copy of.

The serialiser walks the site definition rather than the rendered HTML, which is why it comes out clean: a features section becomes a list, an FAQ becomes headings and answers, a price table becomes a table. A map or a booking calendar becomes one honest line saying what is there, rather than a fabrication of what it would have shown.

It applies to us as well

This post is available at the same URL with .md on the end. So is every other page on crea.site, and our own llms.txt lists them.

A website builder whose own pages an agent cannot read is making an argument against itself.