Republic / Articles /

Article updated 2026-10-11.
Written by Christofer Sandin.

Making the web ready for agents

More than half of the traffic on the web no longer comes from people. In June, Cloudflare reported that 57 percent of requests to a sample of sites on its network came from automated bots. According to Cloudflare’s CEO, AI agents are the main driver of that growth – agents that read, compare and, more and more often, do things on someone else’s behalf.

A couple of years ago we wrote that Google is often called the web’s most important blind visitor. Now that visitor has company. The other day I gave a breakfast seminar on what happens when an AI agent knocks on your website’s door – and what is actually worth doing about it.

Here are the highlights.

Three layers: readable, queryable, actionable

The work can be split into three layers:

  • Readable. The agent should be able to fetch and understand your content.
  • Queryable. Your systems should be able to answer questions from Claude or ChatGPT via MCP.
  • Actionable. The agent should be able to use the features on your website via WebMCP.

We have structured information before

Around the turn of the millennium the argument was search engines, around 2010 it was accessibility, and now it is agents. The work is the same: mark up your content so that a machine understands what is a heading, what is text and what is navigation. Most of it is about doing what we already do, and well-structured HTML is still the foundation.

There are three things you can get done in an afternoon:

  1. Check your robots.txt. Are you blocking GPTBot and ClaudeBot? Some services, like Cloudflare, block AI bots by default on new domains, so it may have happened without anyone ever making a decision. With Content-Signal you can also answer three separate questions: may you appear in search results, may you be used as input for AI answers, and may models be trained on you? These are signals, not locks, and none of the big AI players has yet confirmed that they honour them – but the point is to make an active choice.
  2. Add an llms.txt. This is a proposal, not a standard, and it is unclear how often it is used. But creating the file costs next to nothing – and the idea is that it works roughly like a curated table of contents for agents.
  3. Serve Markdown. The same URL with .md on the end, or the same URL where the client asks for Accept: text/markdown. The client then gets clean text instead of HTML.

robots.txt

There are lots of settings you can make in robots.txt – this is what the setting we are talking about looks like:

93 percent less to read

We tried it on ourselves. At the time of testing, the republic.se home page was 127,129 bytes as HTML and 8,445 bytes as Markdown, roughly 93 percent less. The HTML version is mostly div tags, scripts and CSS classes. Somewhere in there is the text.

Don’t get me wrong: bandwidth savings are not the main benefit. Agents are good at parsing HTML, and 93 percent fewer bytes does not mean a 93 percent reduction in cost. The real benefit is the context window. Every exchange between you and the agent includes the entire conversation so far, so the less noise it contains, the more capacity remains for what the agent is actually supposed to do.

That means faster answers and clearer reasoning. It also reduces the risk of hallucinations, which become more likely when information does not fit within the context window—especially with simpler models that have smaller windows.

Isn’t serving agents different content a form of cloaking? No. Cloaking is when bots receive different content from people. Here, the content is the same; the client simply requests a different format. It is the same mechanism that APIs have used for almost 30 years.

Today, coding tools are the main clients requesting Markdown: Claude Code, Cursor and GitHub Copilot, as well as Microsoft Copilot Chat. Chat applications such as ChatGPT, Claude.ai, Gemini and Perplexity still fetch HTML, according to acceptmarkdown.com. If your audience consists of developers asking coding agents about your documentation, supporting Markdown is already worthwhile. For everyone else, the implementation cost is low enough to make it worthwhile anyway.

How do we serve Markdown in practice?

Technically, there are two complementary approaches. We can provide a URL ending in .md. We can also allow the client to request Markdown from the regular URL by including Accept: text/markdown in its request.

The latter uses HTTP’s existing support for negotiating formats, known as content negotiation. The server must then tell caches that the response can vary depending on the Accept header by including Vary: Accept in its response headers. The content should be equivalent in both versions.

HTTP Semantics describes how this negotiation works.

At Republic, we have built plugins for Craft CMS and Statamic that generate Markdown from a website’s content. The Craft version runs on republic.se and is the one we demonstrated at the seminar.

This does not introduce any additional complexity for editors. They continue updating the content in one place, and the system converts it into whichever format the client requests.

Things that don’t work

This is where it is easy to be misled. Evil Martians reviewed just over thirty blog posts, repositories and specification proposals and evaluated the techniques they described. Eight of the techniques were unsupported, including:

  • Made-up meta tags like ai-content-url and llms, as well as files like /ai.txt.
  • HTML comments. These are removed before the model sees the text.
  • “AI version” buttons. Agents do not click buttons.
  • User-Agent sniffing. Google considers serving bots a separate version to be cloaking, so use content negotiation instead.
  • Important information available only in JSON-LD. In a test by SearchVIU, ChatGPT, Claude, Perplexity, Gemini and Copilot all missed product data that was available only there.

Do not remove your structured data—Google still recommends it. But do not make it the only place where important information appears.

Why, then, do so many people recommend these techniques? Probably because someone invents a file or meta tag and writes a blog post about it. Others cite the post, and the idea spreads. The problem is that no one checks whether any AI system actually benefits from the recommendation.

The most boring advice is the best: write better text and include sources, quotations, and data

If you take away only one thing, make it this.

A research group from Princeton and IIT Delhi tested nine strategies for appearing in AI-generated answers. The successful strategies all involved enriching the visible text. Citing sources increased visibility by up to 115 percent for lower-ranked pages, although the top-ranked page lost around 30 percent. Direct quotations increased visibility by around 40 percent, while statistics produced an increase of around 30 percent.

The best advice for AI optimisation is the most boring advice in the industry: write better text.

Be specific, cite sources and include numbers. It is also the only advice that will remain effective regardless of which standard eventually prevails.

The next section looks at information and functionality within your systems that someone might otherwise have to log in and retrieve manually.

MCP connects AI tools to your systems

An API is like a plug, but each application uses a different type. Model Context Protocol, or MCP, acts as a universal adapter. It gives the agent a standard connection that can interface with systems such as your CMS, online store or booking platform.

Anthropic created the protocol, which is now stewarded by the Linux Foundation with backing from OpenAI, Google, Microsoft and others. In practice, it has become the current standard.

Imagine that you are hosting a seminar and want to identify which attendees have reported an allergy. Or perhaps you need to compile information about all new orders every week. With the right connection, an agent can retrieve that data through a narrowly scoped tool using only the permissions you have chosen to grant it.

For many of our clients, internal workflows are a natural place to start with MCP. MCP can of course also be used for services with external users; I can, for example, use MCP to fetch my email from our email client. What matters is which data and which features you choose to offer, and who gets to use them.

This is where we need to talk about the work that is actually supposed to get easier. Which tasks are repeated often? Which information takes time to pull together? A clear use case makes it easier to decide whether the integration is worth building.

WebMCP describes what the website can do

Then there is the user who wants to get something done on the website itself. She wants to compare products, fill in a form or book an appointment.

An agent can try to do that by interpreting the page structure, or by taking screenshots and clicking its way through. That works in many cases, but it can be fragile when the interface changes or the next step is unclear. And above all it is slow: it does the same job as a person, only worse and slower.

WebMCP offers a way to describe features as tools directly in the web page. A tool has a name, a description and a schema for the input it accepts. An agent that supports this can call the feature through the browser. The WebMCP proposal describes how it is meant to work.

At the seminar we showed our demo store Republic Coffee Co. There, an agent can use the store’s search to fetch information about espresso machines. One of the tools is called search_products and calls the same search function the store already uses.

await document.modelContext.registerTool({
  name: "search_products",
  description: "Search products",
  inputSchema: {
    type: "object",
    properties: { query: { type: "string" } },
    required: ["query"]
  },
  annotations: { readOnlyHint: true },
  execute: async ({ query }) => await searchCatalogue(query)
});

It is quite concrete once you see it. The customer asks about a particular machine and the agent fetches the details from the site without having to hunt around. The customer can then ask for a comparison and get help adding a product to the cart.

In our demo, the store offered three tools to an anonymous visitor and sixteen after login. That shows how the set of tools can follow the situation on the page. The server still has to check that the user is allowed to perform each action, exactly as we do with a human visitor on the web.

I think this is the most interesting part in the long run. Features we have already built can become easier to use for anyone getting help from an agent. A product filter, for example, could let customers describe what they need in their own words.

The user needs to understand and be able to control the actions

When an agent can do things on the user’s behalf, feedback becomes important. It needs to be clear what has happened and whether there is anything the user needs to decide.

A search, a changed delivery address and a payment have different consequences. So I would design the checkpoints around the action. Browsing a product catalogue can happen straight away. Before a payment goes through, on the other hand, the customer needs to see what is being bought and the total amount, and actively confirm the purchase, in order to trust the system.

We also have to validate input and check permissions on the server, just as with any other request. Clear tools make the intent easier to describe, but they replace none of those checks.

This ties in with what I wrote about in UX design for AI. Users need enough insight to judge what the system is doing and to step in where needed. When the agent helps out in an online store, it is still the customer’s money and the customer’s decision, and whatever is done is done directly in the customer’s name and with her authority.

A reasonable first project

WebMCP is still a draft from the W3C Web Machine Learning Community Group. It is not a W3C standard yet, and it is not supported in every browser and agent people use. I would start with a limited pilot and evaluate one concrete use case.

Search and product comparison are good candidates. In HUMAN Security’s observations for April 2026, around 69.6 percent of agent traffic went to product and search URLs, while around 3.2 percent went to checkout and payment. This covers traffic on their platform and describes visits to different types of URLs, not completed purchases, but it still gives us a good indication of where things stand right now. HUMAN Security’s report has more on how it was measured.

My assessment, then, is that there is often more to learn from making product information easy to find than a fully automated checkout would give you. We can test whether the agent finds the right product and whether the answers help the customer choose. Then we can take the next step based on what we see.

What is worth doing now?

For a typical content website, I would start by checking access and reviewing how the text is structured. Is the information your customers ask for there? Can it be found and read outside the visual interface too? After that, Markdown is a reasonable complement when it can be generated and maintained from the same source.

If you have recurring work in internal systems, an MCP connection may be worth looking into. For an online store or portal with clear use cases, a small WebMCP pilot can give you experience of how people and agents use the features together. If your staff are already used to working with Claude, ChatGPT or Gemini in sensible ways, and would benefit from being able to pull in information from the website too, it doesn’t take a huge investment to try.

These parts can be developed independently. The important thing is to pick a need that can be described and followed up.

At Republic we have started with our own website and our own tests, to learn what actually works. That gives us a better basis when we discuss next steps with our clients.

If you want to look at what would make sense for your website or your systems, we are happy to talk. We can start with a question your customers already ask, or a task your colleagues do every week. That is usually a good place to start!

Wrapping up

The layers have to be taken in order. There is no point in WebMCP tools on a site that blocks the bots in robots.txt.

  • Readable – do it now. Robots.txt, llms.txt and Markdown cost little and work no matter how the agent wave plays out.
  • Queryable – may be worth a look already. An MCP server on top of your features can be used in Claude and, depending on your plan, in ChatGPT right now.
  • Actionable – prepare this autumn. WebMCP is not a settled standard yet, but everything suggests it is the one that will apply for the foreseeable future.

So the blind visitor has company, and the new visitor doesn’t just want to read. But the foundation is the same as it always has been: clear structure and good text. If you want to test your own site, run it through isitagentready.com and acceptmarkdown.com while your coffee cools.

We have plugins for Craft CMS and Statamic that serve Markdown and llms.txt straight from the CMS. Here on republic.se we already run them. If you want to know where your site stands, or which MCP and WebMCP tools are worth building for you, we would be happy to meet.

References and further reading

Relaterade artiklar