Hello Agentic World: Making Your Website Agent-Friendly

A few days ago I read a great post by my colleague Guillaume Laforge about turning his blog into something AI agents can read cleanly. His blog was built with Hugo but mine is a from scratch static site generator I built myself.

I was curious how straightforward it would be to make my site 'agent friendly' like Guillaume's so I prompted Gemini 4 via Antigravity to first ask questions and then go ahead with the implementation. At the end of this article you can find a single compacted prompt I prepared that should assist with any website.

Answering the Why

The first question we should always ask ourselves before taking on a task is to answer the "Why" – by that I mean, why are we about to do this task? Why would we care what an AI agent sees when it accesses one's website?

To answer this, let's start by thinking about what a person sees when opening my blog: a person sees a three-column layout with my bio on the left, a list of projects and articles in the middle, and the post itself on the right. That's a nice UX for humans but is it optimal for AI agents to digest and process?

When an AI agent fetches that exact same page, it has to wade through hundreds of lines of HTML tags, CSS classes, sidebar links, and scripts before it even reaches the first sentence of the page's actual content. Roughly 80% of the page is just scaffolding. That wastes the agent's context window, slows it down, and makes it more likely to misread what's actually on the page.

In short: optimisation for AI agents reduces token consumption everywhere, speeds up processing time, and increases the accuracy and understanding of your page for the AI agent audience.

Since I already write every page on this site in markdown, it felt silly to turn clean markdown into noisy HTML, only for visiting AI agents to try and scrape it back into text again.

Step 1: Asking What "Agent-Friendly" Actually Means Today

Obviously I'd read Guillaume's article so I already knew the answer but I felt it was good to setup the discussion with Gemini first with the following prompt (who knows, maybe the best practices had evolved in the last 24 hours!):

Prompt:
What are the best practices for serving an agent-friendly version of a site nowadays? I've seen people rendering Markdown files directly.

In general, I find grounding any major architectural task for a site, app or game is a great starting point with Antigravity. This is the exploratory phase.

The short answer was that you don't need to reinvent your website. If your site is built from markdown (as mine was), the simplest win is to keep/serve a clean .md copy of every public page right next to the .html version.

For example:

• People read: https://tomgreenaway.com/blog/whats-a-programmer
• AI Agents can read: https://tomgreenaway.com/blog/whats-a-programmer.md

And how does an AI agent know that the .md version exists? There are a few options but one of the simplest is to add a single line inside the <head> of your HTML template pointing to it:

<link rel="alternate" type="text/markdown" href="https://tomgreenaway.com/blog/whats-a-programmer.md" title="Markdown" />

This same mechanism has been used for many purposes in website standards for decades including canonical URLs, RSS feeds, and even the Accelerated Mobile Pages format which I helped out on at Google.

Step 2: robots.txt meet llms.txt and agents.txt

Giving every page a .md twin is great once an agent is already on a specific article. But what if an AI agent lands on your home page and wants to know what your site is about and what's worth reading?

Prompt:
What other steps does one need to take? Is there an agents.txt file or something like that?

Right now, the web has settled on a defacto stanard of having a couple of simple text files that live at the root of your site – right alongside the classic robots.txt:

1. /llms.txt (the table of contents): This is the big one. It's a short, plain markdown file at the root of your domain that acts like a map of links for AI models. It has a one-paragraph summary of who you are, followed by a clean list of links to the .md versions of your published articles and pages.
2. /llms-full.txt (a single payload of everything): An optional companion file that combines all of your published markdown posts into one single document, for agents with huge context windows that want to read your whole site in one gulp.
3. /agents.txt (rules/guidance for the agents on how to behave): While robots.txt tells search crawlers where they aren't allowed to go, agents.txt is a newer convention for telling active AI agents how to use your site – pointing them to your /llms.txt file, letting them know they can request markdown directly, and stating whether you want attribution when they quote you.

Because my site has a build script that turns markdown into HTML, I didn't have to write these files by hand. I just asked the agent to generate /llms.txt, /llms-full.txt, /robots.txt, and /agents.txt automatically every time I run npm run build.

Step 3: Teaching Cloudflare to know about both HTML and Markdown

Having .md URLs is helpful, but many modern coding agents (like Antigravity or Claude Code) don't even change the URL when they fetch a link you paste into chat. Instead, they request the normal URL (https://tomgreenaway.com/blog/whats-a-programmer) and quietly attach an HTTP header that says:

Accept: text/markdown

This means that when the AI agent is asking for the page, it is quietly checking if there's a markdown version available instead.

Prompt:
Can Cloudflare support sending a different page based on an HTTP request header?

I host this site on Cloudflare and deploy it from my terminal using Wrangler – Cloudflare's command-line tool.

Normally, a static site on Cloudflare is just a folder of pre-built files (public/). You run npx wrangler deploy, Wrangler uploads your folder to Cloudflare's servers around the world, and Cloudflare hands those files out for free. There's no server code running at all.

To support that Accept: text/markdown header, you only need to add two tiny pieces:

1. A 20-line "server side" script (worker.js): In Cloudflare, a Worker is a tiny bit of JavaScript lets Cloudflare send a markdown file if the requestor (person or AI agent or whatever) requests it via the HTTP header I mentioned above.
2. A pointer in wrangler.jsonc: You tell Wrangler to use worker.js for page requests, while letting images, CSS, and fonts bypass the script completely.

flowchart LR
    V["Visitor requests<br/>/blog/whats-a-programmer"] --> W["Cloudflare Worker<br/>(worker.js checks header)"]
    W -- "Normal browser" --> H["Serves whats-a-programmer.html"]
    W -- "Accept: text/markdown" --> M["Serves whats-a-programmer.md"]

When a human being visits in Chrome, the Worker hands them the regular .html page. When an agent visits with Accept: text/markdown, the Worker swaps in the .md file from the exact same folder.

Step 4: Watch Out for the "Unlisted" Trap

As builders, it's always really important after you've implemented anything to really test thoroughly and catch issues. So once everything was wired up and deployed, I opened my new /llms.txt file to check the list of pages – and immediately spotted a problem.

On my site, I have a few half-baked article drafts and experimental games/pages that aren't linked in the navigation menu. They're meant to be unlisted: live at their URL if I send the link to a friend, but hidden from search engines and indexes.

Because those pages still lived inside my pages/ folder as Markdown files, the build script had dutifully listed every single one of them inside /llms.txt and /llms-full.txt!

Prompt:
Do the individual pages have the ability to be marked as 'unlisted' as well? If something is unlisted, it shouldn't have a public .md mirror and it shouldn't appear in llms.txt

This is an easy trap to fall into when you're making a site agent-friendly. An llms.txt file is a public megaphone. If your build script blindly sweeps up every .md file in your repo, you'll end up advertising your unfinished drafts and private pages to every AI crawler who asks.

One follow-up prompt fixed it so that any article or page with status: unlisted in its front-matter:

• Still builds its .html page (with <meta name="robots" content="noindex">).
• Does not generate a .md file.
• Does not get included in /llms.txt or /llms-full.txt.

Try It on Your Own Site

If you want to add an agentic view to your own blog or portfolio, you don't need a new framework. Just open your project in your favourite coding agent and paste this single prompt:

I want to make this website agent-friendly by implementing the following:

1. Clean Markdown Mirrors: Update the build step so every published page and blog post outputs a clean `.md` copy alongside its `.html` file, and add `<link rel="alternate" type="text/markdown" href="...">` inside the HTML `<head>`.
2. Discovery Files: Auto-generate `/llms.txt` (a concise summary + links to every published `.md` page and article), `/llms-full.txt` (the full published Markdown corpus), `/agents.txt` (declaring Markdown support and attribution policy), and update `/robots.txt` to reference `Llms-Txt`.
3. Respect Unlisted & Draft Content: Ensure any draft, private, or `unlisted` pages/posts are strictly excluded from `.md` mirror generation, `/llms.txt`, and `/llms-full.txt` (and include `<meta name="robots" content="noindex">` on unlisted HTML pages).
4. HTTP Content Negotiation: Add a lightweight edge handler or Worker so requests sent with `Accept: text/markdown` receive the clean `.md` file (with `Content-Type: text/markdown; charset=utf-8` and `Vary: Accept`) while normal browsers continue receiving the HTML page. If the hosting provider supports this capability.

Before starting the implementation, walk me through the steps and be to the point with a low amount of tech jaron please.

And if you want to see what this exact article looks like to an agent right now, you can just run this on your terminal:

curl -H "Accept: text/markdown" https://tomgreenaway.com/blog/hello-agentic-world

Article published on: 9 Nov, 2026