An AI agent sitemap cut downloaded text by 58%, but input tokens rose 3.5%
Ask an agent for the price of small fries, and the site hands it the entire menu. The Deckard team built agent-sitemap, a generator that splits a site into linked Markdown notes that work like a table of contents. With this map, Claude Code made 6 requests instead of 9 and downloaded 58% less text, but input tokens rose by 3.5%. Savings should therefore be measured in tokens and cache usage, not page size.
AI-processed from Habr AI; edited by Hamidun News
Ask an AI agent to visit a restaurant's website and find the price of small fries, and a human would simply open the "Menu" section and find the right line. An agent works differently: the tool gives it HTML, cleaned text, or a screenshot of the whole menu, so the burgers and drinks land in context along with the fries. The Deckard team tried to solve this with a table of contents for agents and released an open-source generator, agent-sitemap.
In a trial measurement, Claude Code with such a map downloaded 58% less text (24 627 characters instead of 58 439), but input tokens, meaning the volume the model read and that you pay for, rose by 3.5%. For a developer, the takeaway is simple: shrinking the size of downloaded pages does not necessarily save money on model requests.
How the map works
A sitemap.xml file says which pages exist on a site, but it does not show where the fries sit inside the menu. An llms.txt file can describe links and serve Markdown versions of pages, but a long menu stays long after you click through. It just no longer has HTML.
The generator takes a different approach and creates a separate Markdown file for each section. The index (index.md) points to a page card. The card, which holds an intro and a list of sections, points to a section note, and the note points to related material. The agent opens the index, goes to "Menu," picks "Potatoes and Sides" from its description, and never reads the burgers and drinks. Each note keeps the address of the original page so the agent can cite it and verify the price. The generator writes the same links to graph.json for anyone who reads the map from their own code.
The input is either local HTML and Markdown files or a site with its own sitemap.xml. Text is split by second-level headings (##). Some links already exist in the document: a page contains sections, one section follows another, and the author may have linked to a neighboring page. The generator finds the missing links by comparing words across pages (TF-IDF and cosine similarity). That can produce false matches: the word "delivery" appears just as often in pricing as in return policies. So you can enable an LLM pass that writes short explanations for the notes and filters out some of the links found by word matching. Links added by the author stay in place.
What the measurement showed
The project's README includes a trial run: two questions about the Deckard site whose answers are buried inside long articles. Claude Code started from either the home page or the map's index, could use curl, and knew the site served Markdown. A model checked correctness against predefined reference answers, and all four answers were marked correct.
With the map, the agent made 6 HTTP requests instead of 9 and downloaded 24 627 characters instead of 58 439. For the developer question, token usage was 33 029 versus 31 828 without the map; for the ship question, 31 958 versus 30 983. In total, that is 64 987 versus 62 811, an increase of about 3.5%.
The author explains the discrepancy. The character counter only counts the content of HTTP responses. The input-token counter also includes all the other context the model receives at each step. In addition, the test harness adds up regular input tokens, cache writes, and cache reads, even though they are priced differently. So the table shows how much text the site served, not how much the answer cost. To find that out, you need to look at the agent's runs and a per-category token report separately.
There is another caveat: the "approximately N tokens" label next to a link is calculated from characters, with different coefficients for Cyrillic and other text, and without the tokenizer of any specific model. The estimate covers the main text, even though the file also contains headings, metadata, and links, which in a short note can be longer than the section itself.
What to do in practice
Treat the conclusions with caution: this is one run, two questions, and one agent. An assistant with ordinary web search and a browser may take a different route. The measurement lacks two control variants: llms.txt with whole Markdown pages (to isolate the effect of HTML cleanup alone) and going straight to the correct page. If that page has two paragraphs, the path through the index and card is almost certainly longer than just reading it in full.
For your own test, the author recommends taking 10–30 real questions and repeating each one several times under identical conditions. Let some answers sit in a single section, let some require several pages, and include some that the site does not answer at all. Measure answer correctness and source citation, number of requests, volume of downloaded text, time, and cost. Record the time the model spends checking answers separately.
Setup is simple: npx agent-sitemap https://example.com builds an ai directory. Publish it next to your site, check that /ai/index.md opens, and give the agent that address explicitly. That way the result of navigating the map is not mixed up with the question of whether the agent found the map at all.
The risks are also clear. A rule and its exception can end up in different notes, and the agent will then be only half right. A heading like "Additional information" says nothing about the content. The Markdown copy can fall behind the site, so for prices it is better to go to the live page or an API. And note text arrives from outside, like any web page, so instructions inside it must not be treated as commands from the user. The code is open source under the MIT license, and the repository is called agent-sitemap.
- Nov 19, 2026A Lawyer Taught an AI Agent to Make Surgical Edits in Word with Track Changes
- Nov 19, 2026RSAC 2026 in Hindsight: AI Assistants Invented 19.7% of Recommended Packages
- Nov 19, 2026AI agent wrote a green refund test that checks only 2 of 6 requirements
- Nov 19, 2026Codex Agents Ran 1700 Investigations Into ChatGPT Slowdowns and Found the Cause in 600
Key Questions Answered
How the map works?
A sitemap.xml file says which pages exist on a site, but it does not show where the fries sit inside the menu. An llms.txt file can describe links and serve Markdown versions of pages, but a long menu stays long after you click through. It just no longer has HTML. The generator takes a different approach and creates a separate Markdown file for each section. The index (index.md) points to a page card.
What the measurement showed?
The project's README includes a trial run: two questions about the Deckard site whose answers are buried inside long articles. Claude Code started from either the home page or the map's index, could use curl, and knew the site served Markdown. A model checked correctness against predefined reference answers, and all four answers were marked correct. With the map, the agent made 6 HTTP requests instead of 9 and downloaded 24 627 characters instead of 58 439.
What to do in practice?
Treat the conclusions with caution: this is one run, two questions, and one agent. An assistant with ordinary web search and a browser may take a different route. The measurement lacks two control variants: llms.txt with whole Markdown pages (to isolate the effect of HTML cleanup alone) and going straight to the correct page. If that page has two paragraphs, the path through the index and card is almost certainly longer than just reading it in full.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.