How to Turn Any Web Page Into Clean Markdown for Your Agent

Last week I watched an agent spend half a dollar reading one Wikipedia page. Not summarising it, not acting on it. Reading it. I pulled the raw HTML of the Markdown article with curl on 19 August 2026 and got 311,074 characters, most of it navigation, scripts and markup a model pays for and ignores. Converting that same web page to markdown first brought it down to 60,027 characters, headings, links and infobox intact.

That gap is the whole article. Here is the receipt, and three ways to get it.

Why "web page to markdown" is a pricing decision, not a formatting one

Anthropic's pricing documentation puts one token at "approximately 4 characters or 0.75 words in English" and lists Claude Opus 5 input at $5 per million tokens. It also carries a caveat most people miss: Claude 4.7 and later models use a newer tokenizer that "produces approximately 30% more tokens for the same text". Opus 5 is one of those, so the 4-character rule of thumb needs the uplift. My two measurements, run through that arithmetic:

• Raw HTML: 311,074 characters, so roughly 101,000 Opus 5 tokens, about $0.50 of input.
• Markdown: 60,027 characters, so roughly 19,500 tokens, about $0.10 of input.

About $0.40 evaporates on a single page, before the model has written one word of output. Loop that over twenty competitor pages and you have paid about $8 for markup you deliberately discarded. These are estimates off a character count, not a tokenizer read, and the same page warns the real increase "depends on the content and workload shape" - so read them as a floor, not an invoice.

Two more things from that page. The fetch is rarely where the money goes: Anthropic prices its web fetch tool at "no additional cost" beyond the tokens of what it drags into context. And its example of an "average web page (10 kB)" is about 2,500 tokens - the raw HTML above is forty times that. Small pages genuinely do not matter. Documentation, pricing pages and long reference articles are the pages you actually send an agent to, and the ones that bite.

Option 1: convert it yourself in about ten lines of Python

Two libraries do the whole job. readability-lxml finds the article and throws away the chrome, and html2text turns what is left into markdown.

import requests, html2text
from readability import Document

raw = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}).text
article = Document(raw).summary()      # drop nav, ads, boilerplate
converter = html2text.HTML2Text()
converter.body_width = 0               # no hard wrapping
markdown = converter.handle(article)

On my test page that returned 30,685 characters, roughly 10,000 tokens, about $0.05 at the same Opus 5 input rate. Cheapest per page by a distance, and you own every line of it.

It is also the option with the honest asterisk. Readability is opinionated: my run started mid-article and binned the infobox, so it carried 8 lines of table markup where the scraped version carried 51. Fine if you wanted the prose. Bad if you wanted the table. Add a page that renders in JavaScript and you get almost nothing back until you bolt on a headless browser - and then you are maintaining a scraper instead of doing the work you scraped for.

Option 2: rent a hosted scrape API

If the job is thousands of pages inside a pipeline you control, pay someone to run the crawler. Firecrawl's pricing page lists a free plan with 1,000 credits and a Hobby plan at $16 a month billed yearly with 5,000, at 1 credit per scrape - about $0.0032 a page if you burn every credit that month, more if you do not.

Good deal at volume. Also another account, another key in another .env, another subscription line, and one more thing that only works inside the code you wrote around it.

Option 3: ask for it from wherever you already are

This is the route I actually use: most of the time I want one page, right now, nowhere near a terminal. Notis has a native scrape tool, so a message works: "read notis.ai/pricing and send me the markdown". From a terminal it is one call:

notis tools exec LOCAL_NOTIS_SCRAPE \
  --arguments '{"url":"https://en.wikipedia.org/wiki/Markdown","formats":["markdown"]}'

That is the call that produced the 60,027 characters above, infobox included. On the Notis on-demand catalogue a page read costs about $0.004, drawn from usage the plan already includes rather than charged on top: the pricing page states Pro includes $20 of usage a month, Pro+ $59 and Ultra $149, and on-demand usage past that keeps running at the same published rates. Pro is $13 a month billed annually.

There is a guard behind it that matters more than the price. Any single pay-per-use call quoted above $1.20 is refused outright, and the agent has to state the price and get a yes before retrying. A page read sits three orders of magnitude under that line, which is the point: you never think about it.

The three routes side by side

Trim it further: summary, question, highlights

Full markdown is still the expensive option if all you needed was one fact. The same tool takes a formats list and two shortcuts.

Asking for summary on the same page returned 708 characters, roughly 230 tokens, about $0.0012 of Opus 5 input against about $0.10 for the full markdown. Asking a question instead ("question": "In what year was the Markdown language first released, and who created it?") came back with an answer field reading "2004, John Gruber", which matches the article's infobox: John Gruber, initial release March 9, 2004.

Now the part nobody puts in a tutorial. summary and answer are generated, not extracted, so the short shapes are where facts go to die. Asked a two-part question - "Which year was CommonMark rebranded, and who launched the effort?" - I first got "September 2014, by John Gruber." Wrong, and the page it read proves it: the Wikipedia article says the effort was launched in 2012 by a group including Jeff Atwood and John MacFarlane, and that September 2014 is when Gruber objected to the name. The answer welded the date onto the wrong person.

The identical call on 25 August 2026 got both halves right, four times longer; the summary re-run came back at 658 characters rather than 708. These shortcuts are not a stable API surface. They are a model compressing a page on your behalf, and how much it keeps varies run to run. One fact per question, and when it matters, check it against the markdown, which ships in the same payload. There is a highlights_query too, returning just the passages relevant to a query.

The rule that saves the most money is boring: decide what you actually need before you fetch, not after.

What to do with the markdown once you have it

Markdown in a terminal is a means, not an end. The reason I care about the $0.004 is what happens in the next second: the same run drops the cleaned page into a Notion database, or writes the Gmail draft, or just answers in the WhatsApp thread I asked from. No repo, no checkout, no copy-paste into a chat window.

That is where the routes separate. A script gives you a file. An API gives you a payload. "Check these five competitor pricing pages every Monday and put the diffs in my Notion tracker" gives you an outcome, on the surfaces the business actually lives on: messages, mail, docs, CRM.

Which one to pick

Write your own if you are batching thousands of pages, you enjoy owning the pipe, and your targets are static. Rent a hosted scrape API if that volume needs to be reliable and you would rather someone else fight Cloudflare. Ask an agent that already has the tool if you want one page now, from your phone, delivered somewhere useful.

Whichever you pick, carry away the number at the top. Converting a web page costs fractions of a cent. Not converting it cost me about $0.40 on one Wikipedia article, and I would have paid it again on the next one without ever seeing the line item.