Turning Web Pages Into Agent Context With Firecrawl

"Search the web" is the most over-promised feature in AI products. Most of the time it means: fetch some HTML, hope the readable text survives, feed the wreckage to a model, and let it write something confident.

At Notis, the step between "a URL exists" and "an agent can use it" is Firecrawl. Here's why that step deserves a dedicated tool rather than an afternoon with an HTTP client.

Raw HTML is not context

A modern page is mostly not the page. Navigation, cookie banners, consent walls, three newsletter modals, a footer with forty links, and — somewhere in there — the six paragraphs the user actually asked about.

Feed that to a language model and two things go wrong. The signal gets buried, so the answer drifts toward whatever was repeated most, which is usually the navigation. And you burn an enormous amount of context on markup that carries no meaning.

Then there's the half of the web that doesn't render server-side at all. Fetch it and you get a <div id="root"> and a prayer.

Our scrape tool calls Firecrawl's v2 scrape endpoint and asks for Markdown by default. What comes back is the content, structured, in the format models handle best. Headings stay headings. Lists stay lists. Tables mostly survive. Other formats are available when a task needs them, but Markdown-first is the right default because it's the format the rest of the pipeline already speaks.

That's the whole pitch, and it sounds small until you've spent a weekend writing extraction heuristics that break on the first site with an unusual layout.

Where it fits in an agent loop

Scraping inside an agent is different from scraping in a crawler, because the agent decides what to fetch based on what it just read.

A typical Notis research task looks like: someone asks a question, the agent searches, gets a set of candidate pages, scrapes the promising ones, reads them, notices a gap, scrapes a source cited in one of them, then writes something up and files it where it belongs.

That's a loop with a human waiting at the end of it. Which means the failure modes that matter are latency and silence. A scrape that takes forever is worse than one that fails, because the agent can route around a failure and can't route around a hang.

So the tool is built to come back — with content when it can, and with a clear, structured failure when it can't. An agent that knows a page was blocked can try a different source. An agent that gets an empty string quietly summarises nothing and sounds certain about it.

We also run a periodic provider heartbeat against the scrape endpoint. If Firecrawl is unreachable or our key is misconfigured, I want to know from a check rather than from a user asking why research got vague this afternoon.

Metering, and the bug that made us care

This is the part I'd tell other builders about, because it cost us real money before it cost us any thought.

Notis has usage-based components. Tools that cost money need to be accounted for. Our scrape tool originally wasn't — it called the API and returned content, and the usage never made it into our accounting. Every scrape was free to the user and not free to us.

The fix was to give scraping a proper model spec, the same way a language model has one, so every call reports usage that flows into the same billing path as everything else. There's even a warning logged if the spec goes missing, because a silent un-metered tool is exactly the kind of thing that stays broken for months.

The general lesson: in an agent product, any tool that calls a paid API is a pricing surface. It doesn't feel like one, because it's a function call in a loop. But agents call tools far more than humans click buttons, and a tool you forgot to meter will find that out for you.

Why we didn't build it

I'm not precious about buying versus building, but scraping is an unusually bad build.

The work isn't the first version. The work is proxies, retries, rendering for JavaScript-heavy pages, rate limits, bot mitigation, per-site quirks, and a content extractor that stays good as the web changes underneath it. That's a permanent maintenance commitment on something that is not our product.

One honest caveat: not all of Notis's web capability is Firecrawl. Search is separate. Deep research is its own pipeline. Browser automation is a different tool again, for when a task needs interaction rather than content. Firecrawl owns one job — page to usable text — and does it well enough that I've stopped thinking about it.

That's the highest compliment I give infrastructure.

If you're wiring this into an agent

• Ask for Markdown, not HTML. Your model is better at it and it costs you less context.
• Make failures structured and fast. An agent can plan around a known failure; it cannot plan around a hang.
• Meter every paid tool at the point of the call, and log loudly if the meter is missing.
• Heartbeat your providers. Silent degradation in a research tool looks like the model getting worse.
• Don't confuse scraping with browsing. Reading a page and operating one are different problems.

Notis is founder-led and still expanding from voice-to-Notion capture into broader connected-tool work. A lot of that work starts with a link someone pasted into WhatsApp.

Firecrawl is what turns that link into something the assistant can actually think with. Take a look.