You Can't Debug an Agent From a Log Line: How We Trace Notis With Langfuse

An agent failure almost never looks like an exception. It looks like a user saying "that's not what I asked for," and a log file that shows every single step returning 200.

That is the problem Langfuse solves for us at Notis. Not uptime. Not errors. Intent, drift, and the thirty-step path between a voice note and a finished document.

Why a log line is useless here

A traditional request is one thing happening. You log the input, you log the output, you're done.

A Notis request is a conversation that spawns work. Someone sends a voice note from WhatsApp. That gets transcribed, routed to an agent, which reads long-term memory, decides it needs three tools, calls a search, calls a scrape, decides the result is thin, calls a different tool, writes a document, and replies. Somewhere in there it might hand a sub-task to a delegated coding agent that runs for ten minutes and comes back.

If the answer is wrong, "which step was wrong" is the entire question. A flat log gives you a wall of lines in timestamp order with no parent-child structure. You cannot see that the scrape returned an empty body, so the model hallucinated a summary, so the document is confident and useless.

Langfuse gives us the tree. Generations for model calls, spans for tool calls, nested under a trace that corresponds to one unit of user-visible work. When I debug, I'm not reading logs. I'm reading a shape.

The one rule we enforce at the boundary

Here's the thing about tracing an assistant: the traces contain people's actual lives. Meeting notes. Client emails. Half-finished thoughts dictated in a car.

So we don't use the Langfuse SDK directly anywhere in the codebase. We use a wrapper module whose only job is to make it structurally impossible to construct an unmasked client. Every client that gets built has a mask function attached at construction time, applied at the export boundary, before anything leaves the process.

That's a deliberate design choice over the more common approach of "remember to redact at each call site." Call sites get added by people in a hurry. Constructors get reviewed once.

I want to be precise about what that buys and what it doesn't. Credential masking at the export boundary means secrets, keys and tokens don't reach the tracing backend. It does not mean traces are free of user content — an agent trace of a document-writing task contains the document. That's why Langfuse's own security posture matters to us, and why they're listed by name in our Help Centre's platform providers overview rather than buried in a subprocessor PDF.

The bug that taught us to cache clients

This one is my favourite, because it's the kind of thing you only find in production.

Every Langfuse instance starts its own ingestion threads, its own media-upload consumer, and its own HTTP connection pool. Only an explicit shutdown() stops them. Our routers construct a client per request step, with process-constant credentials.

You can see where this goes. We were leaking roughly a hundred threads every ten minutes. Memory grew linearly. Eventually the process got OOM-killed, came back, and started over. Nothing in the application logs said "tracing is eating your server," because tracing was working perfectly.

The fix is boring: cache one client per distinct constructor signature, behind a lock. It's what the SDK intends. But we only found it because the leak was linear and predictable enough to correlate — which is itself an argument for having observability on the thing that does your observability.

How tracing changed how we ship

Three habits came out of this.

We attach identity to traces, not just payloads. A trace carries tags and metadata that tell us which user flow, which channel, and which agent configuration produced it. When someone reports "Notis got confused in Telegram this morning," that's a filter, not an archaeology project.

We read traces before we read complaints. A user reporting a bad answer usually can't tell you what went wrong, because the failure is invisible from their side. The trace can. Most of our "the model is bad" reports turn out to be a tool returning something unexpected and the model doing its best with it.

We built feedback into the Portal. There's a route that lets a Langfuse trace be scored from inside the product, so a thumbs-down on an answer attaches to the exact execution that produced it. Sentiment without the trace is noise. Sentiment on the trace is a dataset.

What I'd tell another agent builder

If you're building anything agentic, get tracing in before you get clever.

The instinct is to add observability once the product works. But an agent doesn't have a "works" state you can hold still — it has a distribution of behaviours, and you need to see the distribution to improve it. Every hour I spent on tracing early bought back several hours of guessing later.

A few practical notes:

• Wrap the SDK, don't call it. One module, one place to enforce masking, one place to fix a threading bug.
• Trace the unit of user-visible work, not the HTTP request. Those are not the same thing in an agent.
• Span your tools. Model calls are the obvious thing to trace and rarely the thing that's broken. Tool calls are where reality intrudes.
• Decide your privacy contract before you have data, because retroactively cleaning a tracing backend is not a good weekend.

Notis is a founder-led product, still actively expanding from voice-to-Notion capture into connected-tool workflows. A lot of that expansion is only possible because I can see what the agent actually did, step by step, instead of arguing with a log file.

Langfuse is the reason I can see it. Go look at it if you're shipping agents.