I Spoke to 400 Users. Analytics Still Couldn’t Tell Me What to Build.

I have spent an embarrassing amount of time staring at product analytics and hoping the next chart would tell me what to build. It never did. The clearest product decisions in Notis came from customer discovery interviews—more than 400 conversations with users—not from a dashboard. Analytics showed me where people dropped. The calls explained why they cared, what they were trying to do, and which problem was worth solving next. That distinction matters if you are an early-stage founder. Quantitative data is excellent at measuring behavior once enough behavior exists. It is much worse at discovering the words, anxieties, workarounds, and unexpected jobs behind that behavior. The useful answer is not “interviews or analytics.” It is analytics for the what, conversations for the why, and a disciplined system for turning both into a decision.

Analytics told me what happened. Users told me what it meant.

A dashboard can show that activation fell after step three. It cannot reliably tell you whether the user was confused, distracted, unconvinced, worried about privacy, or simply solving a different problem than the one your onboarding assumed. All five can produce the same event pattern. They require five different product decisions. In my case, the most important pattern was not a feature request. It was the person behind the request. After enough calls, I realized that many of the founders getting the most value from Notis had ADHD traits—even when that was not how they described themselves. They did not need another perfect dashboard. They needed a low-friction way to capture an intention and let an agent move the work forward from the places where they already communicated. No analytics report handed me that positioning. It accumulated slowly, then clicked all at once. That is the slightly annoying truth about founder-led customer discovery: the spreadsheet rarely contains the breakthrough. Your brain builds a model from repeated exposure to the same pain, phrased in different ways.

Launch before your analytics look impressive

Founders often postpone customer conversations until the product feels presentable. That is backwards. Y Combinator’s essential startup advice is blunt: launch right away, talk to customers, and iterate instead of waiting for a “perfect” product. The product only needs enough utility that its value outweighs the rough edges. That matched my experience building Notis. I put up the landing page in the first week and started onboarding people as soon as I had something I personally used. The early version was buggy. People still gave it five-star reviews because they could see I cared, responded, and fixed things. Early users are not allergic to bugs. They are allergic to indifference. The point of launching early is not to collect vanity registrations. It is to shorten the distance between your assumptions and reality. A small number of real users can teach you more than a large amount of hypothetical market research because their friction has stakes. They tried to get a job done and something got in the way.

A customer discovery system that does not collapse at 400 calls

Talking to users is wonderfully unscalable at the beginning. Eventually, the same strength becomes a problem: you remember the loudest call, the latest complaint, or the customer you happen to like. The answer is not to stop talking. It is to add a system that preserves evidence and reduces recency bias.

Start with one analysis per conversation

Do not dump hundreds of transcripts into one giant prompt and ask for “the top insights.” That produces plausible mush. Analyze each conversation separately first. Extract the user’s goal, friction, workaround, emotional stakes, requested outcome, and the exact passages supporting each claim. Keeping the first pass narrow makes it easier to audit.

Roll up by user before rolling up by market

A single user may describe the same underlying problem across onboarding, support, and a founder call. Synthesize those interactions into a per-user view before comparing users. Otherwise, your most talkative customers become three votes while quieter customers become one.

Look for patterns that change a decision

A theme is not useful because it appears often. It is useful when it changes what you do. Connect each pattern to a decision: fix onboarding, improve reliability, change positioning, build a feature, adjust pricing, or deliberately do nothing. Keep the evidence attached so the founder can challenge the summary instead of trusting a confident paragraph from a model.

AI should scale listening, not replace it

The workflow I built for myself starts with conversation traces, sends one sub-agent to each conversation, rolls the findings up per user, and then produces a cross-user report. Cheap models can handle extraction; the stronger model should be reserved for synthesis and prioritization. The report is not the roadmap. It is the evidence pack I use to decide what deserves attention. Tools such as Langfuse already give AI product teams structured traces, sessions, scores, and metadata. That instrumentation is valuable. But observability answers a different question from strategy. Knowing that an agent failed, retried, or received a poor score does not automatically tell you which user segment to prioritize or which use case should define the product. This is where the current generation of “AI product insight” tools risks becoming another dashboard. A founder does not need a prettier cluster of feature requests. The useful output is a weekly brief that says: here are the recurring jobs, here is the evidence, here is who experiences them, here is what changed, and here are the decisions now worth discussing.

Privacy is not a setting you add later

Customer conversations are often the most sensitive data in a product. They contain personal details, business problems, support history, and information users never expected to become generic training material. I would not send raw Notis conversations to a random third-party insight service, no matter how nice the dashboard looked. For sensitive products, the better architecture is local-first or customer-controlled: ship the analysis logic as a skill or MCP, run it inside the company’s own Claude or Codex environment, and return only the report the team asked for. Langfuse also documents a self-hosted deployment option for teams that need infrastructure-level control. Self-hosting alone does not solve every privacy question, but it moves the boundary to a place the customer can inspect and govern.

What I would do if I were launching again

I would launch as soon as one real job worked end to end. I would schedule conversations before building a serious analytics stack. After every call, I would record the user’s desired outcome, current workaround, strongest frustration, and the sentence that best captured the problem. Once the volume became painful, I would automate the synthesis—not the relationship. I would also keep the whole loop connected. Notes should not die in a transcript folder. Patterns should become decisions, decisions should become tasks, and shipped changes should be checked against later conversations. That connected execution layer is the reason we rebuilt Notis around an agentic workspace, rather than treating AI as a chat box that produces more text. Four hundred calls did not give me a perfect roadmap. They gave me something better: a sharper model of the people I was building for. Analytics helped me measure that model. They did not create it. If you are still polishing before launch, stop. Put the useful part in front of someone. Ask what they were trying to accomplish. Listen for the workaround they built before you arrived. Then repeat until the patterns become impossible to ignore. The dashboard can wait a week. The conversation cannot.