AI Agent Evaluation: Stop Grading Demos. Measure Finished Work.

The AI agent evaluation problem is not that we lack benchmarks. It is that most teams are grading the wrong thing.

A polished demo answers, “Can the agent complete this carefully chosen task once?” A founder needs a much harsher answer: “Can it finish my real work, under real constraints, and leave enough evidence that I do not have to redo everything?”

That difference is the gap between impressive automation and actual delegation.

The demo is the easiest part

Demos remove the messy bits. The prompt is clean. The account is already connected. The data is available. Nobody interrupts the run. The successful path gets recorded; the abandoned attempts disappear.

Real work looks different. An email is ambiguous. A calendar has two connected accounts. A contact name matches three people. A website changes its form. A tool times out after the agent has already taken one irreversible action.

This is why I think about an agent as an operating system for work, not a chat box with extra buttons. The architecture has to include tools, memory, permissions, retries, escalation, and proof—not just a clever model. I wrote more about that in AI Agent Architecture for Founders Who Just Want the Work Done.

If an evaluation hides those conditions, it is measuring a stage performance.

Pass/fail is not an AI agent evaluation

A binary success rate sounds reassuring. It also hides almost everything that matters in production.

An agent can technically succeed after 18 minutes, six retries, €4 of API calls, and ten minutes of human checking. Another can fail safely in 20 seconds, explain what it needs, and preserve every prior action. The first gets a pass. The second gets a fail. I would often choose the second system.

AgentSLABench makes this problem explicit by evaluating agents against service-level objectives for correctness, latency, cost, compute, memory, and network usage. Its effective attainment metric penalizes a “successful” run that breaks the operating budget. That is the right direction: success at any cost is not production success.

The token bill is usually the smallest layer. Above it sit retries, human review, missed deadlines, duplicated actions, and the cost of a bad outcome. If your dashboard reports only tokens and task completion, it is hiding the expensive part.

Productivity studies are not contradictory

You have probably seen incompatible headlines about AI at work.

A randomized field experiment involving 4,867 developers across Microsoft, Accenture, and another Fortune 100 company found a pooled 26.08% increase in completed tasks with AI coding assistants, while noting uncertainty in individual company estimates (Management Science).

A different randomized study from METR tested 16 experienced open-source developers on 246 tasks in their own repositories. With early-2025 AI tools, they took 19% longer, even though they believed the tools had made them faster.

And in an advertising experiment with 2,234 participants, human-AI teams produced 50% more ads per worker and stronger text, while human-human teams performed better on image quality and AI-assisted output became more homogeneous (Dell’Acqua and colleagues).

These results are not a referendum where one paper cancels the others. They are evidence that task type, worker expertise, quality criteria, tool design, and the cost of checking the output change the result.

“Does AI make people faster?” is too vague to manage. “Does this agent produce more verified outcomes on this workflow, at acceptable quality and cost?” is useful.

Agents need a scorecard, not applause

Here is the scorecard I would use before trusting an agent with recurring work:

1. Verified completion. Did the intended outcome happen in the system of record—not merely in the agent’s final message?
2. Quality. Did the result meet explicit acceptance criteria? “Email sent” is not enough if it went to the wrong person.
3. Latency. How long did a verified outcome take, including queueing, retries, and human response time?
4. Total cost. Count model calls, paid tools, failed attempts, and human review minutes.
5. Recovery and escalation. Did the agent retry only when safe, avoid duplicate actions, and ask for help at the right moment?
6. Evidence. Can a human inspect the source, decision, action, and final state without reconstructing the whole run?

That last part is not bureaucracy. It is what lets you delegate without surrendering control. NIST’s work on evaluation probes for agentic AI similarly calls for machine-readable audit trails, factual grounding, and adversarial checks inside multi-step workflows.

Good evidence also supports security. An agent should prove what it accessed and changed while respecting narrow permissions. That is the same principle behind how Notis keeps data secure without security theatre.

The metric I would put on the dashboard

My north-star metric would be Verified Outcome Rate:

verified completed outcomes ÷ all assigned outcomes

“Verified” must mean that the expected state exists and the acceptance criteria pass. An email is in Sent. A meeting exists on the correct calendar. A database record contains the required fields. A scheduled post has a platform ID and the intended media.

I would pair that rate with four guardrails:

• cost per verified outcome;
• median human-review minutes;
• retry and duplicate-action rate;
• escalation precision: did the agent ask for help when it should, and stay out of the way when it could continue safely?

This also exposes fake efficiency. EcoAgent-Bench evaluates 304 real-derived tasks with priced actions and explicit budgets. It finds that agents can comply with a stated budget without adapting decisions economically—under-escalating in some cases and overspending in others. Passing a budget check is not the same as making good trade-offs.

What this changes when you buy or build an agent

Do not begin with the vendor’s leaderboard. Begin with 20 examples of your work: routine cases, ambiguous cases, missing-data cases, and one or two dangerous edge cases.

Define the expected end state and the evidence required. Run the same set repeatedly. Record failures, retries, review time, and cost. Then change one thing at a time.

For founders exploring where agents genuinely help, AI Agents for Business maps the workflows that benefit from delegation and the ones that still need a human in the loop.

My own rule is simple:

If checking the work means doing the work again, the agent has not delegated anything.

The best agent is not the one that looks most human in a demo. It is the one that quietly finishes the work, knows when to stop, and hands you proof.

That is what we are building at Notis: an AI assistant you can delegate to from the messaging apps you already use, with connected tools and verifiable outcomes—not another place to watch an agent perform.