---
title: "Deterministic AX Metrics: Measuring Token Usage, Context Load & Recovery"
description: "Agent experience is measurable without vibes: fix a repeatable task, change your interface, and compare tokens, tool calls, and failed calls between runs. Same model only."
pubDate: 2026-07-22
arc: business
tldr: "Most agent evaluation is stochastic — LLM judges, sampled runs, eyeballed transcripts. But three properties of an interface are simply countable: token burn (what a task costs), context load (what your product costs before the first call), and recovery (what an error costs). The method is relative, not absolute: fix a repeatable task, run it before and after a change with the same model, and compare the deltas — tokens spent, tool calls made, calls failed. That makes agent experience regression-testable, which is the difference between AX as an engineering discipline and AX as an opinion."
machineSummary:
  claim: "agent experience is measurable as relative deltas — a fixed task, the same model, before and after an interface change — in token burn, context load, and recovery cost; absolute and cross-model comparisons are noise"
  takeaway: "define a reference task, run it same-model before and after each interface change, and gate releases on deltas in tokens, tool calls, and failed calls; external MCP audits stay open until a standard task suite exists"
  evidenceType: mechanism
  crossLinks: [ax-vs-ux, computational-kindness, search-space-design]
  # add [varynforge-field-report] when that post is live
keywords: [deterministic AX metrics, agent experience metrics, token usage measurement, context load, AX recovery metrics, token accounting, AX SLOs, measuring agent experience, telemetry for agents]
draft: false
---

## Why "deterministic" is the load-bearing word in AX metrics

The [previous piece](/posts/computational-kindness/#how-to-tell-your-mcp-is-being-unkind) ended with an IOU: hand an agent a real task, count the tokens between "confused" and "recovered," and — someday — make those numbers deterministic and comparable. This is that piece. The claim: **agent experience is measurable without vibes.** Three properties of any agent-facing interface — token burn, context load, and recovery cost — are countable, reproducible, and comparable across versions of your product. That puts them in a different epistemic class from most agent evaluation, and it is what separates AX-as-engineering-discipline from AX-as-opinion.

The distinction matters because the default instruments are stochastic. LLM-judged evals, sampled success rates, eyeballed transcripts — all useful, none enforceable. A judge score that moves from 7.2 to 7.4 supports any narrative you like; nobody blocks a release over it. A diff that says *this release costs every agent 800 more tokens per task* supports exactly one narrative, and a CI gate can read it.

Here is the whole thesis in one measurement. This site publishes every article twice: a rendered HTML page and a raw markdown twin at the same URL plus `.md`. Same words, same argument. Run both through one tokenizer (we used o200k_base) and our first post costs **5,115 tokens as HTML and 2,222 as markdown**. Identical content, 2.3× the context load, purely from markup the agent must carry and doesn't need. No judge, no sampling, no error bars: fetch the two URLs and count. Note the shape of the finding: it's the ratio that matters, not the raw counts — a paired comparison, which is the shape every measurement in this piece takes.

Because the method is the actual thesis: **AX measurement is relative, not absolute.** Devise a reference task — small, repeatable, representative: *find the overdue invoices and draft the reminder emails.* Run it against your interface with one model and record three hard numbers: tokens spent, tool calls made, tool calls failed. Change the interface. Run it again — same task, same model — and the delta is the measurement. Agents are stochastic, so "run it" means a handful of runs and a median; the spread tells you how much delta is real. Then repeat the whole comparison under a different model to see whether the improvement generalizes — but compare each model only with itself, never with another. Absolute numbers are hostages to model, prompt, and tokenizer; deltas on a fixed task survive all three. You are not measuring what your interface costs in some cosmic ledger. You are measuring what your last change did.

## The three metrics, defined so you can implement them

### Token burn: what a task costs through your interface

**Definition: the tokens an agent spends completing the reference task through your interface**, with its two blunt companions: the count of tool calls and the count of failed calls. No accounting apparatus needed — your model provider already reports token usage per request, and within one model that's all a relative measurement requires. Where you can, keep categories separate: schema tokens are your API surface's fixed overhead, result tokens are your payload design, and the reasoning between calls is the [guesswork tax](/posts/computational-kindness/#what-computational-kindness-means-for-agent-experience) made visible. The total tells you a change mattered; the category tells you why. And don't underrate the blunt companions — *the same task now takes 6 tool calls instead of 11, with zero failures instead of 3* is the most legible sentence in this genre.

### Context load: what your product costs before the first call

**Definition: the tokens an agent must hold in its window merely to be able to act on your product** — tool schemas, instructions, documentation, and whatever your responses force it to keep. It is the tax before the first call, and the HTML-versus-markdown measurement above is exactly this metric in the wild. Context load is the most purely deterministic of the three: it doesn't depend on agent behavior at all. It is also the most consequential per token, because context is rivalrous — every token of your boilerplate crowds out the user's actual problem, and a crowded window is where [confabulation](/posts/ax-vs-ux/#key-divergences-assumptions-that-break-when-users-are-agents) breeds. An MCP server with seventy tools has a context load whether or not any of them is ever called.

### Recovery: what an error actually costs

**Definition: from the first failed call to the next action that advances the task — count the turns, the tokens, and whether the agent got back on track at all.** Four numbers: turns-to-stabilize, tokens-to-recover, time-to-recover (wall clock, for the latency budget), success-rate-after-recovery. This is the metric nobody instruments and the one that dominates real session cost, because failure is where agents loop: retry variations, re-list tools, re-read documentation — each round trip paid in full. It is also measurable deterministically, by replay: inject the same failure — the malformed parameter, the expired scope, the rate-limit wall — into a pinned harness and count the path back. An error message that names the failure, the fix, and the tool has a measurably shorter path than `403 Forbidden`. Now that difference has a unit.

## From metrics to SLOs: where engineering and product agree

Each metric maps to an outcome someone already cares about, which is what lets a team agree on numbers instead of adjectives. Token burn is cost — literally the line item on the API bill of every agent that touches you, multiplied by however many sessions you serve. Context load is risk — crowding and confabulation — plus a hard ceiling on what agents can do in one session with your product. Recovery is task success: sessions mostly don't fail on the happy path; they bleed out in loops.

And because the measurement is relative, the SLOs are too — they gate movement, not magnitude: *token burn on the reference task regresses no more than 10% release-over-release. A failed call costs at most two turns to recover. Context load doesn't grow without someone deciding it should.* Each is checkable in CI with the same fixed task, the same way you'd gate a performance budget. The rest of the observability apparatus — dashboards, alert thresholds, experiment templates — follows from having a unit, the way it always does; none of it is the hard part. And to keep the claim honest: these three metrics are the floor of AX measurement, not the ceiling. They won't tell you whether your tool semantics are coherent or your abstractions well-chosen — no deterministic number will. But a floor is exactly what the discipline is missing.

## The open problem: measuring an MCP you don't own

The tempting extension is external audit: point the same instruments at any MCP server and publish the table. Half of that works today. You could hand an agent a prompt that says *run these tasks against this server and report back* — the hard numbers (tokens, tool calls, failed calls) plus the soft ones agents are unusually good at articulating: where it guessed, what it had to re-read, which error left it stranded. The collection side is solved; agents are excellent instruments of their own experience.

What's missing is the other half of the comparison. A relative measurement needs a baseline, and for a server you don't own there isn't one — no standard task suite, no reference server, no agreed-on reasonable cost for a given capability. As far as we can tell, no such standard exists for MCP servers at all. Until one does, an external number is an anecdote with units: real, honest, and comparable to nothing. We'd rather flag that gap than paper over it — and note that whoever builds the boring, fixed, public task suite will have written the first page of the audit genre.

What you *can* do without waiting is what this piece is actually arguing for: measure yourself, against yourself, in public if you dare. This blog has spent three articles claiming that clarity, kindness, and machine-readability are better agent experience. The twins are published and the paired comparison above is runnable from your terminal: same content, two formats, 2.3× apart. Measure us. That standing invitation — not any particular number — is what a discipline looks like.