Deterministic AX Metrics: Measuring Token Usage, Context Load & Recovery

machine_summary:
  claim: "agent experience is measurable as relative deltas — a fixed task, the same model, before and after an interface change — in token burn, context load, and recovery cost; absolute and cross-model comparisons are noise"
  takeaway: "define a reference task, run it same-model before and after each interface change, and gate releases on deltas in tokens, tool calls, and failed calls; external MCP audits stay open until a standard task suite exists"
  evidence_type: mechanism
  cross_links: [ax-vs-ux, computational-kindness, search-space-design]

Why “deterministic” is the load-bearing word in AX metrics

The previous piece ended with an IOU: hand an agent a real task, count the tokens between “confused” and “recovered,” and — someday — make those numbers deterministic and comparable. This is that piece. The claim: agent experience is measurable without vibes. Three properties of any agent-facing interface — token burn, context load, and recovery cost — are countable, reproducible, and comparable across versions of your product. That puts them in a different epistemic class from most agent evaluation, and it is what separates AX-as-engineering-discipline from AX-as-opinion.

The distinction matters because the default instruments are stochastic. LLM-judged evals, sampled success rates, eyeballed transcripts — all useful, none enforceable. A judge score that moves from 7.2 to 7.4 supports any narrative you like; nobody blocks a release over it. A diff that says this release costs every agent 800 more tokens per task supports exactly one narrative, and a CI gate can read it.

Here is the whole thesis in one measurement. This site publishes every article twice: a rendered HTML page and a raw markdown twin at the same URL plus .md. Same words, same argument. Run both through one tokenizer (we used o200k_base) and our first post costs 5,115 tokens as HTML and 2,222 as markdown. Identical content, 2.3× the context load, purely from markup the agent must carry and doesn’t need. No judge, no sampling, no error bars: fetch the two URLs and count. Note the shape of the finding: it’s the ratio that matters, not the raw counts — a paired comparison, which is the shape every measurement in this piece takes.

Because the method is the actual thesis: AX measurement is relative, not absolute. Devise a reference task — small, repeatable, representative: find the overdue invoices and draft the reminder emails. Run it against your interface with one model and record three hard numbers: tokens spent, tool calls made, tool calls failed. Change the interface. Run it again — same task, same model — and the delta is the measurement. Agents are stochastic, so “run it” means a handful of runs and a median; the spread tells you how much delta is real. Then repeat the whole comparison under a different model to see whether the improvement generalizes — but compare each model only with itself, never with another. Absolute numbers are hostages to model, prompt, and tokenizer; deltas on a fixed task survive all three. You are not measuring what your interface costs in some cosmic ledger. You are measuring what your last change did.

The three metrics, defined so you can implement them

Token burn: what a task costs through your interface

Definition: the tokens an agent spends completing the reference task through your interface, with its two blunt companions: the count of tool calls and the count of failed calls. No accounting apparatus needed — your model provider already reports token usage per request, and within one model that’s all a relative measurement requires. Where you can, keep categories separate: schema tokens are your API surface’s fixed overhead, result tokens are your payload design, and the reasoning between calls is the guesswork tax made visible. The total tells you a change mattered; the category tells you why. And don’t underrate the blunt companions — the same task now takes 6 tool calls instead of 11, with zero failures instead of 3 is the most legible sentence in this genre.

Context load: what your product costs before the first call

Definition: the tokens an agent must hold in its window merely to be able to act on your product — tool schemas, instructions, documentation, and whatever your responses force it to keep. It is the tax before the first call, and the HTML-versus-markdown measurement above is exactly this metric in the wild. Context load is the most purely deterministic of the three: it doesn’t depend on agent behavior at all. It is also the most consequential per token, because context is rivalrous — every token of your boilerplate crowds out the user’s actual problem, and a crowded window is where confabulation breeds. An MCP server with seventy tools has a context load whether or not any of them is ever called.

Recovery: what an error actually costs

Definition: from the first failed call to the next action that advances the task — count the turns, the tokens, and whether the agent got back on track at all. Four numbers: turns-to-stabilize, tokens-to-recover, time-to-recover (wall clock, for the latency budget), success-rate-after-recovery. This is the metric nobody instruments and the one that dominates real session cost, because failure is where agents loop: retry variations, re-list tools, re-read documentation — each round trip paid in full. It is also measurable deterministically, by replay: inject the same failure — the malformed parameter, the expired scope, the rate-limit wall — into a pinned harness and count the path back. An error message that names the failure, the fix, and the tool has a measurably shorter path than 403 Forbidden. Now that difference has a unit.

From metrics to SLOs: where engineering and product agree

Each metric maps to an outcome someone already cares about, which is what lets a team agree on numbers instead of adjectives. Token burn is cost — literally the line item on the API bill of every agent that touches you, multiplied by however many sessions you serve. Context load is risk — crowding and confabulation — plus a hard ceiling on what agents can do in one session with your product. Recovery is task success: sessions mostly don’t fail on the happy path; they bleed out in loops.

And because the measurement is relative, the SLOs are too — they gate movement, not magnitude: token burn on the reference task regresses no more than 10% release-over-release. A failed call costs at most two turns to recover. Context load doesn’t grow without someone deciding it should. Each is checkable in CI with the same fixed task, the same way you’d gate a performance budget. The rest of the observability apparatus — dashboards, alert thresholds, experiment templates — follows from having a unit, the way it always does; none of it is the hard part. And to keep the claim honest: these three metrics are the floor of AX measurement, not the ceiling. They won’t tell you whether your tool semantics are coherent or your abstractions well-chosen — no deterministic number will. But a floor is exactly what the discipline is missing.

The open problem: measuring an MCP you don’t own

The tempting extension is external audit: point the same instruments at any MCP server and publish the table. Half of that works today. You could hand an agent a prompt that says run these tasks against this server and report back — the hard numbers (tokens, tool calls, failed calls) plus the soft ones agents are unusually good at articulating: where it guessed, what it had to re-read, which error left it stranded. The collection side is solved; agents are excellent instruments of their own experience.

What’s missing is the other half of the comparison. A relative measurement needs a baseline, and for a server you don’t own there isn’t one — no standard task suite, no reference server, no agreed-on reasonable cost for a given capability. As far as we can tell, no such standard exists for MCP servers at all. Until one does, an external number is an anecdote with units: real, honest, and comparable to nothing. We’d rather flag that gap than paper over it — and note that whoever builds the boring, fixed, public task suite will have written the first page of the audit genre.

What you can do without waiting is what this piece is actually arguing for: measure yourself, against yourself, in public if you dare. This blog has spent three articles claiming that clarity, kindness, and machine-readability are better agent experience. The twins are published and the paired comparison above is runnable from your terminal: same content, two formats, 2.3× apart. Measure us. That standing invitation — not any particular number — is what a discipline looks like.