Evals
Agent requests are independently executable, while the machine gives them meaningful state and trajectory context.
Testing without a provider
A test runs the real machine through runToQuiescence and replaces only the model. @statelyai/agent ships no model mock. There are two ways to replace the model:
- Mock the provider with the AI SDK's
MockLanguageModelV3. The call still runs throughcreateAiSdkExecutors, so the test covers prompt rendering, structured output parsing, and decision tool calls. - Pass plain functions as executors. The test covers the machine only.
Both examples below run this machine. It drafts an email, then scores the draft with a structured request.
import { z } from "zod";
import { setupAgent } from "@statelyai/agent";
const agent = setupAgent({
input: z.object({ topic: z.string() }),
context: z.object({ topic: z.string(), draft: z.string(), score: z.number().nullable() }),
output: z.object({ draft: z.string(), score: z.number().nullable() }),
requests: {
draftEmail: {
model: "writer",
schemas: { input: z.object({ topic: z.string() }), output: z.string() },
prompt: ({ input }) => `Write a short email about ${input.topic}.`,
},
scoreDraft: {
model: "judge",
schemas: {
input: z.object({ draft: z.string() }),
output: z.object({ score: z.number() }),
},
prompt: ({ input }) => `Score this email from 1 to 5:\n${input.draft}`,
},
},
});
const machine = agent.createMachine({
context: ({ input }) => ({ topic: input.topic, draft: "", score: null }),
initial: "drafting",
states: {
drafting: {
invoke: {
src: "draftEmail",
input: ({ context }) => ({ topic: context.topic }),
onDone: ({ output }) => ({ target: "scoring", context: { draft: output.result } }),
},
},
scoring: {
invoke: {
src: "scoreDraft",
input: ({ context }) => ({ draft: context.draft }),
onDone: ({ output }) => ({ target: "done", context: { score: output.result.score } }),
},
},
done: {
type: "final",
output: ({ context }) => ({ draft: context.draft, score: context.score }),
},
},
});Mock the model
MockLanguageModelV3 from ai/test is the AI SDK's test double for a language model. Put one in the models map of createAiSdkExecutors under each model key the machine uses.
import { MockLanguageModelV3 } from "ai/test";
import { runToQuiescence } from "@statelyai/agent";
import { createAiSdkExecutors } from "@statelyai/agent/ai-sdk";
function mockModel(text: string) {
return new MockLanguageModelV3({
doGenerate: async () => ({
content: [{ type: "text", text }],
finishReason: { unified: "stop", raw: "stop" },
usage: {
inputTokens: { total: 1, noCache: 1, cacheRead: 0, cacheWrite: 0 },
outputTokens: { total: 1, text: 1, reasoning: 0 },
},
warnings: [],
}),
});
}
const writer = mockModel("Checkout now supports saved cards.");
const judge = mockModel(JSON.stringify({ result: { score: 4 } }));
const result = await runToQuiescence(
createAgentRuntime(machine, { executors: createAiSdkExecutors({ models: { writer, judge } }) }),
{ input: { topic: "saved cards at checkout" } },
);
if (result.status === "done") {
console.log(result.output); // { draft: "Checkout now supports saved cards.", score: 4 }
}
console.log(judge.doGenerateCalls.length); // 1The mock answers in the provider's format, not the request's:
- A plain-text request takes the text as is.
- A structured request asks the provider for JSON and reads the declared output from a
resultkey, so the mock returns{ "result": … }as JSON text. - An
agent.deciderequest expects atool-callcontent part. ItstoolNameissend_event_<TYPE>for the chosen event type, and itsinputis the event payload as JSON text.
doGenerateCalls records the options of every call, including the prompt the adapter rendered.
Plain-function executors
An executor is a function of the request. Route on request.name to answer each request, and return the output value as result.
import { runToQuiescence } from "@statelyai/agent";
const calls: string[] = [];
const result = await runToQuiescence(
createAgentRuntime(machine, {
executors: {
generateText: async (request) => {
calls.push(request.name ?? "");
switch (request.name) {
case "draftEmail":
return { result: "Checkout now supports saved cards." };
case "scoreDraft":
return { result: { score: 4 } };
default:
throw new Error(`No answer for request "${request.name}"`);
}
},
},
}),
{ input: { topic: "saved cards at checkout" } },
);
console.log(calls); // ["draftEmail", "scoreDraft"]A decide executor routes the same way and returns the chosen event: decide: async (request) => ({ event: { type: "APPROVE" } }).
request.name is the request's setupAgent({ requests }) key, the createTextLogic/createDecisionLogic name, or the name on an inline agent.decide input. Without one, it is the invoke id. Route on the name, not on prompt text or call order. A run that takes a different branch then fails with a missing answer instead of receiving an answer meant for another request.
An executor that throws reaches the machine as an ordinary actor error. An invoke that declares onError routes it like a model failure.
Individual request evals
Call the executor directly when an eval targets one LLM call. Use the whole machine for control-flow evals so schemas, transitions, and failure handling are part of the assertion.
Seam evals
runSeam (from @statelyai/agent/testing) runs the whole machine with one named request occurrence under test. The seam gets a candidate executor, and every other text request is answered from scripts. Omit candidate to script the seam too. Score its seamOutput, calls, state path, and XState transition events.
const run = await runSeam(machine, {
input: { topic: "saved cards at checkout" },
scripts: {
draftEmail: ["Checkout now supports saved cards."],
scoreDraft: [{ score: 4 }],
},
seam: { request: "scoreDraft" },
});
run.seamOutput; // { score: 4 }
run.calls.map((call) => call.source); // ["script", "script"]runSeam routes text requests only. For a machine that also decides, pass a decide executor through executors.
Script keys
A scripts key is a request name when scripts has a queue under that name. Otherwise it is the request's model key. A machine whose requests are named routes by name, and an unnamed one routes by model.
A request's name resolves as described in Plain-function executors: its declared name, else the invoke id. Name every request a script answers, so a call with no matching queue throws instead of taking an answer queued for another request.
Entry shapes
An entry (SeamScriptEntry) is the request's output value: a bare string for a text request, or the object a structured request declares.
scripts: {
answer: ["Because transitions constrain behavior."],
assess: [{ score: 4, notes: "clear" }]
}An entry is read as an executor result instead only when its own keys are result plus, optionally, messages, usage and raw. That is how an entry reports token usage, which run.seamUsage reads for the seam call: { result: draft, usage: { totalTokens: 120 } }. A structured request whose declared output is itself { result } needs one more wrap: { result: { result: "…" } }.
An entry may also be a function of the request: answer: [(request) => "Draft about " + request.prompt].
Order and repetition
Entries for a key are consumed in order. The seam consumes its slot even when candidate answers it, so later entries stay aligned with the call plan.
A queue that runs dry throws an AgentError with code seam-script-exhausted, naming the request. Pass repeatLast: true to replay a queue's last entry instead, for a live seam that sends the run down a longer branch.
Per-state stats
foldTraces (from @statelyai/agent/testing) folds many runs' traces into one row per state: how many runs visited it, how many of those failed, how many options the guards withheld at decisions made there, and how many events it ignored.
import { foldTraces } from "@statelyai/agent/testing";
// One AgentTraceEvent[] per run, collected with onTrace.
const stats = foldTraces(traces, {
failed: (trace) => {
const end = trace.at(-1);
return end?.type !== "run.end" || !end.snapshot.matches("refunded");
},
});
// [{ state: "deciding", visits: 2, failures: 1, withheld: 2, ignored: 0, blamed: 0 }, ...]- A trace is one run's events, live or from
serializeTraceEvent.failedjudges the whole trace. - A run visits each state in a
machine.transition'sfrom.valueandsnapshot.value. Nested states are dot-joined paths; a parallel state counts once per active leaf. withheldcomes from a decision's firstrequest.start: the length ofrequest.withheld, counted toward the state the run was in, which is thefrom.valueof the transition the choice causes.ignoredcountsmachine.transitionevents withignored: true: a host event the state had no transition for, counted toward that state.decideNextcalled by the host emits no trace. Push its request onto the trace before sending the event:trace.push({ type: "request.start", request: result.request }).- Rows come in the order states were first seen. Sort them yourself, for example by
failures / visits. failurescounts a failed run against every state it visited, so the states after a bad decision rank as high as the decision. Passblameto count each failed run once, against the state its failure started in. See Where failures start.
Decision requests from traces
getDecisionRequests (from @statelyai/agent/testing) lists the decisions a run asked for, read from its trace: each decision's first request.start, and the states the run was in.
import { getDecisionRequests } from "@statelyai/agent/testing";
for (const { states, request } of getDecisionRequests(trace, { name: "chooseResolution" })) {
const { event } = await decide(request); // a candidate decider
console.log(
states,
request.options?.map((o) => o.event),
event,
);
}- A trace is one run's events, live or from
serializeTraceEvent. Retries of a decision are left out, so each decision appears once. - A serialized request keeps its
options,withheld,metadata,system, andprompt. Event schemas do not survive serialization, so replay closed choices rather than open ones. - Use the requests as eval rows (a decider on each, compared with a label) or as probe inputs for a policy.
Parity with the code you replace
compareRuns (from @statelyai/agent/testing) runs every scenario through the existing implementation and its replacement, and labels each difference. Use it when you port a loop to a machine and need to show where they behave the same and where the machine differs on purpose.
import { compareRuns } from "@statelyai/agent/testing";
const report = await compareRuns({
scenarios, // [{ id, expected?, ...your fields }]
baseline: async (scenario) => {
const run = await runRawLoop(scenario);
return { outcome: run.outcome, effects: run.world.refunds, violations: run.world.violations };
},
candidate: async (scenario) => {
const run = await runDesk(scenario);
return { outcome: run.outcome, effects: run.world.refunds, violations: run.world.violations };
},
});
expect(report.ok).toBe(true); // no regressions
// report.counts: { same: 7, intended: 3, regression: 0 }| Label | When |
|---|---|
same | Same outcome, structurally equal effects, and neither run broke a rule |
intended | The candidate broke no rule, and the baseline broke one (even with the same outcome), or only the candidate reached the scenario's expected outcome |
regression | The candidate broke a rule or threw, or they differ for any other reason |
- Parity is outcome-level: the terminal outcome and the effects in the world. It does not compare tool-call order, because a machine usually reorders steps.
violationscome from ground truth, such as a ledger the world checks after the fact. A port that blocks a rule the loop broke is labeledintended, not a regression, so the machine is never pushed to copy the loop's bugs.- Put every write the world records in
effects, not only the main one. A refund issued without its approval looks the same as a correct one ifeffectsholds only the refunds. - A run that throws is recorded with outcome
'threw'and its message asdetail. - Parity holds for these scenarios and these runs. With a live model, run it more than once, or drive both sides with the same deterministic stand-in model.
- A scenario object reserves two fields:
id: string, which names its row, and an optionalexpected: string, the outcome people know is right. Keep your own fields under other names. A field calledexpectedthat holds something else (an expected amount, an expected reply) changes how rows are labeled.
The run each side returns, and each row of report.scenarios:
interface ParityRun {
outcome: string; // a final state name or the loop's return label; 'threw' when the run threw
effects?: unknown; // what the run changed in the world, compared structurally
violations?: readonly unknown[]; // rules broken, from ground truth
detail?: unknown; // kept in the report, not compared
}
interface ParityResult<TScenario> {
id: string;
scenario: TScenario;
label: "same" | "intended" | "regression";
reason: string; // empty for 'same'
baseline: ParityRun;
candidate: ParityRun;
}examples/retrofit-parity ports a loop with recorded golden runs and asserts its one reviewed difference.
Any model client
createRecorder (from @statelyai/agent/testing) records a run's effects so a port can replay them with no network. recorder.model wraps an AI SDK model and recorder.openai an openai client. Any other client (@anthropic-ai/sdk, @google/genai, a framework's own model client) goes through recorder.client, which wraps the async methods you name and keeps the client's type.
import Anthropic from "@anthropic-ai/sdk";
import { compareRecordings, createRecorder } from "@statelyai/agent/testing";
declare function originalLoop(client: Anthropic): Promise<string>;
declare function runPort(client: Anthropic): Promise<string>;
const methods = ["messages.create"];
// Record the original against the real client.
const recorder = createRecorder();
recorder.output(await originalLoop(recorder.client("anthropic", new Anthropic(), { methods })));
// Replay the port: the client is never called.
const replay = createRecorder({ replay: recorder.recording });
const stand = new Anthropic({ apiKey: "replay" });
replay.output(await runPort(replay.client("anthropic", stand, { methods })));
compareRecordings(recorder.recording, replay.recording); // { equivalent: true, divergences: [] }- Each call is a
calleffect named<name>.<method>, such asanthropic.messages.create, with its arguments and its result or error. recorder.fn(name, asyncFn, options?)wraps one async function the same way, for a client that is a function.- Replay answers each call with the next unused recorded call of the same name, in order, whatever its arguments. A call past the recorded ones throws
RecordingDivergenceError. compareRecordingscompares arguments as recorded and reportscall-args. A port whose prompts differ on purpose passesredact: (args) => …to record only what must match, such as the model and the offered tool names.redactalso keeps secrets out of the recording.- A result that resolves to an async iterable (a stream) records its chunks; replay returns an async iterable of them.
serializemaps a result that is not JSON, or each chunk of a stream.revivebuilds the replayed value from the recorded result or chunks, such as a stream object with helper methods.- In replay the client is never called. A missing object on a method's path is stood in for, so
{} as Anthropicalso works.
Suite coverage
traceCoverage (from @statelyai/agent/testing) reports which states and handlers no trace exercised, so a suite can say how much of the machine it covers.
import { explorePaths, traceCoverage } from "@statelyai/agent/testing";
const report = traceCoverage(machine, traces, {
reachable: await explorePaths(machine, { input }),
});
report.unvisitedStates; // ["refunded"]
report.unvisitedHandlers; // [{ state: "deciding", event: "AUTO_APPROVE" }]
report.ratio; // { states: 0.75, handlers: 0.8 }
report.handlers[0];
// { state: "deciding", event: "AUTO_APPROVE", runs: 0, targets: { declared: ["refunded"], seen: [], unreached: ["refunded"] } }stateslists every atomic and final state, in document order, with the number of runs that visited it. Compound, parallel, choice, and history states are not listed; their leaves are.- A handler is a state and an event descriptor it declares: an
onkey, or the invoke andafterhandlers, which XState keys by bare type (xstate.done.actor,xstate.error.actor,xstate.after), one per state however many invokes it has. The machine root's handlers havestate: "". - A step credits one handler per active leaf: the nearest state, from the leaf up, whose descriptor matches the event. Exact descriptors come first, then partial (
task.*) and wildcard (*) ones, longest first, as XState orders them. A done or error event credits a state only when its invoke's id matches the event's. Guards are not replayed, so a step whose nearest handler a guard rejected is credited to that handler anyway. targets.declaredlists the targets of a handler's object transitions, or is"dynamic"when any is a function transition.targets.seenlists the active states after the steps credited to it.targets.unreachedlists declared targets no run landed in.alwayshandlers are listed inunobservable, never as unvisited. A trace records one step per macrostep, so an eventless transition inside one does not show.- A state an
alwaystransition passes through counts as visited, though no traced snapshot is in it.traceCoveragere-derives each step's microsteps from the snapshot before it with XState'sgetMicrosteps. It sees each snapshot's value and context only, not its live children. A step it cannot recompute, or whose recomputed route ends somewhere other than the traced snapshot, adds no such states. A guard that reads live children can still be recomputed down a different route that ends in the same place, and that route's states are credited. reachabletakes anexplorePathsreport. When the report explored every path (no cap hit, every payload expanded, no branch stopped early), states it never reached and no trace visited are listed inunreachableand leave the state denominator, soratio.statescounts only states a run could visit.reachableCompletesays whether it did. A partial report narrows nothing:unreachableis absent and every state stays in the denominator.ratiois visited over total, for states and for handlers. It is 1 when the total is 0.- Ignored steps credit no handler. Traces may be live or serialized, and a trace may span several resumed passes.
Where failures start
firstDivergence (from @statelyai/agent/testing) finds the first step where a run departs from a reference, and names the state the run was in.
import { firstDivergence, foldTraces } from "@statelyai/agent/testing";
firstDivergence(failingTrace, passingTrace);
// { index: 0, state: "deciding", decision: "chooseResolution", actual: { event: "escalate", … }, expected: { event: "refund", … } }
const stats = foldTraces(traces, {
failed,
blame: (trace) => firstDivergence(trace, passingTraceFor(trace))?.state,
});
// [{ state: "deciding", failures: 2, blamed: 2, … }, { state: "escalated", failures: 2, blamed: 0, … }]- The reference is a trace, such as a passing run of the same scenario, or an expected trajectory (
TrajectoryItem[], as formatchesTrajectory). - Against a trace, steps are compared in order. A step is one
machine.transition: the active states before it, the event type, and the active states after it. The first step that differs is the divergence, and so is the point where either trace ends early. - Against a trajectory, the run is laid out as its initial state, then each step's event and new state. The first expected item the run never reaches is the divergence, blamed on the step after the last item it did reach. A run that goes on past the trajectory has not diverged.
stateis the first active state before the diverging step.decisionis the name of the decision request that chose that step's event, when one did. An output that routes elsewhere is blamed on the state that invoked it.- Steps the machine ignored are left out on both sides.
@xstate.initis not a step. keyprojects a step to the string traces are compared by. The default isfrom|event|to. Override it to leave out a region that may differ between equivalent runs. Steps are still compared in order, so two runs whose parallel regions settle in a different order diverge.key: "payload"also compares each step's event payload, such as an actor's output or a decision's payload, as key-order-independent JSON. It leaves outsessionId, which differs between equivalent runs, and compares errors (anError, or a serialized one withstackorcode) byname,message, stringcode, andcause. Without it, an output that changes no state, such as"review"instead of"alert"from a classifier that always moves on, does not diverge.- Use
key: "payload"for deterministic or mocked runs and fault localization. In live-vs-live comparisons a model's output usually differs between runs, so the first model call usually diverges; use it there only when output-only differences matter. - An empty reference is a trace with no steps, so the run's first step diverges from it.
- Returns
undefinedwhen the run does not diverge. - In
foldTraces,blameis called for failed runs only.undefinedblames no state.
Proposals from traces
proposeApprovalRules and proposeCaps (from @statelyai/agent/testing) read many runs' traces and propose changes for a machine with no decision point: which approvals a rule could answer, and what each cap should be. They are proposals for a person. Nothing promotes them or edits the machine.
import { proposeApprovalRules, proposeCaps } from "@statelyai/agent/testing";
const rules = proposeApprovalRules(traces, {
approvalState: "awaitingApproval",
approveEvents: ["APPROVE"],
rejectEvents: ["REJECT"],
key: ({ snapshot }) => {
const { tool, path } = snapshot.context.pending;
return `${tool}:${path.startsWith("src/") ? "src/**" : path}`;
},
violations: (trace) => brokenRuleKeys(trace), // optional, from ground truth
});
// [{ key: "write:src/**", approved: 3, rejected: 0, violations: 0, runs: [0, 1, 2], proposal: "allow", support: 3 }, ...]
const caps = proposeCaps(traces, {
caps: [{ name: "retries", current: 3, failureState: "gaveUp", valueFromRun: countRetries }],
});
// [{ name: "retries", hits: 2, passing: 2, max: 3, p95: 1, proposal: 3 }]- Both take what
foldTracestakes: one trace per run, live or serialized. - A verdict is a
machine.transitionout ofapprovalState(or a state inside it) on an approve or reject event. Ignored events are not verdicts.keysees the event and the snapshot it arrived in, so it can read the pending call fromsnapshot.context. Returningundefinedskips the verdict. proposalis"allow"when a key was approved and never rejected, with no violation;"deny"when it was rejected and never approved; otherwisenull. A single rejection or violation blocks"allow".supportis the number of runs with a verdict for the key, andrunslists their indexes. Weigh a proposal by its support.- A run hits a cap when it enters
failureStateor itsrun.endoutput satisfiesfailureOutput.valueFromRunsays how much of the cap a run used;undefinedleaves the run out. maxis over all runs;p95is the nearest-rank 95th percentile over runs that did not hit the cap.proposalisp95 + 1. While any run hits the cap, it is never belowcurrent, so a cap runs still hit is never tightened. A cap no run hit can be lowered. With no passing run,proposalisnull.
Repeated runs
repeatRuns (from @statelyai/agent/testing) runs every scenario k times and reports how reliably each one passes. With a live model, one run per scenario says little: τ-bench found the rate at which all k runs of a task pass (pass^k) falls well below the single-run rate.
import { repeatRuns } from "@statelyai/agent/testing";
const report = await repeatRuns({
scenarios, // [{ id, ...your fields }]
k: 5,
run: async (scenario, attempt) => {
const trace: AgentTraceEvent[] = [];
const result = await runToQuiescence(
createAgentRuntime(machine, { executors, onTrace: (event) => trace.push(event) }),
{ input: scenario.input },
);
return { trace, outcome: String(result.snapshot.value) };
},
passed: (scenario, { outcome }) => outcome === scenario.expected,
});
report.scenarios[0];
// { id: "duplicate-charge", passes: 3, passAny: true, passAll: false, distinctPaths: 2, divergedAt: "deciding", runs: [...] }
report.passHat(1); // the single-run pass rate
report.passHat(5); // the chance 5 runs of a scenario all passpassAnyis pass@k for the scenario, andpassAllis pass^k.passHat(j)estimates the suite's pass^j without running more: per scenario, C(passes, j) / C(runs, j), averaged over scenarios. Scenarios with fewer thanjruns are left out. Ajthat is not a whole number from 1 to the most runs any scenario has throws anAgentErrorwith codepass-hat-out-of-range.distinctPathscounts the different step sequences the runs took.divergedAtis the earliest state where a run left the first run's path, found withfirstDivergence. A scenario that passes 3 of 5 runs and diverges atdecidingpoints at one decision.- Runs are sequential, scenario by scenario. A run that throws fails, with its message as
errorand no trace. report.tracesholds every recorded trace, ready forfoldTracesortraceCoverage.kbelow 1 throws anAgentErrorwith codeinvalid-repeat-count.
For runs collected elsewhere, such as CI shards or production replays, summarizeRepeats takes the runs per scenario and returns the same report. repeatRuns collects runs and calls it.
import { summarizeRepeats } from "@statelyai/agent/testing";
const report = summarizeRepeats([
{
id: "duplicate-charge",
scenario,
runs: [
{ passed: true, trace },
{ passed: false, trace: other },
],
},
]);Judge agreement
judgeAgreement (from @statelyai/agent/testing) measures how well a judge agrees with people on the same cases. Check it before a judge-graded eval gates anything.
import { judgeAgreement } from "@statelyai/agent/testing";
const report = judgeAgreement(
confirmed.map((row) => ({
id: row.id,
human: row.label,
judge: verdicts[row.id],
group: row.state,
})),
{ positive: "fail" },
);
report.accuracy; // 0.92
report.binary; // { tpr: 0.8, tnr: 0.95 }
report.perLabel; // { fail: { human: 10, agreed: 8 }, pass: { human: 40, agreed: 38 } }
report.correct?.(0.3); // the estimated true share of failures where the judge flagged 30%
report.groups?.deciding; // the same numbers for one decision state- A row is
{ id, human, judge, group? }. Build rows from confirmed labels only. - The report has
total,accuracy,perLabel(rows per human label, and how many the judge matched),confusion(counts by human label, then judge label), anddisagreementsin input order. positivenames the positive label, and every other label is negative. It addsbinary, the true positive rate and true negative rate. For a binary judge, these two rates say more than accuracy: they say which way the judge errs. A rate isundefinedwhen no row has that human label.correct(p)turns the sharepthe judge labels positive on unlabeled runs into an estimate of the true share: (p + TNR − 1) / (TPR + TNR − 1), clamped to 0..1. It is omitted when either rate isundefined, or when TPR + TNR ≤ 1, where the judge is no better than chance.groupsholds the same numbers per group, present when any row has agrouporgroupByis given.groupByoverridesrow.group. Group by decision state or request name, so each narrow judge is aligned on its own. A row with no group counts in the totals and in no group.- Run it on a dev split, and report the held-out split separately.
- Empty
rowsthrows anAgentErrorwith codeempty-agreement-rows.
Verification
simulateAgent, explorePaths, canReach, and matchesTrajectory operate on the machine artifact. Unknown state targets and empty expected trajectories fail loudly instead of producing misleading successful scores.