Generating machines with an LLM
Have a model author the agent workflow itself, then run the generated config through every pre-run gate before a single model call runs inside it.
Alpha:
@statelyai/agent2.0 is in alpha. APIs can change between releases; pin an exact version. Feedback: github.com/statelyai/agent.
This page covers having a model author an agent machine config, then checking that config with verifyAgentConfig before it runs.
An agent machine is data, so a model can write one. A generated machine can be checked before it runs, at several gates.
- The config's shape is fixed by a JSON Schema that ships with the package. Expressions are dot paths, not code, so a generated config cannot do anything a hand-authored machine could not do.
verifyAgentConfigruns every pre-run gate in one call and returns a record per gate. It is exported from@statelyai/agent/validate, which usesajvas an optional peer dependency, so installajvto use it.- The gates check the config's shape, the names it references, the lowering, Agent lint, and every explored path. Use
canReachandexplorePathsdirectly for other reachability questions. - Replay against real scenarios and scoring are domain-specific. They stay with the caller, after the gates pass.
The loop
generate → verifyAgentConfig (validate → vocabulary → lower → lint → explore) → run| Gate | Checks with | Fails on |
|---|---|---|
validate | validateAgentConfig | malformed config shape |
vocabulary | the allow lists | an invoke src, guard, action, agent.decide event, or model ref outside the vocabulary |
lower | setupAgent.fromConfig | unresolved named guards/actions, onDone on a decision |
lint | lintAgentMachine | decisions with no handled events; warnings too with lint: { warnings: true } |
explore | explorePaths, once per explore.inputs item | a path ending outside explore.terminals, a loop past maxDepth, a maxPaths/maxNodes cap, untried payloads (unexpandedEvents) |
Run the gates on the config the model returned:
import { verifyAgentConfig } from "@statelyai/agent/validate";
const result = await verifyAgentConfig(candidate, {
compileSchema,
guards: { isLarge: ({ context }) => context.amount > 100 },
allow: {
actors: [],
requests: ["draftReply"],
guards: ["isLarge"],
actions: [],
events: ["APPROVE", "DENY"],
models: ["openai/gpt-6-luna"],
},
explore: {
inputs: [{ amount: 5 }, { amount: 500 }],
outputs: { draftReply: [{ reply: "stub" }] },
},
});
if (!result.ok) {
const failed = result.gates.find((gate) => !gate.ok)!;
// Feed this back to the model as a repair prompt.
feedback.push(`gate '${failed.name}' failed: ${failed.detail}`);
}compileSchema is the JSON Schema compiler fromConfig uses for the config's schemas. Build it from ajv. See Machines as data.
The result behaves as follows.
result.gateslists the gates that ran, in order. Each is{ name, ok, detail? }. The gates stop at the first failure, so a failing gate is always the last entry.detailnames the offending path, state, or name, one problem per line. A passinglintgate carries its warnings indetail.result.configis the typed config oncevalidatepasses.result.machineis the lowered machine oncelowerpasses.- A config that throws while lowering is reported as a failed
lowergate.verifyAgentConfigdoes not throw on a bad config. - An omitted
allowkey is unrestricted. Omitallowto skip thevocabularygate, and omitexploreto skip theexploregate. An emptyexplore.inputslist fails theexploregate. - An invoke can name only an
agent.*builtin or a key the config declares underrequestsoractors. Any other src fails thelowergate, even whenactorsbinds it.allow.requestslimits the declared request keys, andallow.actorslimits the declared actor keys. Withallow.actorslisted, every non-builtin src must be inallow.actorsorallow.requests, or be a declared request whenallow.requestsis omitted. - With
allowgiven, a raw xstate expression slot ({ "@expr": ... }) anywhere in the config fails thevocabularygate. xstate evaluates the slot in place of its literal siblings, so it could hide a model ref orallowedEvents. Write{{ }}templates instead. explore.terminalsdefaults to['done']. Otherexplorekeys, such asoutputs,errors,payloads,dedupeBy,maxPaths, andmaxNodes, pass through toexplorePaths. Script text requests and invoked actors withoutputs. See Payloads and invoke outputs.actorsare bound withmachine.provide({ actors })after lowering.guardsandactionsgo tofromConfig.- No gate needs an API key.
validateAgentConfig(config)runs thevalidategate alone. It returns{ valid, errors }, and each error'spathis a JSON Pointer into the config.
The repair loop
A failed gate's detail is the repair prompt. Send it back to the model with the config it rejected, and run the gates again on the new config.
- Cap the repair loop at about 3 attempts. A config that fails three schema-shaped repairs usually asks for something the data form cannot express. In that case, author it in TypeScript instead.
- Replay the passing config against your own scenarios before promoting it.
simulateAgentplays one scripted path with canned responses and no API key.
The skill
The full authoring procedure ships as an agent skill. It covers the environment and policy split, the schema location, the rules a config must follow, a few-shot config, the Ajv adapter, verifyAgentConfig, the repair loop, and persisting traces.
skills/generate-machine/SKILL.md
Copy that directory into your coding agent's skills directory. For Claude Code this is .claude/skills/. Other harnesses use their own location. The agent then loads the skill on requests such as "author an agent machine for X" and runs the gates itself.
To improve a generated policy from its run traces, use skills/improve-machine. See Environment and policy.
Limits
Generated machines have these constraints.
- Named guards and actions are resolved by the host. A config cannot carry functions. A model can emit
guard: "isReady", but the implementation lives in theguardsoption passed tofromConfig(...), and an unresolved name throws at build time. List the names your host implements in the prompt, or forbid named references entirely. - Schema validity is not semantic validity. The schema constrains shape, not meaning. A config can validate and still reference a request that does not exist, or route to a state that does not make progress. The
vocabulary,lint, andexploregates cover those cases. - Exploration covers the inputs you give it. List inputs that reach every branch you care about in
explore.inputs. - Model refs resolve at run time, in the host.
allow.modelsrestricts them to literal strings you list; tool names stay unchecked. Generated machines carry string refs, so passcreateAiSdkExecutors({ resolveModel }). See Models and providers. The ref"openai/gpt-6-luna"used on this page is illustrative. Substitute your provider's current models.
Related
- Machines as data: the config format, its expressions, and its limits.
- Testing and verification: every check and simulation API in full.
- examples/json-agent: a hand-written config, run.