A durable agent loop in ~7100 lines of dependency-free Python. Built to make one
argument concrete: the while loop was never the hard part. It is four lines
here. Everything else — the log, the approvals, the escalation channel, the
permission boundary, the sandbox — is what actually decides whether an agent
survives production.
See also: proof/ — a regulatory-comprehension
artifact built on this loop. What it does and does not prove is in
proof/LIMITS.md.
The root directory can be called anything — steploop, my_ai_agent, whatever
your existing project is named. The src/ layout means the directory name has
no bearing on the import name; pyproject.toml declares where the package
lives. What matters is that these five things sit side by side at the top:
<any-name>/
pyproject.toml uv.lock Makefile README.md
.python-version .gitignore
src/steploop/ the package
tests/ evals/ examples/ scripts/
Do not nest it inside another Python project, and do not create a directory
called steploop next to src/ — a second one on the path is the one class of
mistake the src/ layout is there to prevent, and re-introducing it by hand
defeats it.
cd <any-name>
git init # the archive ships without history
uv sync --all-groups --extra anthropic # .venv, the package, and the dev tools
make doctor # says whether the checkout is wired up correctly
make check # lint, types, 377 tests, examples, eval self-testUnpack with unzip or tar xzf, not a GUI unarchiver: several of those drop
dotfiles, and losing .python-version means uv builds the environment on
whatever Python is default instead of the pinned one. make doctor reports it.
make doctor is there because a broken setup does not announce itself — it
shows up as "unresolved reference" in an editor, or as a test suite that passes
for the wrong reason. It checks the running interpreter is the project .venv,
that the package is genuinely installed rather than found because the working
directory happened to be right (it imports it from a temp directory to prove
it), that .python-version matches what is running, and that every entry point
starts. Each failure prints the command that fixes it.
- Open the root directory, not
src/and not a parent folder containing it. - Settings → Project → Python Interpreter → Add → Existing → select
<root>/.venv/bin/python. If PyCharm offers to create its own environment, decline: you would end up with two, and "works in the terminal, not in the IDE" follows. - Do not mark anything as Sources Root. The package is installed into the
venv, so imports resolve through it. Marking
src/orsteploop/tells PyCharm that directory is the root and its own subpackages stop resolving. - If red underlines persist after step 2: File → Invalidate Caches and Restart. PyCharm keeps an index from before the interpreter changed.
- Run configurations: use module
evals, not the fileevals/cli.py. Running a file inside a package puts that directory onsys.pathinstead of the root. The examples carry a guard for this; the eval runner is reached throughpython -m evals.
If PyCharm made its own environment on a different Python than
.python-version says, make doctor will tell you — that mismatch is two
environments drifting apart, and it is the most common cause of confusing
behaviour here.
Managed with uv. The lockfile is committed, so a fresh checkout reproduces the exact environment.
The distribution is called steploop because the obvious name was already
taken on PyPI by an unrelated project, which is worth checking before settling
on one: a colleague running uv add <name> would otherwise install a stranger's
package and the imports would resolve to it.
In an IDE, point the interpreter at .venv/bin/python after uv sync.
Without it, steploop is not importable and every symbol in every file shows as
unresolved — the code is fine, the editor simply cannot see the environment.
Open the repository root, not a subdirectory, and do not mark evals/ or
steploop/ as a sources root: doing so tells the IDE those directories are
the root, and their own packages stop resolving.
Keep .python-version and the IDE interpreter on the same Python. If the IDE
creates its own environment on a different version, the terminal and the editor
end up with two environments that drift, and "works in the terminal, not in the
IDE" appears for no visible reason. The suite is verified on 3.12 and 3.13.
uv sync # venv + dev group, from uv.lock
uv sync --all-groups --extra anthropic # dev tools and the SDK
make doctor # verify the checkout
uv run pytest # 377 tests, ten seconds, no network
make check # ruff + mypy + tests# scripted demos -- no API key, no cost
uv run python examples/basic.py --db demo.db # code-as-action and one approval
uv run python examples/full_stack.py --serve # pool + console + both HITL channels
uv run python examples/subagents.py # measures the context firewall
# a real run over your own files
uv run python examples/analyst.py --seed-data # sample dataset under ./data
uv run python examples/analyst.py --fake "..." # check the wiring for free first
export ANTHROPIC_API_KEY=sk-ant-...
uv run python examples/analyst.py "Which building is losing the most monthly income, and why?"
uv run python examples/analyst.py "..." --subagents --serve
uv run steploop --db local.db inbox # the console script
uv run steploop --db local.db code <run> # tools the model wrote
uv run steploop --db local.db tree <run> # the subagent treeThe runtime has no dependencies — the store is SQLite, the loop and the HTTP
server are asyncio, the sandbox is subprocess. anthropic is an extra rather
than a dependency because the entire test suite runs without it. Dev tooling
lives in a PEP 735 [dependency-groups] block, so it never reaches anyone who
installs the package.
Keep the database on a local disk. SQLite's WAL mode needs shared-memory mapping, which network and fuse mounts do not provide; the store falls back to rollback journalling if WAL is refused, but a Dropbox or NFS path will be slow and is not worth it.
examples/analyst.py gives the agent three tools over a directory — list, read, and a gated
write — and nothing else. Everything computational it has to author itself,
which is the point: you get to watch it write a tool, run it in the sandbox and
throw it away. --fake swaps a scripted model in behind the same wiring, so the
plumbing can be checked without a key or a bill.
Formatting is not enforced. ruff format would rewrite most of the file set and
the layout here is deliberate; make fmt is there if you disagree.
proof/ is a published run against three real EU regulatory
documents (MiCA Travel Rule, DORA register ITS, MiCAR record-keeping RTS),
where every modelled field carries a quote a reader can search for in the
official text at Cellar — no trust required. make proof-verify re-hashes
the pinned bytes against the manifest, make proof reproduces the run
against your ANTHROPIC_API_KEY.
1. The event log is the only source of truth. Append-only SQLite. The
conversation is never stored — it is derived by build_messages(events), a
pure function. Change the compaction strategy and old runs re-render under the
new one. Store messages as state instead and you are married forever to
whichever truncation policy was in force when the row was written.
2. One step() is the unit of durable work. It reads the log, does one LLM
turn plus its tool calls, writes back, returns. It holds nothing across calls.
A different process on a different machine can take the next step —
test_step_holds_no_state_between_calls builds a brand-new Agent for every
step and only the database carries continuity.
3. Approvals are data, not control flow. No input(), no coroutine parked
for three hours, no timeout to leak. approval_requested lands in the log, the
step returns, the process exits. A human clicks approve hours later,
approval_granted lands, a worker picks the run up. Audit, denial-with-reason,
approval timeouts and re-assignment all fall out for free because they are rows.
4. The model states intent; code decides permission. Every side-effecting
call goes through Policy.decide(call, run, spec) with the real run context. A
prompt is not a permission system. Deny matters as much as RequireApproval:
the agent gets a normal tool result explaining the refusal and adapts, rather
than crashing.
5. A human is called two ways, and the model controls only one of them. See "Escalation" below.
6. Tools the model needs but does not have, it writes. See "Code-as-action".
7. A subagent is a tool whose implementation is another agent loop. See "Delegation".
Instead of fifty narrow tools, ship four generic ones:
define_tool(name, code, description) run_tool(name, payload)
discard_tool(name) list_tools()
In the demo the agent has three read-only domain tools and no arithmetic at all. Asked for a rent roll, it writes one:
def run(payload):
flats = payload["flats"]
occupied = [f for f in flats if f["status"] == "occupied"]
return {"occupied_count": len(occupied),
"monthly_rent_roll": sum(f["rent"] for f in occupied), ...}Runs it, uses the result, discards it. The source lives in the log forever —
steploop.runtime.cli code <run> prints every tool the model ever wrote, marked live or
discarded. The live tool set is derived from tool_defined/tool_discarded
events, so replaying the log reconstructs it exactly.
Why this beats twenty endpoints: one authored tool replaces a dozen round-trips of JSON tool-calling, and the model composes loops and conditionals itself instead of driving them one call at a time through the context window.
make evals-dry # prove the harness can fail; no API key, no cost
make evals-list # what the set covers and why
make evals REPEATS=5 # the real thing
uv run python -m evals --tag comprehension --baseline last.jsonNote python -m evals, not python evals/cli.py. Launching a file inside a
package puts that directory on sys.path instead of the project root, so
import evals fails and every import in the package fails with it. evals is
therefore reached through __main__.py, which repairs sys.path when it is
launched any other way — an IDE's green Run button included — and the package
uses absolute imports throughout, because it is a project directory rather than
something installed into the environment.
Four decisions shape the set, and the first is the one that makes the rest possible.
Assert on the log, not just the answer. A final string tells you whether the agent arrived; the event log tells you how, and how is what this framework makes claims about. "Wrote its own tool", "surfaced the fork before spending anything", "never ran the gated tool without approval" are all checkable and none appear in the answer. The self-test includes an agent that gives the correct answer from memory with no tool, no arithmetic and no file read — answer-diffing scores it full marks; the log assertions fail it.
Synthetic data is the advantage, not the compromise. The fixtures are generated, so ground truth is computed alongside them. Nobody labels anything, nobody gets it wrong, and a fixture can be regenerated at ten times the size without re-deriving the answer. The cost is that generated data is tidier than real data, so each variant deliberately carries one specific mess: a foreign key with no row, a status whose accounting nobody has settled, a file that issues instructions to its reader.
Some cases are expected to fail. A set where everything passes measures
nothing but its own timidity. Cases carry an expectation, so a known-hard case
going green is reported as progress rather than vanishing into the noise, and a
case that was passing and stops is a REGRESSION rather than a bad day.
Repeats, not runs. Models are stochastic; one execution is an anecdote. Every case runs N times and the result is a rate, because 3/5 and 5/5 are different facts that a boolean flattens into "passes".
The set is four groups — capability, safety (written as negatives, because
"and nothing else happened" cannot be said positively), traps, and known-hard —
with a counterweight for every trap. trap.material_ambiguity_is_surfaced
requires the agent to ask; trap.no_ambiguity_no_interruption requires it not
to. Without the second, an agent that asks about everything scores perfectly.
An eval suite nobody has watched fail is indistinguishable from one that cannot
fail, so make evals-dry runs scripted agents with known behaviour and checks
the harness scores them differently. It found two real problems on its first
run.
The first attempt at "an agent that ignores the approval gate" scored 100% —
because the gate held: the run parked, the harness approved it, and the write
went through with approval. The framework working, not the eval working. The
failure worth detecting is a configuration regression, so the check now removes
the policy rule instead. That exposed the second thing: removing the explicit
rule still changed nothing, because Policy requires approval for external side
effects by default. Closed unless you deliberately open it — worth knowing,
and now worth the eval proving.
Nothing checks tone, structure or length, and most cases do not check exact wording. Those checks are easy to write, they fail for reasons nobody cares about, and that is how a suite becomes something people mute.
Everything else here guards the process. The budget catches "too expensive", the detectors catch "going nowhere", the policy catches "not allowed to", the contract catches "wrong shape", cancellation catches "no longer wanted". None of them catch the failure that matters most: the agent confidently understood the wrong task. That run passes every check and returns a fluent, well-sourced answer to a question nobody asked.
Two small phases close it, and neither of them is a plan.
Clarification, once, before any work. One cheap call that restates the task, names its own assumptions, lists what a complete answer must contain, and surfaces ambiguities. Not "what are the steps" — steps go stale the moment the first file is read, which is why explicit plan-and-execute was abandoned. This is the input contract, and an input contract does not go stale.
Acceptance, once, before finishing. One cheap call comparing the answer to
the original request. Not "did you follow the plan" but "does this answer the
question that was asked", with wrong_question as a verdict distinct from
incomplete — a misread and a gap need different repairs, and the repair for a
misread restates the original request.
A model asked "what is ambiguous here" will invent ambiguities; it is agreeable
and costs it nothing. So an ambiguity is only acted on if each reading states
concretely what different answer it produces (leads_to). If the model
cannot say what changes, nothing changes, and a person is not interrupted. That
one required field is most of what separates this from decoration.
On the sample dataset it works as intended. The data contains a deliberate fork
— units on notice, which finance has not decided whether to count as let — and
the phase surfaces it at step 0, before a single tool has run:
clarify.done criteria=3 assumptions=1 ambiguities=1 blocking=1
clarify.asking "Should units on notice count as let income or as lost?"
status awaiting_help steps 0 tools 0 $0.0050
A grounded ambiguity is routed down the existing escalation path rather than a parallel one, so it lands in the same operator inbox, obeys the same escalation budget and gets the same timeout behaviour.
Both phases cost one model call each, both are configurable, and both are off for subagents by default — a subagent's task was written by another machine, is already precise, and carries a return contract of its own. A failing or unparseable phase never blocks the work it was meant to protect.
The naive version of this feature is "give the model an ask_human tool". That
fails in three specific ways, and the design is shaped by all three.
Models are badly calibrated about being stuck. An agent that is confused often cannot tell it is confused — asking the confused thing to self-report confusion is unreliable in exactly the cases that matter. So there are two independent channels, and the model has a vote in only one:
| Channel | Trigger | Model's say |
|---|---|---|
| approval | policy saw a side effect worth a signature | none |
| help — agent | the model called ask_human |
it decides |
| help — detector | code observed the run going nowhere | none |
Detectors fire on observable trouble, never on self-report: N failing tool
calls in a row, identical tool results repeating, budget crossing 80% with the
task unfinished. In examples/full_stack.py the agent retries a dead meter API three
times and never thinks to ask; the detector asks for it.
"Help me" is not a question. An unstructured cry for help makes a human re-derive the whole context. The schema refuses a bare question — the model must supply concrete options, what it already tried, a recommendation and a confidence. A colleague should be able to answer by clicking, in ten seconds, without reading the transcript:
[79493abc] QUESTION (agent) Should a flat on notice count as occupied
for this month's rent roll?
[count] Count it as occupied
[exclude] Exclude it <- agent recommends
Free escalation is a lazy escape hatch. If asking costs nothing, a model
asks instead of thinking and you have rebuilt a human doing the work with extra
latency. So: a per-run escalation budget, rejection of duplicate questions by
normalised fingerprint, and a required record of prior effort. Every rejection
is phrased to teach — "escalation is for after an attempt, not instead of one" — so the model can fix it and retry rather than crash.
Every question carries a timeout and a declared action. APPLY_DEFAULT uses
the model's own recommendation, CONTINUE proceeds and notes the gap,
ABANDON stops the run. Without this the inbox silently fills with zombies. The
pool's sweeper applies them.
Non-blocking questions exist. blocking=false lets the run carry on with
everything that does not depend on the answer; the reply lands in context
whenever it arrives.
One reason survives scrutiny: a context firewall.
A tool's result lands in the parent's context in full. A subagent reads in a
context that is thrown away and hands back a summary. examples/subagents.py runs
the same task — review six maintenance reports, flag the urgent ones — both
ways and measures it:
approach parent context children held below
plain tools 15,442 0 0
subagents 1,927 6 17,978
Same answer, 8x smaller parent context. The 18,000 characters are still on
disk, still auditable via steploop.runtime.cli tree, simply never in the parent's
window. Everything else people claim about multi-agent systems — role personas,
"a team of specialists" — is decoration on this one mechanic.
The second real reason is narrower capability. A child is built from an explicit tool allowlist, so a researcher that may read but never spend simply has no spending tool in its schema list. That is a boundary in code, not a sentence in a prompt.
When not to delegate. If the subtask is deterministic, write a tool — a subagent costs a model loop per step. If the parent needs the details, the summary is exactly what you lose. And if two subtasks share implicit decisions, splitting them actively hurts: subagents cannot agree on assumptions they never articulated, which is why this works for search and exploration and fails for writing code against a shared interface. Delegate read-heavy, parallelisable work with large input and small output; refuse the rest.
A child is a run. Same store, parent_run_id set. So it inherits leases,
the worker pool, approvals, escalation and the audit log with no new machinery.
A child parking for a human parks its parent too, and the tree resumes when the
human answers the child. Guardrails: depth cap, fan-out cap, and the child's
spend charged back to the parent so delegation is not a way around its budget.
Two bugs the tests caught here, both worth knowing about:
Two results for one call. Delegation acknowledges immediately and answers
later. Writing a tool_result at both moments leaves one tool_use with two
results, which providers reject — and the 400 surfaces a step later, far from
its cause. Hence the DEFERRED sentinel: the acknowledging call writes nothing,
and the collected summary becomes the single result.
The fan-out cap that did nothing. Concurrent delegate calls in one turn all read the same pre-execution event snapshot, so every one of them believed it was the first and the cap never fired. The count and the create now happen together against live state, under a lock.
A parent that is abandoned — timed out, over budget, failed, or stopped by an operator — used to leave its subagents running. They burned budget and, worse, kept taking external actions for a task nobody was waiting on. Cancellation without a cascade is not cancellation.
Three properties shape it:
Cancellation is data, not a signal. A child may be mid-step on another
worker on another machine, so task.cancel() cannot reach it. A
cancel_requested event is appended to its log and it notices at its own step
boundary — cooperative, durable, cross-process, for the same reason approvals
are.
Status is not the flag. Five places in the loop write a run back to
RUNNING — collecting a finished child, answering an approval, resuming a held
call. Any could resurrect a run cancelled a moment earlier. So is_cancelled
reads the log, which only grows, and status stays what it always was: a cache.
The check sits in front of side effects. Noticing at the top of a step is
not enough: a step already under way could still dispatch a payment, because
the top-of-step check can be a whole model call stale. _execute re-checks
immediately before running anything. Between the two checks the exposure is
zero new external actions — test_a_cancel_landing_mid_step_still_blocks_the_tools
cancels the run from inside the model call to prove it.
Cascades fire automatically on every path a run can end by: an operator stop, a
budget breach, a fatal error, an abandoned escalation nobody answered, and a
parent that answered without waiting for a subagent it had started. cancel_tree
is breadth-first, skips runs that already ended rather than rewriting history,
and is safe to call twice.
Withdrawing what was asked of a person. Cancelling a run parked on an approval must take that approval back, or an operator is left answering a question about work that no longer exists and the console looks broken. Same for an open escalation.
The crash-safety net. A cascade can die halfway. EventStore.orphans finds
live runs whose parent already ended, and the pool's sweeper finishes the job.
Two bugs this work exposed, both found by tests written before the fix:
A parent blocked on a hung child waited for ever. The budget check sat below the child-parking branch, so a parent waiting on a subagent that never finished returned early every step and never consulted its own budget. The check now runs before parking on a child — but still below the approval and escalation waits, because a person's thinking time should not burn the clock.
The fan-out cap that did nothing — see Delegation above.
A step used to answer every question by loading the whole log and scanning it — five separate loads per step, each growing with the run, so a run's cost was quadratic in its own length. Measured:
events 5 full loads 1 tail + indexes
100 2.3ms 0.8ms
400 9.8ms 2.4ms
1600 39.6ms 9.6ms
6400 168.1ms 37.3ms
Two changes, different in kind. The tail is loaded once (RunLog) and kept
current as the step appends, so nothing re-reads. Rare questions go to an
index — "was this cancelled", "is an approval pending", "did this
tool_call_id already produce a result" — because some of them must consider
the whole log for correctness and a full scan each would put the quadratic
behaviour straight back. With compaction moving the watermark, per-step cost
stops growing with the run at all.
The pure functions in escalation, cancellation and subagent still exist as
the readable definition of what each query means; verify_agreement checks the
fast path against them, because an optimisation that disagrees with its
specification is a bug.
A bug the measurement exposed: load_tail excluded the compaction event itself,
so the summary standing in for all earlier history vanished with it. The obvious
off-by-one silently drops the past instead of condensing it.
The previous implementation was a placeholder that stitched bullet points with string slicing. Two mechanisms now, and they are not interchangeable.
Offloading happens at write time and is the one that matters. A tool returns
forty kilobytes; the model needs three lines. The body goes to an artifact store
and the conversation keeps a preview plus a reference; read_artifact pulls
back a slice, or the regions matching a string. Nothing is lost — the full text
is on disk and auditable — it simply stops being re-sent every turn. Artifacts
are scoped to the thread, so a parent can hand a reference to a subagent and
the body never passes through either context.
Summarisation happens at read time and is the fallback. LLMCompactor asks
a model for a digest and is told what a later reader will need: decisions that
still bind, facts with their numbers intact, what was already tried, what a
human approved. DeterministicCompactor needs no model call and is the fallback
when that fails.
Triggered on estimated tokens, not event count — five events carrying large results crowd a context far harder than fifty carrying one line each, and counting events gets that backwards.
The pool would fire fifty concurrent requests and collect fifty 429s. The breaker then opens, but only after the damage, and failing fast is the wrong response to a rate limit — that is not an outage, it is a queue.
Two buckets. Providers limit requests and tokens per minute, and for an agent the token limit binds first: a long context is one request but tens of thousands of tokens.
Reserve, then reconcile. Token cost is unknown until the reply arrives, so an estimate is reserved and corrected afterwards.
A 429 belongs to everyone. penalise gates the whole limiter, because
backing off only the rejected caller leaves the rest hammering a limit already
tripped and turns a one-second dip into a sustained outage.
Bug found here, and the test that finally caught it is worth the note: checking
and consuming were one operation, so a request blocked on tokens had already
spent the request budget, and the refund that followed put tokens back into the
bucket that had refused. Point assertions about bucket contents miss this —
an inflated bucket still makes the current caller wait and only over-grants
later. test_the_rate_is_never_exceeded_over_a_window counts grants against
what was actually earned, and catches it.
Tenant scoping lives in the store, not in call sites. Passing a tenant into
every query is how leaks happen: one method added later forgets the filter and
nobody notices until an incident. store.for_tenant("acme") returns a handle
that cannot reach anything else, sharing the same connection so switching
tenants is not a reopen. load_run raises TenantMismatch rather than
returning None, because a silent empty result gets handled as a 404 and hides
the fact that a boundary was crossed. as_admin() is unscoped and deliberately
awkward to construct.
SubagentSpec.returns was a sentence in a prompt — a wish. A spec may now
declare return_schema, and the child's final answer has to satisfy it. A
failure is handed back naming exactly what broke and the child tries again;
only after the repair budget is spent does the parent hear about it, and then as
an explicit "unusable", not a plausible-looking summary. The validator is fifty
lines covering type, required, properties, enum and array items — reaching for
jsonschema to check five keys would be the wrong trade. Extraction is
forgiving about fences and preamble, strict about content.
Idempotency-Key on any POST replays the stored response instead of acting
twice, so a flaky network or a client with its own retry loop cannot start two
runs or approve a spend twice. The request fingerprint is stored alongside, so
reusing a key for a different body is a 409 rather than something silently
honoured. Records are per-tenant and expirable.
Claim under a lease, heartbeat while you hold it, release when done, and let a
crashed worker's lease expire so someone else takes over. That last property is
only safe because tool execution is idempotent by tool_call_id — otherwise
lease expiry would mean duplicated side effects.
No broker. SQLite plus a lease column is enough for a few hundred runs a second
and keeps the claim transactional with the event append. Swap EventStore.claim
for SELECT ... FOR UPDATE SKIP LOCKED on Postgres and nothing else changes.
A bug the tests caught and worth repeating: check_same_thread=False permits
cross-thread use of a SQLite connection but does not make it
concurrency-safe. Two threads calling execute() at the same instant raise
InterfaceError: bad parameter or other API misuse. Every statement is
serialised behind one lock in store.py.
asyncio.start_server, no framework, no dependency. A JSON API plus a small
console where approvals and questions land in one inbox with clickable options.
GET /api/inbox everything waiting on a human
POST /api/runs/{id}/approve {"note": "..."}
POST /api/runs/{id}/deny {"reason": "..."}
POST /api/runs/{id}/answer {"option_id": "repair", "text": "..."}
POST /api/runs/{id}/cancel {"reason": "..."} — stops the whole subtree
When it earns its place, and when it does not. Everything the console does,
the CLI does — inbox, approve, deny, answer, cancel — so a developer
in a terminal never needs it. It exists for the person who is not in a
terminal: the operations colleague who will click "Repair first" but will not
type steploop answer 4928a38b --actor you@example.com --option repair. The
inbox is the other half: every parked run across every thread in one place, with
each option showing what it leads to.
examples/analyst.py used to start it on every invocation, which bound a port
and printed a URL for one-shot terminal runs that had no use for either, and
made two concurrent runs collide. It now starts only with --serve, the port
defaults to 0, and a run that parks prints the CLI command that unsticks it.
The HTML page is a reference client. The JSON API is the surface you would actually build against, and the page exists to prove it is usable.
It is an ops surface, not a public API: no auth, no TLS. Put it behind your
authenticating proxy and have the proxy set X-Actor — the actor is written
into the log and is the entire point of an audit trail. The header beats the
request body, so a caller cannot claim to be someone else.
sandbox.py is defense in depth against a confused model, not a security
boundary against an adversary.
It reliably stops the realistic failure modes, each covered by a test: runaway
loops (rlimit CPU + wall clock + process-group kill), memory blowups (rlimit
AS), network access (import blocklist via sys.meta_path, plus socket
neutered), writes outside the scratch dir (guarded open), subprocess spawning,
eval/exec/__subclasses__ traversal, and credential leakage (scrubbed env —
test_secrets_do_not_leak_into_the_child asserts ANTHROPIC_API_KEY is
invisible inside the child).
It does not stop a determined attacker. CPython segfaults, recursion-limit
abuse and kernel exploits are out of scope. If anything untrusted can influence
the prompt that produces this code, run it in a container with seccomp, a
read-only rootfs and no network namespace. Sandbox._spawn is deliberately
isolated so swapping in gVisor or Firecracker is a local change.
ResilientLLM composes a circuit breaker inside a retry loop:
retry( breaker( call ) )
Every physical attempt is counted, so the breaker opens fast under a real
outage. CircuitOpenError is non-retryable, so once it trips the retry loop
aborts instead of burning its remaining attempts on a dead dependency. The
reverse nesting sees one aggregated failure per logical call and opens ~n times
slower. test_open_breaker_aborts_the_retry_loop_immediately pins this: two
physical calls, not ten.
Other details that matter in practice:
- Full jitter (
uniform(0, backoff)), so concurrent workers decorrelate instead of re-colliding in lockstep. Retry-Afterwins over computed backoff when the provider sends one.- A 400 never trips the breaker. A malformed request is our bug, not the provider's outage; counting it would take down healthy traffic.
- Unknown exceptions are bugs, not faults — never retried.
Safety rails in the loop itself: token/cost/step/wall-clock budgets, duplicate
detection by tool_call_id (makes at-least-once delivery and mid-step crashes
safe), and loop detection that nudges at 2 identical calls and refuses at 4.
src/steploop/
events.py the log vocabulary
store.py append-only SQLite store, leases, artifacts
observability.py structured logging
agent/ the loop and everything that shapes what it sees
loop.py context.py runlog.py comprehension.py compaction.py
tools/ what the agent can do
registry.py sandbox.py contract.py subagent.py
control/ what it is allowed to do, and who decides
policy.py escalation.py cancellation.py
llm/ reaching the model reliably
client.py resilience.py ratelimit.py
runtime/ running many of them, and looking at them
worker.py httpd.py cli.py
evals/ golden set, generated fixtures, harness
examples/ runnable scripts and their demo domain
tests/ mirrors the package
src/ layout on purpose: the package is not importable from the repository
root, so the tests exercise the installed copy. Without it, a module missing
from the packaging config passes locally and fails the moment anyone installs
it. events.py, store.py and observability.py stay at the top because
everything depends on them and nesting would only add a hop.
Worth recording, because it says something about what test suites do not cover.
The "prefer acomplete, else await, else thread" dispatch was written three
times — in the agent loop, in the compactor, and inside ResilientLLM — and the
copy in ResilientLLM was missing the acomplete branch. Every one of 320
tests passed. It surfaced on the first invocation of examples/analyst.py, because that was
the first time a rate limiter was wrapped around an async-only client, and the
wrapper reached for a synchronous complete that did not exist.
Duplicated dispatch drifts. There is now one call_llm, and a parametrised test
over all three client shapes.
The Anthropic adapter had the same problem in a milder form: written early,
exported, documented, and never executed. Exercising it against a stubbed SDK
confirmed the response mapping and the error taxonomy — including 529
"overloaded" landing on ServerError rather than BadRequest, which is the
difference between retrying an outage and giving up on it.
Escalating is the easy half. The half that gets skipped is the return.
BOT the agent answers
ESCALATED the agent has stopped; nobody has picked it up
HUMAN an operator is writing; the agent is silent
BOT handed back; the agent resumes
The agent is muted, not paused. Pausing implies resuming from where it stopped; by then the conversation has moved.
What the operator promised is binding. A colleague who wrote "we'll refund
you by Friday" has committed the company, and the bot may not later explain
that refunds are not offered. Operator turns render as authority rather than as
more user input, with a standing instruction that survives a second handover.
test_the_binding_instruction_reaches_the_model pins it.
Handback is not a return value. interrupt()-style primitives model "stop
and receive one answer"; here a person takes the wheel for several turns and
gives it back.
Detectors split the same way as everywhere else: observed (asked for a person, two empty retrievals, three turns on the same subject, a regulated topic) outranks self-reported (the bot says it cannot help). In support the observed half carries more weight — a frustrated customer does not announce that they are stuck, they repeat themselves and get shorter. Priority is the order: asking for a person outranks every heuristic, and a regulated topic outranks a bot that thinks it is doing fine.
brief_for_operator produces what the colleague needs — why it came to them,
the opening question, the last thing said, what the bot already tried — because
an escalation that requires reading forty messages is moving work sideways
rather than escalating it.
This lives beside the agent, not inside it. In a support chat the durable thing is the conversation, hours or days long; one agent turn is seconds. The heavy durable-execution machinery solves a problem this does not have.
Two different things go by that name, and conflating them is why "does it stream" is a bad question.
Progress — "reading units.csv", "wrote a tool", "waiting on a colleague". Nearly free here: the run already records each of these in the log, so streaming is a second subscriber to writes that happen anyway. This is the kind that matters in a messenger, where a message arrives whole and token-by-token delivery means nothing, but four seconds of silence reads as a dead bot.
Tokens — the characters of the answer as they generate. Worth something only when a person is watching a text area fill in.
config = AgentConfig(stream=StreamPolicy.for_messenger()) # progress only
config = AgentConfig(stream=StreamPolicy.for_chat_ui()) # both
agent = Agent(..., sink=CallbackSink(send_to_messenger))Two switches, both off by default, and a sink alone does not turn either on — an agent with streaming off does no extra work and takes no extra branch. A client that cannot stream simply produces no token events; turning tokens on is never a hard requirement on the adapter.
Tokens never reach the log. They are shown before the step commits, so a crash mid-generation would otherwise leave a half-written sentence in an append-only record — and put an unfinished thought back into the context on resume. Streamed text is ephemeral, the log gets the complete message or nothing, and the retry re-streams.
GET /api/runs/{id}/stream is server-sent events, so a listener can follow a
run the worker pool owns. Progress events are advisory, so a slow reader loses
the oldest rather than applying backpressure to the agent.
Server.close() took thirty seconds with a stream open. Three causes, each
worth knowing:
asyncio's own serve_forever() awaits every handler when cancelled, and a
stream is a handler that will not end on its own. But start_server already
accepts connections — the extra task only existed to keep a script alive, and
the tests did not need it at all.
writer.wait_closed() can hang on a half-closed peer, so an unresponsive
client could hold shutdown open. It is now bounded.
And a stream over a run that has ended is just a held socket, so the pump stops on a terminal event rather than sending keep-alives for ever.
The core is async: astep/adrive, asyncio.gather for parallel tools, a
native asyncio.create_subprocess_exec sandbox driver, an asyncio.start_server
HTTP layer. step()/drive() remain as sync facades that refuse to nest inside
a running loop rather than deadlocking quietly.
The bug this nearly shipped with. AnthropicLLM.complete is synchronous,
and the loop called it inline from a coroutine. Nothing in the signatures looks
wrong, and the whole test suite passed — because every test used an instant
fake. Against a real API, one run waiting thirty seconds on the model would
have frozen every other run in the pool and the HTTP console for thirty
seconds. tests/test_async.py is the answer: every assertion is a wall-clock
measurement, because "is this async" cannot be read off a type signature.
Reintroduce the old one-line version of _call_llm and four of those tests go
red — parallel throughput, loop responsiveness, path selection, and console
latency under load.
The model call now takes one of three paths, in order: acomplete if the
client offers it (ResilientLLM does, and AsyncAnthropicLLM under it), await
if complete is itself a coroutine function, asyncio.to_thread otherwise.
Retry backoff in acomplete waits on asyncio.sleep — time.sleep inside a
coroutine would have reintroduced the same freeze during the backoff instead of
during the request.
Sync handlers go to a thread, so one blocking database call in a tool cannot stall the other tools in the same step.
What is still synchronous, deliberately. SQLite is called inline from
coroutines. An append measures well under a millisecond, so wrapping every one
in asyncio.to_thread would cost more in scheduling than it saves — and
test_store_calls_are_synchronous_and_that_is_a_deliberate_choice pins that
assumption with a measurement rather than leaving it as a belief. On a slow
disk, or behind Postgres over a network, that trade flips and the store needs
an async driver.
377 tests, ten seconds, no network. The ones that carry the argument:
test_step_holds_no_state_between_calls a fresh Agent per step
test_no_process_is_required_while_awaiting_approval
test_completed_tool_calls_are_never_re_executed idempotency by tool_call_id
test_runs_row_is_a_faithful_projection_of_the_log
test_concurrent_appends_do_not_collide 4 threads, 40 events, no holes
test_open_breaker_aborts_the_retry_loop_immediately
test_runtime_import_of_a_blocked_module_is_refused bypasses the static screen
test_secrets_do_not_leak_into_the_child
test_denial_is_fed_back_as_a_tool_result or the provider 400s
test_model_authors_runs_and_discards_its_own_tool
test_detector_forces_a_question_the_model_never_asked
test_lazy_escalations_are_rejected_with_a_teachable_message (8 cases)
test_the_same_question_cannot_be_asked_twice
test_escalation_budget_is_enforced
test_healthy_run_is_never_interrupted_by_a_detector
test_non_blocking_question_lets_the_run_continue
test_timeout_applying_the_default_resumes_the_run
test_claim_is_exclusive two workers, one winner
test_expired_lease_is_reclaimable crash recovery
test_every_run_is_processed_exactly_once 10 runs, 5 workers
test_runs_actually_execute_concurrently wall-clock assertion
test_a_crashing_run_does_not_kill_the_worker
test_actor_header_wins_over_the_body audit integrity
test_the_childs_bulk_never_reaches_the_parents_context
test_a_child_cannot_call_a_tool_it_was_not_given
test_child_spend_is_charged_to_the_parent
test_a_child_waiting_on_a_human_parks_the_whole_tree
test_one_tool_use_yields_exactly_one_tool_result
test_fan_out_is_capped_per_run
test_a_sync_llm_client_does_not_freeze_the_loop
test_the_event_loop_stays_responsive_during_a_model_call
test_the_http_server_answers_while_a_run_holds_the_model
test_resilient_llm_backoff_does_not_sleep_the_loop
test_a_blocking_tool_handler_does_not_stall_its_siblings
test_the_sync_facade_refuses_to_nest_inside_a_running_loop
test_a_cancelled_run_performs_no_further_side_effects
test_a_cancel_landing_mid_step_still_blocks_the_tools
test_a_cancelled_run_cannot_be_brought_back_by_a_status_write
test_a_parent_blocked_on_a_hung_child_still_runs_out_of_budget
test_cancelling_withdraws_a_pending_approval
test_the_sweeper_reaps_orphans_left_by_a_half_finished_cascade
test_the_rate_is_never_exceeded_over_a_window
test_a_429_pauses_every_caller_not_just_the_unlucky_one
test_reaching_another_tenants_run_raises_rather_than_returning_nothing
test_the_worker_pool_never_claims_another_tenants_work
test_an_unrepairable_answer_is_reported_as_unusable_not_summarised
test_a_repeated_post_does_not_start_a_second_run
test_an_ambiguity_whose_readings_change_nothing_is_not_grounded
test_a_grounded_ambiguity_parks_the_run_before_any_work
test_an_answer_to_the_wrong_question_is_sent_back
test_the_retry_budget_is_bounded_and_the_shortfall_is_logged
test_an_unparseable_brief_does_not_stop_the_work
Not a framework to adopt. It is the ~1400 lines worth writing before choosing between your own code, LangGraph and Temporal — because after writing them you will know which one you actually need.
Still absent, in rough order of how much they would be missed in production:
a real cost model — Usage.cost_usd is supplied by the caller, so budgets
rest on a number nobody computes; PII redaction and
retention — the log keeps everything verbatim for ever; metrics — logs
only, so nothing can be graphed or alerted on; streaming; Postgres; and
a sandbox that is a real boundary rather than defence in depth.
For long-running human-in-the-loop workflows specifically, look hard at
Temporal before building your own durability layer. It solved this in 2019,
it is battle-tested on far more than LLM workloads, and its replay model lets
you keep an ordinary while loop instead of restructuring into a graph.