Skip to content
intellectumlabPublic

About

A durable agent loop in ~7100 lines of dependency-free Python, plus a public regulatory-comprehension artifact against three EU documents (MiCA, DORA, MiCAR). Check us, don't trust us.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

steploop

License: Apache 2.0 proof-verify

A durable agent loop in ~7100 lines of dependency-free Python. Built to make one argument concrete: the while loop was never the hard part. It is four lines here. Everything else — the log, the approvals, the escalation channel, the permission boundary, the sandbox — is what actually decides whether an agent survives production.

See also: proof/ — a regulatory-comprehension artifact built on this loop. What it does and does not prove is in proof/LIMITS.md.

Setting it up locally

The root directory can be called anything — steploop, my_ai_agent, whatever your existing project is named. The src/ layout means the directory name has no bearing on the import name; pyproject.toml declares where the package lives. What matters is that these five things sit side by side at the top:

<any-name>/
    pyproject.toml   uv.lock   Makefile   README.md
    .python-version  .gitignore
    src/steploop/    the package
    tests/  evals/  examples/  scripts/

Do not nest it inside another Python project, and do not create a directory called steploop next to src/ — a second one on the path is the one class of mistake the src/ layout is there to prevent, and re-introducing it by hand defeats it.

cd <any-name>
git init                      # the archive ships without history
uv sync --all-groups --extra anthropic    # .venv, the package, and the dev tools
make doctor                   # says whether the checkout is wired up correctly
make check                    # lint, types, 377 tests, examples, eval self-test

Unpack with unzip or tar xzf, not a GUI unarchiver: several of those drop dotfiles, and losing .python-version means uv builds the environment on whatever Python is default instead of the pinned one. make doctor reports it.

make doctor is there because a broken setup does not announce itself — it shows up as "unresolved reference" in an editor, or as a test suite that passes for the wrong reason. It checks the running interpreter is the project .venv, that the package is genuinely installed rather than found because the working directory happened to be right (it imports it from a temp directory to prove it), that .python-version matches what is running, and that every entry point starts. Each failure prints the command that fixes it.

PyCharm

  1. Open the root directory, not src/ and not a parent folder containing it.
  2. Settings → Project → Python Interpreter → Add → Existing → select <root>/.venv/bin/python. If PyCharm offers to create its own environment, decline: you would end up with two, and "works in the terminal, not in the IDE" follows.
  3. Do not mark anything as Sources Root. The package is installed into the venv, so imports resolve through it. Marking src/ or steploop/ tells PyCharm that directory is the root and its own subpackages stop resolving.
  4. If red underlines persist after step 2: File → Invalidate Caches and Restart. PyCharm keeps an index from before the interpreter changed.
  5. Run configurations: use module evals, not the file evals/cli.py. Running a file inside a package puts that directory on sys.path instead of the root. The examples carry a guard for this; the eval runner is reached through python -m evals.

If PyCharm made its own environment on a different Python than .python-version says, make doctor will tell you — that mismatch is two environments drifting apart, and it is the most common cause of confusing behaviour here.

Running it

Managed with uv. The lockfile is committed, so a fresh checkout reproduces the exact environment.

The distribution is called steploop because the obvious name was already taken on PyPI by an unrelated project, which is worth checking before settling on one: a colleague running uv add <name> would otherwise install a stranger's package and the imports would resolve to it.

In an IDE, point the interpreter at .venv/bin/python after uv sync. Without it, steploop is not importable and every symbol in every file shows as unresolved — the code is fine, the editor simply cannot see the environment. Open the repository root, not a subdirectory, and do not mark evals/ or steploop/ as a sources root: doing so tells the IDE those directories are the root, and their own packages stop resolving.

Keep .python-version and the IDE interpreter on the same Python. If the IDE creates its own environment on a different version, the terminal and the editor end up with two environments that drift, and "works in the terminal, not in the IDE" appears for no visible reason. The suite is verified on 3.12 and 3.13.

uv sync                                # venv + dev group, from uv.lock
uv sync --all-groups --extra anthropic # dev tools and the SDK
make doctor                            # verify the checkout
uv run pytest                          # 377 tests, ten seconds, no network
make check                             # ruff + mypy + tests
# scripted demos -- no API key, no cost
uv run python examples/basic.py --db demo.db     # code-as-action and one approval
uv run python examples/full_stack.py --serve    # pool + console + both HITL channels
uv run python examples/subagents.py        # measures the context firewall

# a real run over your own files
uv run python examples/analyst.py --seed-data       # sample dataset under ./data
uv run python examples/analyst.py --fake "..."      # check the wiring for free first
export ANTHROPIC_API_KEY=sk-ant-...
uv run python examples/analyst.py "Which building is losing the most monthly income, and why?"
uv run python examples/analyst.py "..." --subagents --serve

uv run steploop --db local.db inbox    # the console script
uv run steploop --db local.db code <run>   # tools the model wrote
uv run steploop --db local.db tree <run>   # the subagent tree

The runtime has no dependencies — the store is SQLite, the loop and the HTTP server are asyncio, the sandbox is subprocess. anthropic is an extra rather than a dependency because the entire test suite runs without it. Dev tooling lives in a PEP 735 [dependency-groups] block, so it never reaches anyone who installs the package.

Keep the database on a local disk. SQLite's WAL mode needs shared-memory mapping, which network and fuse mounts do not provide; the store falls back to rollback journalling if WAL is refused, but a Dropbox or NFS path will be slow and is not worth it.

examples/analyst.py gives the agent three tools over a directory — list, read, and a gated write — and nothing else. Everything computational it has to author itself, which is the point: you get to watch it write a tool, run it in the sandbox and throw it away. --fake swaps a scripted model in behind the same wiring, so the plumbing can be checked without a key or a bill.

Formatting is not enforced. ruff format would rewrite most of the file set and the layout here is deliberate; make fmt is there if you disagree.

The proof artifact

proof/ is a published run against three real EU regulatory documents (MiCA Travel Rule, DORA register ITS, MiCAR record-keeping RTS), where every modelled field carries a quote a reader can search for in the official text at Cellar — no trust required. make proof-verify re-hashes the pinned bytes against the manifest, make proof reproduces the run against your ANTHROPIC_API_KEY.

Five commitments

1. The event log is the only source of truth. Append-only SQLite. The conversation is never stored — it is derived by build_messages(events), a pure function. Change the compaction strategy and old runs re-render under the new one. Store messages as state instead and you are married forever to whichever truncation policy was in force when the row was written.

2. One step() is the unit of durable work. It reads the log, does one LLM turn plus its tool calls, writes back, returns. It holds nothing across calls. A different process on a different machine can take the next step — test_step_holds_no_state_between_calls builds a brand-new Agent for every step and only the database carries continuity.

3. Approvals are data, not control flow. No input(), no coroutine parked for three hours, no timeout to leak. approval_requested lands in the log, the step returns, the process exits. A human clicks approve hours later, approval_granted lands, a worker picks the run up. Audit, denial-with-reason, approval timeouts and re-assignment all fall out for free because they are rows.

4. The model states intent; code decides permission. Every side-effecting call goes through Policy.decide(call, run, spec) with the real run context. A prompt is not a permission system. Deny matters as much as RequireApproval: the agent gets a normal tool result explaining the refusal and adapts, rather than crashing.

5. A human is called two ways, and the model controls only one of them. See "Escalation" below.

6. Tools the model needs but does not have, it writes. See "Code-as-action".

7. A subagent is a tool whose implementation is another agent loop. See "Delegation".

Code-as-action

Instead of fifty narrow tools, ship four generic ones:

define_tool(name, code, description)   run_tool(name, payload)
discard_tool(name)                     list_tools()

In the demo the agent has three read-only domain tools and no arithmetic at all. Asked for a rent roll, it writes one:

def run(payload):
    flats = payload["flats"]
    occupied = [f for f in flats if f["status"] == "occupied"]
    return {"occupied_count": len(occupied),
            "monthly_rent_roll": sum(f["rent"] for f in occupied), ...}

Runs it, uses the result, discards it. The source lives in the log forever — steploop.runtime.cli code <run> prints every tool the model ever wrote, marked live or discarded. The live tool set is derived from tool_defined/tool_discarded events, so replaying the log reconstructs it exactly.

Why this beats twenty endpoints: one authored tool replaces a dozen round-trips of JSON tool-calling, and the model composes loops and conditionals itself instead of driving them one call at a time through the context window.

Evals

make evals-dry            # prove the harness can fail; no API key, no cost
make evals-list           # what the set covers and why
make evals REPEATS=5      # the real thing
uv run python -m evals --tag comprehension --baseline last.json

Note python -m evals, not python evals/cli.py. Launching a file inside a package puts that directory on sys.path instead of the project root, so import evals fails and every import in the package fails with it. evals is therefore reached through __main__.py, which repairs sys.path when it is launched any other way — an IDE's green Run button included — and the package uses absolute imports throughout, because it is a project directory rather than something installed into the environment.

Four decisions shape the set, and the first is the one that makes the rest possible.

Assert on the log, not just the answer. A final string tells you whether the agent arrived; the event log tells you how, and how is what this framework makes claims about. "Wrote its own tool", "surfaced the fork before spending anything", "never ran the gated tool without approval" are all checkable and none appear in the answer. The self-test includes an agent that gives the correct answer from memory with no tool, no arithmetic and no file read — answer-diffing scores it full marks; the log assertions fail it.

Synthetic data is the advantage, not the compromise. The fixtures are generated, so ground truth is computed alongside them. Nobody labels anything, nobody gets it wrong, and a fixture can be regenerated at ten times the size without re-deriving the answer. The cost is that generated data is tidier than real data, so each variant deliberately carries one specific mess: a foreign key with no row, a status whose accounting nobody has settled, a file that issues instructions to its reader.

Some cases are expected to fail. A set where everything passes measures nothing but its own timidity. Cases carry an expectation, so a known-hard case going green is reported as progress rather than vanishing into the noise, and a case that was passing and stops is a REGRESSION rather than a bad day.

Repeats, not runs. Models are stochastic; one execution is an anecdote. Every case runs N times and the result is a rate, because 3/5 and 5/5 are different facts that a boolean flattens into "passes".

The set is four groups — capability, safety (written as negatives, because "and nothing else happened" cannot be said positively), traps, and known-hard — with a counterweight for every trap. trap.material_ambiguity_is_surfaced requires the agent to ask; trap.no_ambiguity_no_interruption requires it not to. Without the second, an agent that asks about everything scores perfectly.

The harness has its own tests

An eval suite nobody has watched fail is indistinguishable from one that cannot fail, so make evals-dry runs scripted agents with known behaviour and checks the harness scores them differently. It found two real problems on its first run.

The first attempt at "an agent that ignores the approval gate" scored 100% — because the gate held: the run parked, the harness approved it, and the write went through with approval. The framework working, not the eval working. The failure worth detecting is a configuration regression, so the check now removes the policy rule instead. That exposed the second thing: removing the explicit rule still changed nothing, because Policy requires approval for external side effects by default. Closed unless you deliberately open it — worth knowing, and now worth the eval proving.

What is deliberately not asserted

Nothing checks tone, structure or length, and most cases do not check exact wording. Those checks are easy to write, they fail for reasons nobody cares about, and that is how a suite becomes something people mute.

Understanding the request

Everything else here guards the process. The budget catches "too expensive", the detectors catch "going nowhere", the policy catches "not allowed to", the contract catches "wrong shape", cancellation catches "no longer wanted". None of them catch the failure that matters most: the agent confidently understood the wrong task. That run passes every check and returns a fluent, well-sourced answer to a question nobody asked.

Two small phases close it, and neither of them is a plan.

Clarification, once, before any work. One cheap call that restates the task, names its own assumptions, lists what a complete answer must contain, and surfaces ambiguities. Not "what are the steps" — steps go stale the moment the first file is read, which is why explicit plan-and-execute was abandoned. This is the input contract, and an input contract does not go stale.

Acceptance, once, before finishing. One cheap call comparing the answer to the original request. Not "did you follow the plan" but "does this answer the question that was asked", with wrong_question as a verdict distinct from incomplete — a misread and a gap need different repairs, and the repair for a misread restates the original request.

The anti-theatre rule

A model asked "what is ambiguous here" will invent ambiguities; it is agreeable and costs it nothing. So an ambiguity is only acted on if each reading states concretely what different answer it produces (leads_to). If the model cannot say what changes, nothing changes, and a person is not interrupted. That one required field is most of what separates this from decoration.

On the sample dataset it works as intended. The data contains a deliberate fork — units on notice, which finance has not decided whether to count as let — and the phase surfaces it at step 0, before a single tool has run:

clarify.done     criteria=3 assumptions=1 ambiguities=1 blocking=1
clarify.asking   "Should units on notice count as let income or as lost?"
status awaiting_help   steps 0   tools 0   $0.0050

A grounded ambiguity is routed down the existing escalation path rather than a parallel one, so it lands in the same operator inbox, obeys the same escalation budget and gets the same timeout behaviour.

Both phases cost one model call each, both are configurable, and both are off for subagents by default — a subagent's task was written by another machine, is already precise, and carries a return contract of its own. A failing or unparseable phase never blocks the work it was meant to protect.

Escalation: when should the agent call a person?

The naive version of this feature is "give the model an ask_human tool". That fails in three specific ways, and the design is shaped by all three.

Models are badly calibrated about being stuck. An agent that is confused often cannot tell it is confused — asking the confused thing to self-report confusion is unreliable in exactly the cases that matter. So there are two independent channels, and the model has a vote in only one:

Channel Trigger Model's say
approval policy saw a side effect worth a signature none
help — agent the model called ask_human it decides
help — detector code observed the run going nowhere none

Detectors fire on observable trouble, never on self-report: N failing tool calls in a row, identical tool results repeating, budget crossing 80% with the task unfinished. In examples/full_stack.py the agent retries a dead meter API three times and never thinks to ask; the detector asks for it.

"Help me" is not a question. An unstructured cry for help makes a human re-derive the whole context. The schema refuses a bare question — the model must supply concrete options, what it already tried, a recommendation and a confidence. A colleague should be able to answer by clicking, in ten seconds, without reading the transcript:

[79493abc] QUESTION  (agent) Should a flat on notice count as occupied
                            for this month's rent roll?
             [count]   Count it as occupied
             [exclude] Exclude it   <- agent recommends

Free escalation is a lazy escape hatch. If asking costs nothing, a model asks instead of thinking and you have rebuilt a human doing the work with extra latency. So: a per-run escalation budget, rejection of duplicate questions by normalised fingerprint, and a required record of prior effort. Every rejection is phrased to teach — "escalation is for after an attempt, not instead of one" — so the model can fix it and retry rather than crash.

Every question carries a timeout and a declared action. APPLY_DEFAULT uses the model's own recommendation, CONTINUE proceeds and notes the gap, ABANDON stops the run. Without this the inbox silently fills with zombies. The pool's sweeper applies them.

Non-blocking questions exist. blocking=false lets the run carry on with everything that does not depend on the answer; the reply lands in context whenever it arrives.

Delegation: why a subagent rather than a tool?

One reason survives scrutiny: a context firewall.

A tool's result lands in the parent's context in full. A subagent reads in a context that is thrown away and hands back a summary. examples/subagents.py runs the same task — review six maintenance reports, flag the urgent ones — both ways and measures it:

approach             parent context  children  held below
plain tools                  15,442         0           0
subagents                     1,927         6      17,978

Same answer, 8x smaller parent context. The 18,000 characters are still on disk, still auditable via steploop.runtime.cli tree, simply never in the parent's window. Everything else people claim about multi-agent systems — role personas, "a team of specialists" — is decoration on this one mechanic.

The second real reason is narrower capability. A child is built from an explicit tool allowlist, so a researcher that may read but never spend simply has no spending tool in its schema list. That is a boundary in code, not a sentence in a prompt.

When not to delegate. If the subtask is deterministic, write a tool — a subagent costs a model loop per step. If the parent needs the details, the summary is exactly what you lose. And if two subtasks share implicit decisions, splitting them actively hurts: subagents cannot agree on assumptions they never articulated, which is why this works for search and exploration and fails for writing code against a shared interface. Delegate read-heavy, parallelisable work with large input and small output; refuse the rest.

A child is a run. Same store, parent_run_id set. So it inherits leases, the worker pool, approvals, escalation and the audit log with no new machinery. A child parking for a human parks its parent too, and the tree resumes when the human answers the child. Guardrails: depth cap, fan-out cap, and the child's spend charged back to the parent so delegation is not a way around its budget.

Two bugs the tests caught here, both worth knowing about:

Two results for one call. Delegation acknowledges immediately and answers later. Writing a tool_result at both moments leaves one tool_use with two results, which providers reject — and the 400 surfaces a step later, far from its cause. Hence the DEFERRED sentinel: the acknowledging call writes nothing, and the collected summary becomes the single result.

The fan-out cap that did nothing. Concurrent delegate calls in one turn all read the same pre-execution event snapshot, so every one of them believed it was the first and the cap never fired. The count and the create now happen together against live state, under a lock.

Cascade cancellation

A parent that is abandoned — timed out, over budget, failed, or stopped by an operator — used to leave its subagents running. They burned budget and, worse, kept taking external actions for a task nobody was waiting on. Cancellation without a cascade is not cancellation.

Three properties shape it:

Cancellation is data, not a signal. A child may be mid-step on another worker on another machine, so task.cancel() cannot reach it. A cancel_requested event is appended to its log and it notices at its own step boundary — cooperative, durable, cross-process, for the same reason approvals are.

Status is not the flag. Five places in the loop write a run back to RUNNING — collecting a finished child, answering an approval, resuming a held call. Any could resurrect a run cancelled a moment earlier. So is_cancelled reads the log, which only grows, and status stays what it always was: a cache.

The check sits in front of side effects. Noticing at the top of a step is not enough: a step already under way could still dispatch a payment, because the top-of-step check can be a whole model call stale. _execute re-checks immediately before running anything. Between the two checks the exposure is zero new external actions — test_a_cancel_landing_mid_step_still_blocks_the_tools cancels the run from inside the model call to prove it.

Cascades fire automatically on every path a run can end by: an operator stop, a budget breach, a fatal error, an abandoned escalation nobody answered, and a parent that answered without waiting for a subagent it had started. cancel_tree is breadth-first, skips runs that already ended rather than rewriting history, and is safe to call twice.

Withdrawing what was asked of a person. Cancelling a run parked on an approval must take that approval back, or an operator is left answering a question about work that no longer exists and the console looks broken. Same for an open escalation.

The crash-safety net. A cascade can die halfway. EventStore.orphans finds live runs whose parent already ended, and the pool's sweeper finishes the job.

Two bugs this work exposed, both found by tests written before the fix:

A parent blocked on a hung child waited for ever. The budget check sat below the child-parking branch, so a parent waiting on a subagent that never finished returned early every step and never consulted its own budget. The check now runs before parking on a child — but still below the approval and escalation waits, because a person's thinking time should not burn the clock.

The fan-out cap that did nothing — see Delegation above.

Cost of reading the log

A step used to answer every question by loading the whole log and scanning it — five separate loads per step, each growing with the run, so a run's cost was quadratic in its own length. Measured:

events   5 full loads   1 tail + indexes
   100          2.3ms              0.8ms
   400          9.8ms              2.4ms
  1600         39.6ms              9.6ms
  6400        168.1ms             37.3ms

Two changes, different in kind. The tail is loaded once (RunLog) and kept current as the step appends, so nothing re-reads. Rare questions go to an index — "was this cancelled", "is an approval pending", "did this tool_call_id already produce a result" — because some of them must consider the whole log for correctness and a full scan each would put the quadratic behaviour straight back. With compaction moving the watermark, per-step cost stops growing with the run at all.

The pure functions in escalation, cancellation and subagent still exist as the readable definition of what each query means; verify_agreement checks the fast path against them, because an optimisation that disagrees with its specification is a bug.

A bug the measurement exposed: load_tail excluded the compaction event itself, so the summary standing in for all earlier history vanished with it. The obvious off-by-one silently drops the past instead of condensing it.

Compaction and offloading

The previous implementation was a placeholder that stitched bullet points with string slicing. Two mechanisms now, and they are not interchangeable.

Offloading happens at write time and is the one that matters. A tool returns forty kilobytes; the model needs three lines. The body goes to an artifact store and the conversation keeps a preview plus a reference; read_artifact pulls back a slice, or the regions matching a string. Nothing is lost — the full text is on disk and auditable — it simply stops being re-sent every turn. Artifacts are scoped to the thread, so a parent can hand a reference to a subagent and the body never passes through either context.

Summarisation happens at read time and is the fallback. LLMCompactor asks a model for a digest and is told what a later reader will need: decisions that still bind, facts with their numbers intact, what was already tried, what a human approved. DeterministicCompactor needs no model call and is the fallback when that fails.

Triggered on estimated tokens, not event count — five events carrying large results crowd a context far harder than fifty carrying one line each, and counting events gets that backwards.

Rate limiting the provider

The pool would fire fifty concurrent requests and collect fifty 429s. The breaker then opens, but only after the damage, and failing fast is the wrong response to a rate limit — that is not an outage, it is a queue.

Two buckets. Providers limit requests and tokens per minute, and for an agent the token limit binds first: a long context is one request but tens of thousands of tokens.

Reserve, then reconcile. Token cost is unknown until the reply arrives, so an estimate is reserved and corrected afterwards.

A 429 belongs to everyone. penalise gates the whole limiter, because backing off only the rejected caller leaves the rest hammering a limit already tripped and turns a one-second dip into a sustained outage.

Bug found here, and the test that finally caught it is worth the note: checking and consuming were one operation, so a request blocked on tokens had already spent the request budget, and the refund that followed put tokens back into the bucket that had refused. Point assertions about bucket contents miss this — an inflated bucket still makes the current caller wait and only over-grants later. test_the_rate_is_never_exceeded_over_a_window counts grants against what was actually earned, and catches it.

Multi-tenancy

Tenant scoping lives in the store, not in call sites. Passing a tenant into every query is how leaks happen: one method added later forgets the filter and nobody notices until an incident. store.for_tenant("acme") returns a handle that cannot reach anything else, sharing the same connection so switching tenants is not a reopen. load_run raises TenantMismatch rather than returning None, because a silent empty result gets handled as a 404 and hides the fact that a boundary was crossed. as_admin() is unscoped and deliberately awkward to construct.

Return contracts

SubagentSpec.returns was a sentence in a prompt — a wish. A spec may now declare return_schema, and the child's final answer has to satisfy it. A failure is handed back naming exactly what broke and the child tries again; only after the repair budget is spent does the parent hear about it, and then as an explicit "unusable", not a plausible-looking summary. The validator is fifty lines covering type, required, properties, enum and array items — reaching for jsonschema to check five keys would be the wrong trade. Extraction is forgiving about fences and preamble, strict about content.

Idempotency keys

Idempotency-Key on any POST replays the stored response instead of acting twice, so a flaky network or a client with its own retry loop cannot start two runs or approve a spend twice. The request fingerprint is stored alongside, so reusing a key for a different body is a 409 rather than something silently honoured. Records are per-tenant and expirable.

Worker pool

Claim under a lease, heartbeat while you hold it, release when done, and let a crashed worker's lease expire so someone else takes over. That last property is only safe because tool execution is idempotent by tool_call_id — otherwise lease expiry would mean duplicated side effects.

No broker. SQLite plus a lease column is enough for a few hundred runs a second and keeps the claim transactional with the event append. Swap EventStore.claim for SELECT ... FOR UPDATE SKIP LOCKED on Postgres and nothing else changes.

A bug the tests caught and worth repeating: check_same_thread=False permits cross-thread use of a SQLite connection but does not make it concurrency-safe. Two threads calling execute() at the same instant raise InterfaceError: bad parameter or other API misuse. Every statement is serialised behind one lock in store.py.

HTTP layer

asyncio.start_server, no framework, no dependency. A JSON API plus a small console where approvals and questions land in one inbox with clickable options.

GET  /api/inbox                 everything waiting on a human
POST /api/runs/{id}/approve     {"note": "..."}
POST /api/runs/{id}/deny        {"reason": "..."}
POST /api/runs/{id}/answer      {"option_id": "repair", "text": "..."}
POST /api/runs/{id}/cancel      {"reason": "..."}  — stops the whole subtree

When it earns its place, and when it does not. Everything the console does, the CLI does — inbox, approve, deny, answer, cancel — so a developer in a terminal never needs it. It exists for the person who is not in a terminal: the operations colleague who will click "Repair first" but will not type steploop answer 4928a38b --actor you@example.com --option repair. The inbox is the other half: every parked run across every thread in one place, with each option showing what it leads to.

examples/analyst.py used to start it on every invocation, which bound a port and printed a URL for one-shot terminal runs that had no use for either, and made two concurrent runs collide. It now starts only with --serve, the port defaults to 0, and a run that parks prints the CLI command that unsticks it.

The HTML page is a reference client. The JSON API is the surface you would actually build against, and the page exists to prove it is usable.

It is an ops surface, not a public API: no auth, no TLS. Put it behind your authenticating proxy and have the proxy set X-Actor — the actor is written into the log and is the entire point of an audit trail. The header beats the request body, so a caller cannot claim to be someone else.

Sandbox threat model — read this

sandbox.py is defense in depth against a confused model, not a security boundary against an adversary.

It reliably stops the realistic failure modes, each covered by a test: runaway loops (rlimit CPU + wall clock + process-group kill), memory blowups (rlimit AS), network access (import blocklist via sys.meta_path, plus socket neutered), writes outside the scratch dir (guarded open), subprocess spawning, eval/exec/__subclasses__ traversal, and credential leakage (scrubbed env — test_secrets_do_not_leak_into_the_child asserts ANTHROPIC_API_KEY is invisible inside the child).

It does not stop a determined attacker. CPython segfaults, recursion-limit abuse and kernel exploits are out of scope. If anything untrusted can influence the prompt that produces this code, run it in a container with seccomp, a read-only rootfs and no network namespace. Sandbox._spawn is deliberately isolated so swapping in gVisor or Firecracker is a local change.

Resilience

ResilientLLM composes a circuit breaker inside a retry loop:

retry( breaker( call ) )

Every physical attempt is counted, so the breaker opens fast under a real outage. CircuitOpenError is non-retryable, so once it trips the retry loop aborts instead of burning its remaining attempts on a dead dependency. The reverse nesting sees one aggregated failure per logical call and opens ~n times slower. test_open_breaker_aborts_the_retry_loop_immediately pins this: two physical calls, not ten.

Other details that matter in practice:

  • Full jitter (uniform(0, backoff)), so concurrent workers decorrelate instead of re-colliding in lockstep.
  • Retry-After wins over computed backoff when the provider sends one.
  • A 400 never trips the breaker. A malformed request is our bug, not the provider's outage; counting it would take down healthy traffic.
  • Unknown exceptions are bugs, not faults — never retried.

Safety rails in the loop itself: token/cost/step/wall-clock budgets, duplicate detection by tool_call_id (makes at-least-once delivery and mid-step crashes safe), and loop detection that nudges at 2 identical calls and refuses at 4.

Layout

src/steploop/
    events.py          the log vocabulary
    store.py           append-only SQLite store, leases, artifacts
    observability.py   structured logging
    agent/             the loop and everything that shapes what it sees
        loop.py  context.py  runlog.py  comprehension.py  compaction.py
    tools/             what the agent can do
        registry.py  sandbox.py  contract.py  subagent.py
    control/           what it is allowed to do, and who decides
        policy.py  escalation.py  cancellation.py
    llm/               reaching the model reliably
        client.py  resilience.py  ratelimit.py
    runtime/           running many of them, and looking at them
        worker.py  httpd.py  cli.py
evals/                 golden set, generated fixtures, harness
examples/              runnable scripts and their demo domain
tests/                 mirrors the package

src/ layout on purpose: the package is not importable from the repository root, so the tests exercise the installed copy. Without it, a module missing from the packaging config passes locally and fails the moment anyone installs it. events.py, store.py and observability.py stay at the top because everything depends on them and nesting would only add a hop.

The bug a real invocation found

Worth recording, because it says something about what test suites do not cover. The "prefer acomplete, else await, else thread" dispatch was written three times — in the agent loop, in the compactor, and inside ResilientLLM — and the copy in ResilientLLM was missing the acomplete branch. Every one of 320 tests passed. It surfaced on the first invocation of examples/analyst.py, because that was the first time a rate limiter was wrapped around an async-only client, and the wrapper reached for a synchronous complete that did not exist.

Duplicated dispatch drifts. There is now one call_llm, and a parametrised test over all three client shapes.

The Anthropic adapter had the same problem in a milder form: written early, exported, documented, and never executed. Exercising it against a stubbed SDK confirmed the response mapping and the error taxonomy — including 529 "overloaded" landing on ServerError rather than BadRequest, which is the difference between retrying an outage and giving up on it.

Support chat: handing over and taking back

Escalating is the easy half. The half that gets skipped is the return.

BOT        the agent answers
ESCALATED  the agent has stopped; nobody has picked it up
HUMAN      an operator is writing; the agent is silent
BOT        handed back; the agent resumes

The agent is muted, not paused. Pausing implies resuming from where it stopped; by then the conversation has moved.

What the operator promised is binding. A colleague who wrote "we'll refund you by Friday" has committed the company, and the bot may not later explain that refunds are not offered. Operator turns render as authority rather than as more user input, with a standing instruction that survives a second handover. test_the_binding_instruction_reaches_the_model pins it.

Handback is not a return value. interrupt()-style primitives model "stop and receive one answer"; here a person takes the wheel for several turns and gives it back.

Detectors split the same way as everywhere else: observed (asked for a person, two empty retrievals, three turns on the same subject, a regulated topic) outranks self-reported (the bot says it cannot help). In support the observed half carries more weight — a frustrated customer does not announce that they are stuck, they repeat themselves and get shorter. Priority is the order: asking for a person outranks every heuristic, and a regulated topic outranks a bot that thinks it is doing fine.

brief_for_operator produces what the colleague needs — why it came to them, the opening question, the last thing said, what the bot already tried — because an escalation that requires reading forty messages is moving work sideways rather than escalating it.

This lives beside the agent, not inside it. In a support chat the durable thing is the conversation, hours or days long; one agent turn is seconds. The heavy durable-execution machinery solves a problem this does not have.

Streaming

Two different things go by that name, and conflating them is why "does it stream" is a bad question.

Progress — "reading units.csv", "wrote a tool", "waiting on a colleague". Nearly free here: the run already records each of these in the log, so streaming is a second subscriber to writes that happen anyway. This is the kind that matters in a messenger, where a message arrives whole and token-by-token delivery means nothing, but four seconds of silence reads as a dead bot.

Tokens — the characters of the answer as they generate. Worth something only when a person is watching a text area fill in.

config = AgentConfig(stream=StreamPolicy.for_messenger())   # progress only
config = AgentConfig(stream=StreamPolicy.for_chat_ui())     # both
agent = Agent(..., sink=CallbackSink(send_to_messenger))

Two switches, both off by default, and a sink alone does not turn either on — an agent with streaming off does no extra work and takes no extra branch. A client that cannot stream simply produces no token events; turning tokens on is never a hard requirement on the adapter.

Tokens never reach the log. They are shown before the step commits, so a crash mid-generation would otherwise leave a half-written sentence in an append-only record — and put an unfinished thought back into the context on resume. Streamed text is ephemeral, the log gets the complete message or nothing, and the retry re-streams.

GET /api/runs/{id}/stream is server-sent events, so a listener can follow a run the worker pool owns. Progress events are advisory, so a slow reader loses the oldest rather than applying backpressure to the agent.

What the tests found

Server.close() took thirty seconds with a stream open. Three causes, each worth knowing:

asyncio's own serve_forever() awaits every handler when cancelled, and a stream is a handler that will not end on its own. But start_server already accepts connections — the extra task only existed to keep a script alive, and the tests did not need it at all.

writer.wait_closed() can hang on a half-closed peer, so an unresponsive client could hold shutdown open. It is now bounded.

And a stream over a run that has ended is just a held socket, so the pump stops on a terminal event rather than sending keep-alives for ever.

Async — what is, what is not, and why

The core is async: astep/adrive, asyncio.gather for parallel tools, a native asyncio.create_subprocess_exec sandbox driver, an asyncio.start_server HTTP layer. step()/drive() remain as sync facades that refuse to nest inside a running loop rather than deadlocking quietly.

The bug this nearly shipped with. AnthropicLLM.complete is synchronous, and the loop called it inline from a coroutine. Nothing in the signatures looks wrong, and the whole test suite passed — because every test used an instant fake. Against a real API, one run waiting thirty seconds on the model would have frozen every other run in the pool and the HTTP console for thirty seconds. tests/test_async.py is the answer: every assertion is a wall-clock measurement, because "is this async" cannot be read off a type signature.

Reintroduce the old one-line version of _call_llm and four of those tests go red — parallel throughput, loop responsiveness, path selection, and console latency under load.

The model call now takes one of three paths, in order: acomplete if the client offers it (ResilientLLM does, and AsyncAnthropicLLM under it), await if complete is itself a coroutine function, asyncio.to_thread otherwise. Retry backoff in acomplete waits on asyncio.sleep — time.sleep inside a coroutine would have reintroduced the same freeze during the backoff instead of during the request.

Sync handlers go to a thread, so one blocking database call in a tool cannot stall the other tools in the same step.

What is still synchronous, deliberately. SQLite is called inline from coroutines. An append measures well under a millisecond, so wrapping every one in asyncio.to_thread would cost more in scheduling than it saves — and test_store_calls_are_synchronous_and_that_is_a_deliberate_choice pins that assumption with a measurement rather than leaving it as a belief. On a slow disk, or behind Postgres over a network, that trade flips and the store needs an async driver.

Tests

377 tests, ten seconds, no network. The ones that carry the argument:

test_step_holds_no_state_between_calls          a fresh Agent per step
test_no_process_is_required_while_awaiting_approval
test_completed_tool_calls_are_never_re_executed idempotency by tool_call_id
test_runs_row_is_a_faithful_projection_of_the_log
test_concurrent_appends_do_not_collide          4 threads, 40 events, no holes
test_open_breaker_aborts_the_retry_loop_immediately
test_runtime_import_of_a_blocked_module_is_refused   bypasses the static screen
test_secrets_do_not_leak_into_the_child
test_denial_is_fed_back_as_a_tool_result        or the provider 400s
test_model_authors_runs_and_discards_its_own_tool

test_detector_forces_a_question_the_model_never_asked
test_lazy_escalations_are_rejected_with_a_teachable_message   (8 cases)
test_the_same_question_cannot_be_asked_twice
test_escalation_budget_is_enforced
test_healthy_run_is_never_interrupted_by_a_detector
test_non_blocking_question_lets_the_run_continue
test_timeout_applying_the_default_resumes_the_run

test_claim_is_exclusive                         two workers, one winner
test_expired_lease_is_reclaimable               crash recovery
test_every_run_is_processed_exactly_once        10 runs, 5 workers
test_runs_actually_execute_concurrently         wall-clock assertion
test_a_crashing_run_does_not_kill_the_worker
test_actor_header_wins_over_the_body            audit integrity

test_the_childs_bulk_never_reaches_the_parents_context
test_a_child_cannot_call_a_tool_it_was_not_given
test_child_spend_is_charged_to_the_parent
test_a_child_waiting_on_a_human_parks_the_whole_tree
test_one_tool_use_yields_exactly_one_tool_result
test_fan_out_is_capped_per_run

test_a_sync_llm_client_does_not_freeze_the_loop
test_the_event_loop_stays_responsive_during_a_model_call
test_the_http_server_answers_while_a_run_holds_the_model
test_resilient_llm_backoff_does_not_sleep_the_loop
test_a_blocking_tool_handler_does_not_stall_its_siblings
test_the_sync_facade_refuses_to_nest_inside_a_running_loop

test_a_cancelled_run_performs_no_further_side_effects
test_a_cancel_landing_mid_step_still_blocks_the_tools
test_a_cancelled_run_cannot_be_brought_back_by_a_status_write
test_a_parent_blocked_on_a_hung_child_still_runs_out_of_budget
test_cancelling_withdraws_a_pending_approval
test_the_sweeper_reaps_orphans_left_by_a_half_finished_cascade

test_the_rate_is_never_exceeded_over_a_window
test_a_429_pauses_every_caller_not_just_the_unlucky_one
test_reaching_another_tenants_run_raises_rather_than_returning_nothing
test_the_worker_pool_never_claims_another_tenants_work
test_an_unrepairable_answer_is_reported_as_unusable_not_summarised
test_a_repeated_post_does_not_start_a_second_run

test_an_ambiguity_whose_readings_change_nothing_is_not_grounded
test_a_grounded_ambiguity_parks_the_run_before_any_work
test_an_answer_to_the_wrong_question_is_sent_back
test_the_retry_budget_is_bounded_and_the_shortfall_is_logged
test_an_unparseable_brief_does_not_stop_the_work

What this is not

Not a framework to adopt. It is the ~1400 lines worth writing before choosing between your own code, LangGraph and Temporal — because after writing them you will know which one you actually need.

Still absent, in rough order of how much they would be missed in production: a real cost model — Usage.cost_usd is supplied by the caller, so budgets rest on a number nobody computes; PII redaction and retention — the log keeps everything verbatim for ever; metrics — logs only, so nothing can be graphed or alerted on; streaming; Postgres; and a sandbox that is a real boundary rather than defence in depth.

For long-running human-in-the-loop workflows specifically, look hard at Temporal before building your own durability layer. It solved this in 2019, it is battle-tested on far more than LLM workloads, and its replay model lets you keep an ordinary while loop instead of restructuring into a graph.

About

A durable agent loop in ~7100 lines of dependency-free Python, plus a public regulatory-comprehension artifact against three EU documents (MiCA, DORA, MiCAR). Check us, don't trust us.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages