Repository navigation
Add the E0 sweep profile and its runner - #77
Conversation
RodCor
left a comment
There was a problem hiding this comment.
fail against the bar at cc7b729.
- ci: 20/21 green.
cargo-audit (RUSTSEC)is red, but that comes from the repo-wide advisories on quinn-proto, quinn-udp, rustix and pin-utils in main's Cargo.lock. this diff doesn't touch Cargo.lock. - mergeable, base main, 0 behind. two linear commits, no force-push.
- no rust and no adr 0001 areas touched. there are no benchmark numbers in the body. i checked the code refs:
DEFAULT_RRF_Kis at fusion.rs:64/188 and the bge auto floor 0.35 is at project.rs:389-391. - must-fix: the new profile's only test,
e0_eval.py --self-test, isn't in ci, so a green ci says nothing about this diff. it passes locally and needs no binary. - should-fix:
rerank_offisn't covered by the one-knob check. memcap misses reparented descendants and doesn't forward signals. the repo copy of the runner can't run from the repo and has no README saying it's an archival copy. - nits: hardcoded dataset counts, exit 0 on a determinism diff, bot and room names in the script header.
fixes go to fuse.
| raise SystemExit(f"bench lexical patch: {lexical}") | ||
| if "fused_rerank_floor_0.3" not in bench["arms"]: | ||
| raise SystemExit(f"bench arms: {bench['arms']}") | ||
| sweep = plan(fixture, "fixture", "bench-sweep", 0) |
There was a problem hiding this comment.
must-fix: this self-test is the PR's only test, and ci never runs it. .github/workflows/ci.yml runs check-version-sync, test-assemble-npm and the examples/decisions unittest, but nothing under scripts/e0_eval.py or scripts/retrieval_eval.py. with merge-on-green, that leaves the sweep profile untested at merge.
i ran python3 scripts/e0_eval.py --self-test at cc7b729 and it passes (e0 bench self-test ok). it's stdlib only and needs no binary.
passing means a ci step runs python3 scripts/e0_eval.py --self-test, plus python3 scripts/retrieval_eval.py --self-test if it's equally cheap, and it goes green on the head.
There was a problem hiding this comment.
CI job eval-scripts now runs e0_eval.py, retrieval_eval.py, prepare_e0_data.py and memcap.py with --self-test, plus run_e0s.sh --self-test. d4c4ac9
There was a problem hiding this comment.
closed at d4c4ac9. eval-scripts runs all 5 self-tests and went green on the merge ref. i ran the same 5 locally and all passed.
| arms = [ | ||
| bench["arms"][0], | ||
| ("fused_defaults", dict(defaults), "fused defaults: linear, auto semantic floor, tinybert floor 0.3"), | ||
| ("rerank_off", {"embedder.reranker": '"off"'}, "embedder.reranker=off"), |
There was a problem hiding this comment.
should-fix: rerank_off is the one arm not built from defaults, and the self-test skips it (e0_eval.py:203). so the PR body's claim that each arm differs from the defaults by exactly one knob isn't checked for it. against fused_defaults it differs in 2 keys today, embedder.reranker and broker.rerank_min_score (checked locally).
behavior is the same, since the floor is moot with the reranker off. use {**defaults, "embedder.reranker": '"off"'} and drop "rerank_off" from the skip tuple, and the test covers it.
There was a problem hiding this comment.
rerank_off is now the fused defaults with only embedder.reranker overridden; the self-test checks each arm differs from fused defaults only in its documented key. 0e64812
There was a problem hiding this comment.
closed at 0e64812. rerank_off is now defaults plus one key, and expected_diff pins the key for each arm. mutation check: i reverted the line locally and the self-test fails with changes ['broker.rerank_min_score', 'embedder.reranker'], expected ['embedder.reranker'].
| peak = max(peak, total) | ||
| if total > limit: | ||
| log.write(f"[{stamp()}] KILL: tree RSS {total // 1024} MiB > {a.limit_mib} MiB; SIGKILL {len(pids)} pids {pids}\n") | ||
| for p in tree(child.pid): |
There was a problem hiding this comment.
should-fix: the kill and the RSS sum both walk the ppid tree. any descendant reparented to init (a double fork, or a grandchild whose parent already exited) drops out of both, so it's neither counted nor killed. a child spawned between tree() and the kill gets missed too.
the child already has its own session (line 51), so os.killpg(child.pid, signal.SIGKILL) covers the whole group.
also, there's no SIGTERM/SIGINT handler. since the child is in a new session, killing memcap or hitting ctrl-c leaves the eval running with no cap.
passing means killpg plus signal forwarding, and a small test: a python child that allocates past --limit-mib gives rc 137 with no survivors, and a normal child passes its rc through.
There was a problem hiding this comment.
memcap starts the child with start_new_session, sums RSS and SIGKILLs the process group, forwards SIGTERM/SIGINT, and --self-test covers the cap kill (rc 137), passthrough and both signals. d913f3c
There was a problem hiding this comment.
closed at d913f3c. start_new_session, group RSS, killpg, and SIGTERM/SIGINT forwarding all check out. the self-test covers a reparented grandchild over the cap (rc 137, group empty) and both signals (128+sig, pids dead). it passed locally and in ci.
optional, non-blocking: if the leader exits normally, the poll loop stops and any group members left behind keep running. a final killpg(SIGKILL) after the loop would reap them.
| S=/workspace/fuse/e0s | ||
| WT=$S/kimetsu | ||
| MAIN_SHA=19112dae937fb3b1842983943a0def6fad75f4b1 # origin/main, fetched 2026-11-13 13:13 -03 | ||
| SHA=61dc9f01a60999d18d8614e0337272f893643fa0 # cursor/e0-sweep-profile = main + bench-sweep profile (scripts only) |
There was a problem hiding this comment.
should-fix: this copy can't be rerun from the repo, and nothing says so.
SHAis pinned to 61dc9f0, the PR's first commit, so a checkout at the head dies at line 43 with exit 2.S,DATAandBENCH_Eare hardcoded/workspace/...box paths.- line 55 checks hardcoded llama-server pids.
- line 116 writes "not pushed" into the manifest, which is now stale.
that's fine for a byte-identical archival copy, but the dir should carry the context, the way scripts/experiments/e0/README.md does. passing means a short scripts/experiments/e0s/README.md saying it's the box copy, which commit it ran at, where its sha256s are, and how to reproduce it (which vars to override).
There was a problem hiding this comment.
still open at d4c4ac9. the flags, env vars and SHA=HEAD work, and the box paths are gone from code. but the defaults don't run, and i reproduced both with a stub --bin:
- the default
--outis$REPO/e0s-results, which isn't in.gitignore. the run creates it, then L272 exits 3 (worktree dirty after build). every later run exits 2 at L229 (worktree has local changes). the default$REPO/e0-datahits the same check. - the default baseline is
RECORDED_MAIN=19112da. main is already at 16e6232, which changes 11 files under crates/Cargo. on a local merge of this PR into main, the run exits 2 at L231 (checkout changes Rust vs 19112da), so after merge it won't run from main withoutE0S_MAIN_SHA.
--self-test passes in both cases because it only checks how the config is parsed.
closing means: add /e0s-results/ and /e0-data/ to .gitignore (or default them outside the repo, or leave $OUT out of the porcelain check), and either default the baseline to HEAD or document E0S_MAIN_SHA in the usage text. a self-test case that runs one non-dry preflight with a stub bin would catch both. the header usage text is fine in place of a README.
There was a problem hiding this comment.
Default data and output dirs (e0-data/, e0s-results/) are gitignored and left out of the dirty-tree check, so a fresh run and a rerun both pass it. The Rust baseline defaults to the merge-base of HEAD and origin/main (HEAD when that ref is absent); E0S_MAIN_SHA still overrides. --self-test now runs both on a clean clone with a stub binary. eedd1fb
There was a problem hiding this comment.
closed at eedd1fb. i re-ran my stub-bin repro:
- default run + rerun at eedd1fb: both log
preflight ok, and the tree stays clean. baseline = merge-base, 19112da. - local merge into main 16e6232: both runs log
preflight okwith baseline 16e6232. no morecheckout changes Rust. - (both stop at exit 4 in eval, which is expected with a stub bin.)
mutation check on --self-test:
- with the old 19112da baseline restored, it fails (
default run 1 did not pass preflight). - with the old .gitignore, it fails (
e0s-results is not ignored). - with the old .gitignore plus the old plain porcelain check, it fails in preflight.
so it would have caught the old bug. bonus: the 9>&- fix for the lock fd inherited by sleep is right.
| echo "- code: RodCor/kimetsu main @ $MAIN_SHA; worktree $WT @ $SHA (branch cursor/e0-sweep-profile, scripts-only diff, not pushed)" | ||
| echo "- binary: $BIN, $VERSION, sha256 $BIN_SHA; build log $LOGS/build.log" | ||
| echo "- build: (cd $WT && ${build_env[*]} ${build_cmd[*]})" | ||
| echo "- data: $DATA/locomo-conv26.json (199 q / 197 scored), $DATA/lme-oracle.json (500 q / 479 scored)" |
There was a problem hiding this comment.
nit: the question and scored counts (199/197, 500/479) are hardcoded into the manifest. the dry-run at line 53 already computes them, so capture and reuse those values and the manifest can't drift from the data.
There was a problem hiding this comment.
The manifest now records question and scored counts from the dry-run. 002ba59
There was a problem hiding this comment.
closed at 002ba59. the counts now come from the --dry-run plan, and the self-test asserts 3 questions, 3 scored for both datasets.
| echo; cat "$RES/run1/capsule-hist.md" | ||
| } > "$RES/MANIFEST.md" | ||
| log "E0+sweep complete. determinism diff exit=$DET. manifest $RES/MANIFEST.md" | ||
| exit 0 |
There was a problem hiding this comment.
nit: this exits 0 even when the determinism diff fails (DET != 0). the header documents it, but anything checking rc won't see the nondeterminism. consider exit $DET or a distinct code.
There was a problem hiding this comment.
The runner now exits with the determinism-diff status. 002ba59
There was a problem hiding this comment.
closed at 002ba59. the script ends with exit "$DET", and the header documents 1 = determinism diff failed.
| @@ -0,0 +1,133 @@ | |||
| #!/usr/bin/env bash | |||
| # E0+sweep runner (#68). Spec: gauge, ML room, 2026-11-13. Prepared by fuse. | |||
There was a problem hiding this comment.
nit: "gauge, ML room" and "Prepared by fuse" are team-internal names in repo text. keep it neutral and reference #68 only. same for the spec: line in the manifest.
There was a problem hiding this comment.
closed at 002ba59. no bot or room names in code or output. the only ones left are /workspace/fuse paths in the provenance comment for the recorded run, and that's fine.
rerank_off omitted broker.rerank_min_score, so it differed from fused defaults in two keys. Build it from those defaults, and assert the intended key for every arm except lexical. Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
Start the child in its own session, sum RSS across that process group, and SIGKILL the group over the limit. Forward SIGTERM and SIGINT to the group. --self-test checks the cap, exit-code passthrough, and both signals. Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
Resolve the repo from the script path. Data, output, binary, and commit come from flags or environment variables, and the commit defaults to HEAD. The header records the 2026-11-13 run at 61dc9f0 with /workspace/fuse paths. Dataset counts come from the dry-run, and the process exits with the determinism-diff status. Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
The job runs the Python self-tests and the sweep runner path check. It does not build Rust or download models. Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
RodCor
left a comment
There was a problem hiding this comment.
re-review at d4c4ac9: fail, narrowly. the must-fix is closed, but 1 should-fix is still open and the pr body is out of date.
history: linear, 4 commits on cc7b729 (0e64812, d913f3c, 002ba59, d4c4ac9). cc7b729 is an ancestor, so no force-push.
threads, 6 of 7 closed:
- closed: self-test in ci, rerank_off one-knob (mutation-checked), memcap killpg/signals, counts, exit code, names
- open: run_e0s defaults. the default
--outis$REPO/e0s-results, which isn't gitignored, so a run exits 3 at L272 and every rerun exits 2 at L229. the baseline defaults to 19112da, so on current main the run exits 2 at L231. i reproduced both locally with a stub bin, and--self-testdoesn't catch either.
new should-fix, pr body: these lines are now false.
- "No Rust, Cargo or workflow files are changed": ci.yml is changed.
- "byte-identical copies of the scripts used for the box run": memcap.py and run_e0s.sh were rewritten after the run, so the hashes in wrapper-sha256.txt match cc7b729, not the head.
- "kills only its own process tree": it now kills the process group.
say which commit the recorded run's scripts came from, and list the eval-scripts job under tests.
ci on d4c4ac9 (merge ref against main 16e6232): 19 pass, including the new python eval self-tests job (job, 5 of 5 ok). still running: test (windows-latest) and Analyze (rust). cargo-audit fails on RUSTSEC-2026-0523/0524 (quinn-proto), 0538 (quinn-udp), 0486 (rustix, unsound) and 0441 (pin-utils, unmaintained). that's main's Cargo.lock, which this pr doesn't touch, and #78 fixes it.
nothing in this pr touches ADR 0001 areas.
to pass: close r4566145713, fix the body, windows and Analyze (rust) go green, and cargo-audit goes green once #78 lands and this branch picks up main with a merge commit.
Ignore e0-data and e0s-results, and leave those directories out of the dirty-tree check so a run and a rerun stay clean. The Rust baseline defaults to the merge-base with origin/main, or HEAD when that ref is absent. The self-test runs both against a clean clone with a stub binary. Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
RodCor
left a comment
There was a problem hiding this comment.
re-review at eedd1fb: pass on code and findings. it's not merge-ready until CI finishes green.
history: 1 commit on d4c4ac9. d4c4ac9 is an ancestor, so no force-push.
findings: all 7 threads are closed. r4566145713 is closed now too. i verified it with my stub repro (default run, rerun, and a local merge into main 16e6232) and with mutations: --self-test fails on the old baseline and on the old .gitignore.
pr body, checked against the code:
- wrapper-sha256.txt on the box matches cc7b729 for run_e0s.sh, memcap.py and sweep_hist.py.
- e0_eval.py, retrieval_eval.py, retrieval_eval_profiles.py, prepare_e0_data.py and summarize_e0.py match 61dc9f0.
- the ci.yml
eval-scriptsjob, the process-group kill with signal forwarding, and "no Cargo.lock change" are all accurate (no crates/Cargo diff vs base).
still open, optional: the memcap orphan note on r4566145704 wasn't addressed. memcap.py is unchanged since d4c4ac9. repro: memcap.py ... -- sh -c 'sleep 47 & exit 0' returns 0 and leaves the sleep running. this doesn't block, and it can be a follow-up.
ci on eedd1fb (merge ref vs 16e6232), as of 14:36 ART:
- pass: python eval self-tests, clippy, rustfmt, TypeScript SDK, decision provider examples, npm staging regression tests, integration asset drift, cargo/npm version sync, test (ubuntu-latest), test (embeddings, ubuntu), Analyze (python/actions/js-ts) x2. CodeQL is neutral.
- pending: test (windows-latest), test (macos-15) (queued), Analyze (rust) x2.
- fail: cargo-audit, on the known set: RUSTSEC-2026-0523/0524 quinn-proto, 0538 quinn-udp, 0486 rustix, 0441 pin-utils. that's main's lockfile, which #78 fixes.
to merge: the pending jobs go green, and cargo-audit goes green after #78 lands and this branch takes main with a merge commit.
RodCor
left a comment
There was a problem hiding this comment.
pass at 8b85e90 for the merge itself. my code pass from eedd1fb carries over.
- 8b85e90 is a real merge commit with parents 707948b and main eaf8016. eedd1fb is in its history, so there was no force-push.
- its tree (1ceede3) is exactly what
git merge-treeproduces from those two parents, so no conflicts were resolved by hand. - the net diff vs main has the same stable patch-id (5603ef1) as eedd1fb vs its merge-base: 7 files, +1027/-1. the merge adds no new content.
the only thing left is ci on 8b85e90. comments only, i pushed nothing.
Adds a
bench-sweepprofile toscripts/retrieval_eval_profiles.pyandscripts/e0_eval.py. Lexical is the FTS baseline. Each later arm changes one knob from the fused defaults: rerank off, tinybert floor 0 (fill-to-cap, reorder only), 0.1 and 0.2, semantic floor 0, and rrf (k=60, hardcoded as DEFAULT_RRF_K at fusion.rs:64/188). The vector arm is the existing script extra.scripts/experiments/e0s/run_e0s.shruns from a checkout. Data, output, and the binary come from flags or the environment. The commit defaults to HEAD. The default data directory ise0-data/and the default output directory ise0s-results/; both are gitignored, and the dirty-tree check ignores them so a rerun into that output directory stays clean. The Rust baseline defaults to the merge-base of HEAD andorigin/main, or to HEAD when that ref is absent.E0S_MAIN_SHAoverrides it.The recorded 2026-11-13 run used the scripts at cc7b729, built from 61dc9f0, with
/workspace/fusepaths. The boxwrapper-sha256.txthashes match cc7b729, not the head of this branch.memcap.pyandrun_e0s.shhave changed since that run.memcap.pykills the child process group when RSS exceeds the cap and forwards SIGTERM and SIGINT to that group.sweep_hist.pywrites the 0–8 capsule histogram for the run directory it is given..github/workflows/ci.ymladds theeval-scriptsjob (python eval self-tests). Tests:python3 scripts/e0_eval.py --self-test(each sweep arm except lexical differs from fused defaults in its documented key)python3 scripts/retrieval_eval.py --self-testpython3 scripts/prepare_e0_data.py --self-testpython3 scripts/experiments/e0s/memcap.py --self-testbash scripts/experiments/e0s/run_e0s.sh --self-testNo Cargo.lock change.