Skip to content

Add the E0 sweep profile and its runner - #77

Merged
RodCor merged 9 commits into
mainfrom
cursor/e0-sweep-profile
Nov 13, 2026
Merged

RodCor merged 9 commits into
mainfrom
cursor/e0-sweep-profile

Conversation

@RodCor

@RodCor RodCor commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Adds a bench-sweep profile to scripts/retrieval_eval_profiles.py and scripts/e0_eval.py. Lexical is the FTS baseline. Each later arm changes one knob from the fused defaults: rerank off, tinybert floor 0 (fill-to-cap, reorder only), 0.1 and 0.2, semantic floor 0, and rrf (k=60, hardcoded as DEFAULT_RRF_K at fusion.rs:64/188). The vector arm is the existing script extra.

scripts/experiments/e0s/run_e0s.sh runs from a checkout. Data, output, and the binary come from flags or the environment. The commit defaults to HEAD. The default data directory is e0-data/ and the default output directory is e0s-results/; both are gitignored, and the dirty-tree check ignores them so a rerun into that output directory stays clean. The Rust baseline defaults to the merge-base of HEAD and origin/main, or to HEAD when that ref is absent. E0S_MAIN_SHA overrides it.

The recorded 2026-11-13 run used the scripts at cc7b729, built from 61dc9f0, with /workspace/fuse paths. The box wrapper-sha256.txt hashes match cc7b729, not the head of this branch. memcap.py and run_e0s.sh have changed since that run. memcap.py kills the child process group when RSS exceeds the cap and forwards SIGTERM and SIGINT to that group. sweep_hist.py writes the 0–8 capsule histogram for the run directory it is given.

.github/workflows/ci.yml adds the eval-scripts job (python eval self-tests). Tests:

  • python3 scripts/e0_eval.py --self-test (each sweep arm except lexical differs from fused defaults in its documented key)
  • python3 scripts/retrieval_eval.py --self-test
  • python3 scripts/prepare_e0_data.py --self-test
  • python3 scripts/experiments/e0s/memcap.py --self-test
  • bash scripts/experiments/e0s/run_e0s.sh --self-test

No Cargo.lock change.

@RodCor RodCor left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fail against the bar at cc7b729.

  • ci: 20/21 green. cargo-audit (RUSTSEC) is red, but that comes from the repo-wide advisories on quinn-proto, quinn-udp, rustix and pin-utils in main's Cargo.lock. this diff doesn't touch Cargo.lock.
  • mergeable, base main, 0 behind. two linear commits, no force-push.
  • no rust and no adr 0001 areas touched. there are no benchmark numbers in the body. i checked the code refs: DEFAULT_RRF_K is at fusion.rs:64/188 and the bge auto floor 0.35 is at project.rs:389-391.
  • must-fix: the new profile's only test, e0_eval.py --self-test, isn't in ci, so a green ci says nothing about this diff. it passes locally and needs no binary.
  • should-fix: rerank_off isn't covered by the one-knob check. memcap misses reparented descendants and doesn't forward signals. the repo copy of the runner can't run from the repo and has no README saying it's an archival copy.
  • nits: hardcoded dataset counts, exit 0 on a determinism diff, bot and room names in the script header.

fixes go to fuse.

Comment thread scripts/e0_eval.py
raise SystemExit(f"bench lexical patch: {lexical}")
if "fused_rerank_floor_0.3" not in bench["arms"]:
raise SystemExit(f"bench arms: {bench['arms']}")
sweep = plan(fixture, "fixture", "bench-sweep", 0)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must-fix: this self-test is the PR's only test, and ci never runs it. .github/workflows/ci.yml runs check-version-sync, test-assemble-npm and the examples/decisions unittest, but nothing under scripts/e0_eval.py or scripts/retrieval_eval.py. with merge-on-green, that leaves the sweep profile untested at merge.

i ran python3 scripts/e0_eval.py --self-test at cc7b729 and it passes (e0 bench self-test ok). it's stdlib only and needs no binary.

passing means a ci step runs python3 scripts/e0_eval.py --self-test, plus python3 scripts/retrieval_eval.py --self-test if it's equally cheap, and it goes green on the head.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI job eval-scripts now runs e0_eval.py, retrieval_eval.py, prepare_e0_data.py and memcap.py with --self-test, plus run_e0s.sh --self-test. d4c4ac9

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at d4c4ac9. eval-scripts runs all 5 self-tests and went green on the merge ref. i ran the same 5 locally and all passed.

Comment thread scripts/retrieval_eval_profiles.py Outdated
arms = [
bench["arms"][0],
("fused_defaults", dict(defaults), "fused defaults: linear, auto semantic floor, tinybert floor 0.3"),
("rerank_off", {"embedder.reranker": '"off"'}, "embedder.reranker=off"),

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix: rerank_off is the one arm not built from defaults, and the self-test skips it (e0_eval.py:203). so the PR body's claim that each arm differs from the defaults by exactly one knob isn't checked for it. against fused_defaults it differs in 2 keys today, embedder.reranker and broker.rerank_min_score (checked locally).

behavior is the same, since the floor is moot with the reranker off. use {**defaults, "embedder.reranker": '"off"'} and drop "rerank_off" from the skip tuple, and the test covers it.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rerank_off is now the fused defaults with only embedder.reranker overridden; the self-test checks each arm differs from fused defaults only in its documented key. 0e64812

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at 0e64812. rerank_off is now defaults plus one key, and expected_diff pins the key for each arm. mutation check: i reverted the line locally and the self-test fails with changes ['broker.rerank_min_score', 'embedder.reranker'], expected ['embedder.reranker'].

Comment thread scripts/experiments/e0s/memcap.py Outdated
peak = max(peak, total)
if total > limit:
log.write(f"[{stamp()}] KILL: tree RSS {total // 1024} MiB > {a.limit_mib} MiB; SIGKILL {len(pids)} pids {pids}\n")
for p in tree(child.pid):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix: the kill and the RSS sum both walk the ppid tree. any descendant reparented to init (a double fork, or a grandchild whose parent already exited) drops out of both, so it's neither counted nor killed. a child spawned between tree() and the kill gets missed too.

the child already has its own session (line 51), so os.killpg(child.pid, signal.SIGKILL) covers the whole group.

also, there's no SIGTERM/SIGINT handler. since the child is in a new session, killing memcap or hitting ctrl-c leaves the eval running with no cap.

passing means killpg plus signal forwarding, and a small test: a python child that allocates past --limit-mib gives rc 137 with no survivors, and a normal child passes its rc through.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

memcap starts the child with start_new_session, sums RSS and SIGKILLs the process group, forwards SIGTERM/SIGINT, and --self-test covers the cap kill (rc 137), passthrough and both signals. d913f3c

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at d913f3c. start_new_session, group RSS, killpg, and SIGTERM/SIGINT forwarding all check out. the self-test covers a reparented grandchild over the cap (rc 137, group empty) and both signals (128+sig, pids dead). it passed locally and in ci.

optional, non-blocking: if the leader exits normally, the poll loop stops and any group members left behind keep running. a final killpg(SIGKILL) after the loop would reap them.

Comment thread scripts/experiments/e0s/run_e0s.sh Outdated
S=/workspace/fuse/e0s
WT=$S/kimetsu
MAIN_SHA=19112dae937fb3b1842983943a0def6fad75f4b1 # origin/main, fetched 2026-11-13 13:13 -03
SHA=61dc9f01a60999d18d8614e0337272f893643fa0 # cursor/e0-sweep-profile = main + bench-sweep profile (scripts only)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should-fix: this copy can't be rerun from the repo, and nothing says so.

  • SHA is pinned to 61dc9f0, the PR's first commit, so a checkout at the head dies at line 43 with exit 2.
  • S, DATA and BENCH_E are hardcoded /workspace/... box paths.
  • line 55 checks hardcoded llama-server pids.
  • line 116 writes "not pushed" into the manifest, which is now stale.

that's fine for a byte-identical archival copy, but the dir should carry the context, the way scripts/experiments/e0/README.md does. passing means a short scripts/experiments/e0s/README.md saying it's the box copy, which commit it ran at, where its sha256s are, and how to reproduce it (which vars to override).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_e0s.sh now derives the repo root from its path, takes data/output/binary/SHA from flags or env (SHA defaults to HEAD), and its header records the 2026-11-13 run at 61dc9f0 with /workspace/fuse paths. sweep_hist.py already takes a run directory. 002ba59

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still open at d4c4ac9. the flags, env vars and SHA=HEAD work, and the box paths are gone from code. but the defaults don't run, and i reproduced both with a stub --bin:

  1. the default --out is $REPO/e0s-results, which isn't in .gitignore. the run creates it, then L272 exits 3 (worktree dirty after build). every later run exits 2 at L229 (worktree has local changes). the default $REPO/e0-data hits the same check.
  2. the default baseline is RECORDED_MAIN=19112da. main is already at 16e6232, which changes 11 files under crates/Cargo. on a local merge of this PR into main, the run exits 2 at L231 (checkout changes Rust vs 19112da), so after merge it won't run from main without E0S_MAIN_SHA.

--self-test passes in both cases because it only checks how the config is parsed.

closing means: add /e0s-results/ and /e0-data/ to .gitignore (or default them outside the repo, or leave $OUT out of the porcelain check), and either default the baseline to HEAD or document E0S_MAIN_SHA in the usage text. a self-test case that runs one non-dry preflight with a stub bin would catch both. the header usage text is fine in place of a README.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Default data and output dirs (e0-data/, e0s-results/) are gitignored and left out of the dirty-tree check, so a fresh run and a rerun both pass it. The Rust baseline defaults to the merge-base of HEAD and origin/main (HEAD when that ref is absent); E0S_MAIN_SHA still overrides. --self-test now runs both on a clean clone with a stub binary. eedd1fb

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e0-data/ is gitignored and excluded from the dirty-tree check too, and the baseline no longer defaults to 19112da, so it holds after merging main without setting E0S_MAIN_SHA. eedd1fb

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at eedd1fb. i re-ran my stub-bin repro:

  • default run + rerun at eedd1fb: both log preflight ok, and the tree stays clean. baseline = merge-base, 19112da.
  • local merge into main 16e6232: both runs log preflight ok with baseline 16e6232. no more checkout changes Rust.
  • (both stop at exit 4 in eval, which is expected with a stub bin.)

mutation check on --self-test:

  • with the old 19112da baseline restored, it fails (default run 1 did not pass preflight).
  • with the old .gitignore, it fails (e0s-results is not ignored).
  • with the old .gitignore plus the old plain porcelain check, it fails in preflight.

so it would have caught the old bug. bonus: the 9>&- fix for the lock fd inherited by sleep is right.

Comment thread scripts/experiments/e0s/run_e0s.sh Outdated
echo "- code: RodCor/kimetsu main @ $MAIN_SHA; worktree $WT @ $SHA (branch cursor/e0-sweep-profile, scripts-only diff, not pushed)"
echo "- binary: $BIN, $VERSION, sha256 $BIN_SHA; build log $LOGS/build.log"
echo "- build: (cd $WT && ${build_env[*]} ${build_cmd[*]})"
echo "- data: $DATA/locomo-conv26.json (199 q / 197 scored), $DATA/lme-oracle.json (500 q / 479 scored)"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the question and scored counts (199/197, 500/479) are hardcoded into the manifest. the dry-run at line 53 already computes them, so capture and reuse those values and the manifest can't drift from the data.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manifest now records question and scored counts from the dry-run. 002ba59

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at 002ba59. the counts now come from the --dry-run plan, and the self-test asserts 3 questions, 3 scored for both datasets.

Comment thread scripts/experiments/e0s/run_e0s.sh Outdated
echo; cat "$RES/run1/capsule-hist.md"
} > "$RES/MANIFEST.md"
log "E0+sweep complete. determinism diff exit=$DET. manifest $RES/MANIFEST.md"
exit 0

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this exits 0 even when the determinism diff fails (DET != 0). the header documents it, but anything checking rc won't see the nondeterminism. consider exit $DET or a distinct code.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runner now exits with the determinism-diff status. 002ba59

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at 002ba59. the script ends with exit "$DET", and the header documents 1 = determinism diff failed.

Comment thread scripts/experiments/e0s/run_e0s.sh Outdated
@@ -0,0 +1,133 @@
#!/usr/bin/env bash
# E0+sweep runner (#68). Spec: gauge, ML room, 2026-11-13. Prepared by fuse.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: "gauge, ML room" and "Prepared by fuse" are team-internal names in repo text. keep it neutral and reference #68 only. same for the spec: line in the manifest.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Header and manifest now say E0 sweep (#68). 002ba59

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed at 002ba59. no bot or room names in code or output. the only ones left are /workspace/fuse paths in the provenance comment for the recorded run, and that's fine.

cursoragent and others added 4 commits November 13, 2026 14:12
rerank_off omitted broker.rerank_min_score, so it differed from fused
defaults in two keys. Build it from those defaults, and assert the
intended key for every arm except lexical.

Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
Start the child in its own session, sum RSS across that process group,
and SIGKILL the group over the limit. Forward SIGTERM and SIGINT to the
group. --self-test checks the cap, exit-code passthrough, and both signals.

Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
Resolve the repo from the script path. Data, output, binary, and commit
come from flags or environment variables, and the commit defaults to
HEAD. The header records the 2026-11-13 run at 61dc9f0 with
/workspace/fuse paths. Dataset counts come from the dry-run, and the
process exits with the determinism-diff status.

Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>
The job runs the Python self-tests and the sweep runner path check. It
does not build Rust or download models.

Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>

@RodCor RodCor left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-review at d4c4ac9: fail, narrowly. the must-fix is closed, but 1 should-fix is still open and the pr body is out of date.

history: linear, 4 commits on cc7b729 (0e64812, d913f3c, 002ba59, d4c4ac9). cc7b729 is an ancestor, so no force-push.

threads, 6 of 7 closed:

new should-fix, pr body: these lines are now false.

  • "No Rust, Cargo or workflow files are changed": ci.yml is changed.
  • "byte-identical copies of the scripts used for the box run": memcap.py and run_e0s.sh were rewritten after the run, so the hashes in wrapper-sha256.txt match cc7b729, not the head.
  • "kills only its own process tree": it now kills the process group.

say which commit the recorded run's scripts came from, and list the eval-scripts job under tests.

ci on d4c4ac9 (merge ref against main 16e6232): 19 pass, including the new python eval self-tests job (job, 5 of 5 ok). still running: test (windows-latest) and Analyze (rust). cargo-audit fails on RUSTSEC-2026-0523/0524 (quinn-proto), 0538 (quinn-udp), 0486 (rustix, unsound) and 0441 (pin-utils, unmaintained). that's main's Cargo.lock, which this pr doesn't touch, and #78 fixes it.

nothing in this pr touches ADR 0001 areas.

to pass: close r4566145713, fix the body, windows and Analyze (rust) go green, and cargo-audit goes green once #78 lands and this branch picks up main with a merge commit.

Ignore e0-data and e0s-results, and leave those directories out of the
dirty-tree check so a run and a rerun stay clean. The Rust baseline
defaults to the merge-base with origin/main, or HEAD when that ref is
absent. The self-test runs both against a clean clone with a stub binary.

Co-authored-by: Rodrigo <rodrigo.ccordoba@hotmail.com>

@RodCor RodCor left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-review at eedd1fb: pass on code and findings. it's not merge-ready until CI finishes green.

history: 1 commit on d4c4ac9. d4c4ac9 is an ancestor, so no force-push.

findings: all 7 threads are closed. r4566145713 is closed now too. i verified it with my stub repro (default run, rerun, and a local merge into main 16e6232) and with mutations: --self-test fails on the old baseline and on the old .gitignore.

pr body, checked against the code:

  • wrapper-sha256.txt on the box matches cc7b729 for run_e0s.sh, memcap.py and sweep_hist.py.
  • e0_eval.py, retrieval_eval.py, retrieval_eval_profiles.py, prepare_e0_data.py and summarize_e0.py match 61dc9f0.
  • the ci.yml eval-scripts job, the process-group kill with signal forwarding, and "no Cargo.lock change" are all accurate (no crates/Cargo diff vs base).

still open, optional: the memcap orphan note on r4566145704 wasn't addressed. memcap.py is unchanged since d4c4ac9. repro: memcap.py ... -- sh -c 'sleep 47 & exit 0' returns 0 and leaves the sleep running. this doesn't block, and it can be a follow-up.

ci on eedd1fb (merge ref vs 16e6232), as of 14:36 ART:

  • pass: python eval self-tests, clippy, rustfmt, TypeScript SDK, decision provider examples, npm staging regression tests, integration asset drift, cargo/npm version sync, test (ubuntu-latest), test (embeddings, ubuntu), Analyze (python/actions/js-ts) x2. CodeQL is neutral.
  • pending: test (windows-latest), test (macos-15) (queued), Analyze (rust) x2.
  • fail: cargo-audit, on the known set: RUSTSEC-2026-0523/0524 quinn-proto, 0538 quinn-udp, 0486 rustix, 0441 pin-utils. that's main's lockfile, which #78 fixes.

to merge: the pending jobs go green, and cargo-audit goes green after #78 lands and this branch takes main with a merge commit.

@RodCor RodCor left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pass at 8b85e90 for the merge itself. my code pass from eedd1fb carries over.

  • 8b85e90 is a real merge commit with parents 707948b and main eaf8016. eedd1fb is in its history, so there was no force-push.
  • its tree (1ceede3) is exactly what git merge-tree produces from those two parents, so no conflicts were resolved by hand.
  • the net diff vs main has the same stable patch-id (5603ef1) as eedd1fb vs its merge-base: 7 files, +1027/-1. the merge adds no new content.

the only thing left is ci on 8b85e90. comments only, i pushed nothing.

@RodCor
RodCor merged commit b42158d into main Nov 13, 2026
22 checks passed
@RodCor
RodCor deleted the cursor/e0-sweep-profile branch November 13, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants