[Intelligence Research Project (sunghunkwag)]

What this adds

When a multi-edit candidate passes the Sleep gate, the report shows which validation tasks moved, but not which edit moved them. The maintainers' reply in #174 left that open: "true per-rule causal attribution would require separate ablation runs and can be explored as a follow-up." This PR is that follow-up.

With gate_edit_ablation: true (default false), each skill or memory candidate the night accepts is replayed on the same validation tasks once per edit, each time without that one edit. Every edit gets a row on its trial: the gate score with and without the edit, an observed_change of higher_with_edit / no_change / lower_with_edit / unmeasured, the validation tasks whose score differs, and the edit's source_task_ids / source_session_ids.

A row is an observed change on one replay, not a causal attribution. The rows are report-only, and ablation runs after every gate decision of the night, so accepted edits and documents are the same with the option on or off, including when the token budget binds.

Example

The #174 shape: one edit lifts a task, another lowers a different task, and the slice mean still rises (0.400 → 0.850). This is what the report renderer produces for that case (the scenario in test_ablation_names_the_edit_behind_a_hidden_regression), shown as source:

### `skill` candidate — accepted (0.400 → 0.850)

| Task | Tags | Baseline | Candidate | Change |
|---|---|---:|---:|---|
| `essay` | essay | 0.000 | 1.000 | improved |
| `router` | router | 0.800 | 0.700 | regressed |

#### Leave-one-out edit ablation

_Observed change on one replay, not a causal attribution. "With" is the replay that passed the gate, so it is selected for scoring high; "Without" is one further replay of the candidate minus that edit. Rows therefore lean toward `higher with edit`, and a nondeterministic backend or judge can move any row by noise._

| Edit | With | Without | Observed (one replay) | Task changes |
|---|---:|---:|---|---|
| 1. \[add\] Rule A: answer the question that was asked. | 0.850 | 0.350 | higher with edit | `essay` 0.000 → 1.000 |
| 2. \[add\] Rule B: always open with a fixed heading. | 0.850 | 0.900 | lower with edit | `router` 0.800 → 0.700 |

The task table already shows that router regressed. The ablation table adds which edit that replay points to, and that the candidate scored 0.900 without it.

Behavior and cost

• Default off. No extra replay, and report.md, report.json and the CLI --json output are unchanged. The only default-visible change is a gate_edit_ablation: false field in diagnostics.json and in the logged cycle config.
• Runs after every decision. A trial only records what its ablation needs. The replays run after the final decision of an accepted epoch; run_sleep_cycle queues them and runs them after the held-out test replay and every fan-out group, and consolidate_groups defers them past its last group. They can use leftover budget but cannot change what a budget-dependent step decides.
• Token accounting. Ablation replays are ordinary model calls: their tokens are included in tokens used and count toward max_tokens_per_night, which is checked before each replay. Replays that do not start leave their rows unmeasured and add one edit ablation entry to the budget note; a skip never marks a fan-out group as cut short. In a fan-out night with a token-counting mock, spend rose from 11,280 to 15,600 tokens (+38%).
• One-edit candidate: no extra replay. The trial already compared the document with and without that edit, so the row reuses it. An N-edit candidate costs up to N validation replays, tagged gate_ablation:<target> in the evidence log.
• Rejected candidates are not ablated: neither a trial the gate rejects (including gate_no_regression) nor an epoch whose final decision rejects.
• An edit that another edit of the same trial overrides is reported as no_change without a replay, because the document is the same without it.
• Edits that only matched through the removed edit (for example a replace anchored on a line that an add in the same trial created) are listed as co_dropped, so the row is not read as an observation about one edit.
• A missing or non-finite ablation score is unmeasured, with no score and no per-task direction.
• gate_mode: off has no validation replay, so the option has no effect there.
• Rows travel through the existing gate_trials path, so they reach report.md, report.json, diagnostics.json, the CLI --json output and the evidence log with the existing redaction. In multi-skill fan-out each group's rows are in the evidence log; surfacing them in report.md is left for a follow-up.

Limits

• Each comparison is a single replay, and the two sides are not symmetric: the "with" score is the replay the gate selected, the "without" score is a fresh one. Rows lean toward higher_with_edit, and a nondeterministic backend or judge can move any row by noise.
• Where the response cache serves a replay, repeating the ablation returns the same result instead of a new sample. Re-replaying the full candidate alongside the ablations would need a cache-bypassing sample and is not part of this PR.
• Leave-one-out does not measure interactions. Two edits that are each redundant given the other both show no_change.
• I ran this with the mock backend and stubbed replays only. I have not run it against a live model backend.

Tests

• tests/test_gate_edit_ablation.py, 61 test cases. Beyond the first version: a token-based budget with evolve_memory=True at every budget from 0 to 10 replays, consolidate_groups under a shared budget from 0 to 12 replays, fan-out and held-out-test nights with a token-counting mock under max_tokens_per_night, the per-replay budget check, the deferral queue, and a rejected final decision.
• python -m pytest -q: 2276 passed, 13 skipped, 4 xfailed on the branch merged with current main (Python 3.13, project dependencies plus the parallel extra).
• ruff check on the changed files: clean.
• python -m mkdocs build --strict: passes.

Docs: docs/sleep/README.md (the option and the nightly token budget section), the advanced-config key list in plugins/README.md, and CHANGELOG.md. I left the per-plugin SKILL.md key lists alone.

🤖 Generated with Claude Code

https://claude.ai/code/session_015s4y5QVk9hfpeGNR7dzrHb

[Intelligence Research Project (sunghunkwag)]

@microsoft-github-policy-service agree

[Yifan Yang (Yif-Yang)]

Thanks, this is a careful and well-tested change. It is off by default, the default report is byte-identical, single-edit and overridden-edit cases avoid extra replays, the redaction path is covered, and I could not find token double-counting. One blocking issue, plus a few suggestions:

1. (blocking) Ablation can change decisions when the nightly token budget binds, and max_tokens_per_night defaults to 400000. Ablation runs inside _gate_apply, right after the skill trial, so its replays spend the budget that the memory step, the final validation replay and later fan-out groups check afterwards. In a small consolidate repro (2 skill edits plus 1 improving memory edit, budget of 4 replays), the memory edit is accepted with ablation off and never proposed with it on (budget_skipped=['memory edits', …]). In a fan-out run_sleep_cycle run with a token-counting mock backend, both groups are accepted with ablation off, while with it on one group is consolidated but not accepted (skipped skill edits, final validation replay) and the other is skipped. Could ablation be deferred until every decision is final (after the final decision in consolidate, and after all groups in fan-out), with a regression test using a token-based budget_exhausted, evolve_memory=True, and a fan-out case? The current invariance test runs with no budget and evolve_memory=False.
2. The budget is checked once per trial rather than before each ablation replay, so a 4-edit trial can run 3 full validation replays after the budget is spent (4000-token cap, 7000 spent). The docs say replays are skipped once the budget is spent. A per-replay check that keeps the single budget_skipped entry would fix it.
3. On interpretation: the "with" score is the replay that just cleared the gate, so it is selected for being high, while each "without" score is a fresh single replay. That biases rows toward positive, and the attempt cache means re-running does not resample. Could the docs and the report caption hedge all three labels (e.g. "observed change on one replay, not a causal attribution"), and perhaps rename effect? Optionally, re-replay the full candidate alongside the ablations when budget allows.
4. Please state explicitly that ablation spend is included in tokens used and counts toward max_tokens_per_night. In my runs, fan-out mock spend rose by about 43%.
5. Nice to have: include each edit's source_task_ids/source_session_ids (Add source task and session provenance to proposed edits #365) in the ablation rows, and a follow-up to surface fan-out group rows in the report, not only the evidence log.

[Intelligence Research Project (sunghunkwag)]

Thanks for the careful review and for the two repros. You were right about the blocking issue: my claim that the option leaves decisions unchanged only held when the budget was not binding. I pushed 941560c on top of the reviewed commit (no force-push) and updated the description to match.

1. Decisions under a binding budget. I reproduced both of your cases first. Ablation replays no longer run inside _gate_apply: a trial only records what its ablation needs, and the replays run after the final decision of an accepted epoch. run_sleep_cycle queues them through edit_ablation_queue and runs them after the held-out test replay and every fan-out group; consolidate_groups defers them past its last group in the same way when it is called on its own. A skipped ablation also no longer lands in a group's budget_skipped, so it cannot mark a group as cut short.

   Tests: a token-based budget_exhausted with evolve_memory=True (your 2 skill edits + 1 memory edit case at 4 replays, then every budget from 0 to 10 replays), consolidate_groups at every budget from 0 to 12, and run_sleep_cycle fan-out and held-out-test nights with a token-counting mock under max_tokens_per_night. On the reviewed commit, with the cap set to the ablation-off spend, the fan-out night skipped the second group and the held-out-test night skipped the test replay. Both now match ablation off.

   Two side effects of the move: a candidate that the final decision rejects is no longer ablated, and consolidate reads call_error / reflect_raw before ablation runs, so those match as well.

2. Per-replay budget check. The budget is checked before each ablation replay, still with a single edit ablation entry. Test: a 4-edit trial with room for one replay runs one and leaves three rows unmeasured.

3. Interpretation. effect is now observed_change, with higher_with_edit / no_change / lower_with_edit / unmeasured. The report caption and the docs say that a row is an observed change on one replay, not a causal attribution, and why rows lean toward higher_with_edit (the "with" score is the replay the gate selected). I did not add the re-replay of the full candidate: through replay_batch it is served from the attempt cache and returns the same sample, so it needs a cache-bypassing sample first. I can do that as a follow-up if you want it.

4. Token accounting. The option's section, the nightly token budget section and the CHANGELOG now state that ablation spend is part of tokens used and counts toward max_tokens_per_night. With my token-counting mock the fan-out night went from 11,280 to 15,600 tokens (+38%), in line with your ~43%.

5. Rows now carry source_task_ids / source_session_ids. Fan-out group rows in report.md are left for a follow-up, as you suggested.

python -m pytest -q: 2276 passed, 13 skipped, 4 xfailed on the branch merged with current main; ruff check on the changed files is clean; mkdocs build --strict passes. This is still mock backend only, with no live model run.