Repository navigation
feat(agent): record post-compaction fog observations (opt-in JSONL ledger) - #300
Closed
raymondginger2018-sudo wants to merge 1 commit into
Closed
raymondginger2018-sudo wants to merge 1 commit into
raymondginger2018-sudo wants to merge 1 commit into
Conversation
…dger) Port the read-only post-compaction fog measurement as an opt-in AgentRunSpec field (fog_ledger_path): each successful compaction appends one six-count FogObservation row (LangWatch: 28-step post-compaction correction window, error rate 41.9% vs 17.7% = 2.37x). Only counts; never decides — the row is measured on the pre-compaction history and persistence failure is logged and swallowed so it cannot abort the run it documents. None (default) records nothing: zero I/O. Ships core/agent_runtime/harness_metrics.py (stdlib-only imports) as the dependency closure for FogLedger/measure_post_compaction_fog, plus the runner call site and wiring tests.
This was referenced Nov 8, 2026
Contributor
Author
|
Splitting this into two smaller PRs, as discussed in the review norm of keeping each PR to one mechanism:
No content is lost: the union of the two PRs is exactly this PR's head ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Records post-compaction fog observations as an opt-in, read-only JSONL ledger. LangWatch measured a 28-step post-compaction correction window where the model's error rate is 41.9% vs 17.7% baseline (2.37×) — compaction clears the context but leaves the model "foggy" for a while. This PR only counts those observations; it never changes compaction behavior.
Only counts; never decides. The row is measured on the pre-compaction history, appended after a successful compaction, and persistence failure is logged and swallowed (
logger.exception) so it can never abort the run it documents.Changes
core/agent_runtime/runner.py(+13): newAgentRunSpec.fog_ledger_path: str | Path | None = Nonespec field; fail-open write block after successful compaction.None(default) records nothing — zero I/O.core/agent_runtime/harness_metrics.py(new, stdlib-only imports):FogLedger/measure_post_compaction_fog/FogObservation— six-count observation, appended as one JSONL row.tests/test_harness_metrics.py(new): unit coverage for the measurement + ledger.tests/test_session_compaction.py(+52): wiring tests — read the JSONL directly, assertfog_ratioisNone(not ∞) when the denominator is zero, assert the opt-in default writes nothing.Design constraints honored:
compaction.resetevent only (compaction.py:159); the measurement never importsSUMMARY_PREFIXand does not alter thresholds, prompts, or history in any way.HarnessRecordis not extended — the observation lives in its own ledger, not in the run record.fog_ratioisNonewhen the denominator is zero (neverinf/NaNleaking into JSON).harness_metrics.py(no new dependencies).Tests
test_harness_metrics+test_session_compaction+ session wiring), ruff clean.pr-ready): 5 rounds (r1-r5) on this head - final:PREFLIGHT: KNOWN-RED | jobs=10 | new=0 | known=2(py3.12/3.13/3.14 legs green; the 2 known reds are pre-existing Windows platform legs matched to theknown-reds.txtbaseline;rc=1without--allow-known-redis the expected verdict shape). Two reproducible flakes surfaced mid-flow and were fixed or baselined with isolated-green evidence: an xdist-paralleltest_unified_switchflake (baselined [入账20261103]) and a load-timing flake intest_exec_error_exits_nonzero(isolated re-run green x3, baselined [入账20261106]). Desktop-bundle legs green;sanitize=clean;tree=0->0.Dependency note
This is a single 4-file PR (one intent: record fog observations).
harness_metrics.pyships in the same PR as its only consumer (FogLedgerhas zero call sites outside this diff) — splitting it into a stacked PR would leave an orphan module with zero call sites and a CI-red stacked leg. ~+1555 lines total, acknowledged: ~840 of them are the self-containedharness_metrics.pymodule plus its tests.Out of scope (follow-up): manual
compact_historycalls are excluded from the fog marker (only the nestedcompaction.resetevent counts); extending to manual compaction is a separate change.