Skip to content

feat(agent): record post-compaction fog observations (opt-in JSONL ledger) - #300

Closed
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:pr/post-compaction-fog
Closed

raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:pr/post-compaction-fog

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Summary

Records post-compaction fog observations as an opt-in, read-only JSONL ledger. LangWatch measured a 28-step post-compaction correction window where the model's error rate is 41.9% vs 17.7% baseline (2.37×) — compaction clears the context but leaves the model "foggy" for a while. This PR only counts those observations; it never changes compaction behavior.

Only counts; never decides. The row is measured on the pre-compaction history, appended after a successful compaction, and persistence failure is logged and swallowed (logger.exception) so it can never abort the run it documents.

Changes

  • core/agent_runtime/runner.py (+13): new AgentRunSpec.fog_ledger_path: str | Path | None = None spec field; fail-open write block after successful compaction. None (default) records nothing — zero I/O.
  • core/agent_runtime/harness_metrics.py (new, stdlib-only imports): FogLedger / measure_post_compaction_fog / FogObservation — six-count observation, appended as one JSONL row.
  • tests/test_harness_metrics.py (new): unit coverage for the measurement + ledger.
  • tests/test_session_compaction.py (+52): wiring tests — read the JSONL directly, assert fog_ratio is None (not ∞) when the denominator is zero, assert the opt-in default writes nothing.

Design constraints honored:

  • Read-only w.r.t. compaction: the marker is the nested compaction.reset event only (compaction.py:159); the measurement never imports SUMMARY_PREFIX and does not alter thresholds, prompts, or history in any way.
  • HarnessRecord is not extended — the observation lives in its own ledger, not in the run record.
  • fog_ratio is None when the denominator is zero (never inf/NaN leaking into JSON).
  • Fail-open: any persistence error is caught and logged; the compaction result is returned normally.
  • stdlib-only in harness_metrics.py (no new dependencies).

Tests

  • Local: 75 passed (test_harness_metrics + test_session_compaction + session wiring), ruff clean.
  • Local CI preflight (Windows lane, pr-ready): 5 rounds (r1-r5) on this head - final: PREFLIGHT: KNOWN-RED | jobs=10 | new=0 | known=2 (py3.12/3.13/3.14 legs green; the 2 known reds are pre-existing Windows platform legs matched to the known-reds.txt baseline; rc=1 without --allow-known-red is the expected verdict shape). Two reproducible flakes surfaced mid-flow and were fixed or baselined with isolated-green evidence: an xdist-parallel test_unified_switch flake (baselined [入账20261103]) and a load-timing flake in test_exec_error_exits_nonzero (isolated re-run green x3, baselined [入账20261106]). Desktop-bundle legs green; sanitize=clean; tree=0->0.
  • Note: local Windows lane carries ~100 known platform failures (baseline-qualified); this PR adds no new red beyond them.

Dependency note

This is a single 4-file PR (one intent: record fog observations). harness_metrics.py ships in the same PR as its only consumer (FogLedger has zero call sites outside this diff) — splitting it into a stacked PR would leave an orphan module with zero call sites and a CI-red stacked leg. ~+1555 lines total, acknowledged: ~840 of them are the self-contained harness_metrics.py module plus its tests.

Out of scope (follow-up): manual compact_history calls are excluded from the fog marker (only the nested compaction.reset event counts); extending to manual compaction is a separate change.

…dger)

Port the read-only post-compaction fog measurement as an opt-in AgentRunSpec
field (fog_ledger_path): each successful compaction appends one six-count
FogObservation row (LangWatch: 28-step post-compaction correction window,
error rate 41.9% vs 17.7% = 2.37x). Only counts; never decides — the row is
measured on the pre-compaction history and persistence failure is logged and
swallowed so it cannot abort the run it documents. None (default) records
nothing: zero I/O.

Ships core/agent_runtime/harness_metrics.py (stdlib-only imports) as the
dependency closure for FogLedger/measure_post_compaction_fog, plus the
runner call site and wiring tests.
@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor Author

Splitting this into two smaller PRs, as discussed in the review norm of keeping each PR to one mechanism:

No content is lost: the union of the two PRs is exactly this PR's head (f342fc98) after the module was split at the fog/gate boundary (HarnessMetricsError and the pairing/storage classes stay with the gates; FogObservation / measure_post_compaction_fog / FogLedger move to the new module with their own error type). Closing this one in favour of the two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant