Skip to content

feat(agent): count post-compaction fog in an opt-in JSONL ledger - #301

Open
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:pr/post-compaction-fog-ledger
Open

raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:pr/post-compaction-fog-ledger

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Summary

LangWatch measured a ~28-step correction window after a context compaction where the model's error rate is 41.9% vs a 17.7% baseline (2.37x): the summary restores the context budget but not the working state, so the agent keeps re-deriving what it dropped. This records the cheap half of that finding.

Each successful compaction appends one six-count FogObservation row to the opt-in AgentRunSpec.fog_ledger_path. None (the default) does zero I/O.

Only counts; never decides. The seam is the existing compaction.reset checkpoint key (read by key, no import of the compactor), the message list is never mutated, and a failed ledger write is logged and swallowed so a measurement can never abort the run it documents.

Split out of #300, which bundled this mechanism with an unrelated paired-gate framework; #300 is closed in favour of the two smaller PRs.

Changes

  • core/agent_runtime/fog_ledger.py (new, stdlib only): FogObservation (six counts, None rates with no samples, fog_ratio is None — never inf/NaN — when the baseline is zero), measure_post_compaction_fog (window after each checkpoint, retained tail skipped, orphan tool results not counted), FogLedger (append-only JSONL, O_APPEND + fsync).
  • core/agent_runtime/runner.py (+13): AgentRunSpec.fog_ledger_path: str | Path | None = None; after a successful auto-compaction, measure the pre-compaction history and append one row. A failing write is logger.exception-ed and swallowed.
  • tests/test_fog_ledger.py (new): counting, window cap, retained tail, checkpoint detection, error-signal, non-string tool content, rates/ratio edge cases, ledger rows.
  • tests/test_session_compaction.py (+52): wiring — one row per successful compaction, nothing written when compaction is skipped, ledger failure swallowed.

Notes

  • fog_ratio is None rather than infinite when the denominator has no samples, so readers never parse inf/NaN.
  • Rows store raw counts only; readers recompute the ratio, so a rounding change cannot make a persisted ratio drift from its own counts.
  • No new dependencies; no network, no model calls, no global state.

Tests

  • Local: pytest tests/test_fog_ledger.py tests/test_session_compaction.py → 36 passed; ruff check + ruff format clean; pre-commit hooks pass.
  • Related existing suites (test_agent_runner_kernel, test_compaction_memory, test_compaction_tool_call_budget) pass except one pre-existing Windows environment_context failure that reproduces identically on unmodified main.

LangWatch measured a ~28-step post-compaction correction window where the
model's error rate is 41.9% against a 17.7% baseline (2.37x): a summary
restores the context budget but not the working state, so the agent keeps
re-deriving what it dropped. This records the cheap half of that finding.

Each successful compaction appends one six-count FogObservation row to the
opt-in `AgentRunSpec.fog_ledger_path`; `None` (the default) does zero I/O.
Only counts, never decides: the seam is the existing `compaction.reset`
checkpoint key, read by key only, and the message list is never mutated.
A failed ledger write is logged and swallowed so a measurement can never
abort the run it documents. Rates are recomputed by readers; a zero
denominator yields `None`, never `inf`/`NaN`.

Split from HKUDS#300: this PR carries the fog mechanism, its runner call site and
its tests. The paired capability/efficiency gate framework is a separate
change with no runtime consumer yet.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant