Repository navigation
Conversation
…ad CPU resets getSessionMetadataSummary/getSession rescanned every message event with json_extract + sum() on each read, so UI polling against a large session exceeded the Durable Object CPU limit (Sentry KILO-CLOUD-AGENT-M). Own the total in one place (cost-total.ts): a durable per-scope running sum updated in the same transaction as every cost-bearing row write, so a DO reset can never leave it stale against committed rows. Reads become two single-row lookups. Existing sessions converge through a bounded, resumable id-keyed backfill (row window + payload byte budget) instead of an unbounded first-read scan. Preserve the exact prior semantics: assistant-only, both message.updated and conversation.message.updated shapes, payload-sessionID attribution, nonnegative filtering, MAX_SAFE_INTEGER clamping, and SQLite boolean-cost coercion. Session wipe clears the total and backfill watermark. Add unit tests for the contribution semantics and Workers-runtime integration tests: a large seeded history reads without rescanning rows (fails on the old full scan), backfill resumes after eviction and counts concurrent writes, replay/import/replacement/removal keep the total exact, and both wipe paths clear it.
Code Review SummaryStatus: 2 Issues Found | Recommendation: Address before merge Executive SummaryThe durable cost total and its id-keyed backfill reproduce the removed full-scan semantics, but the backfill window still materializes every payload it touches and the read path now performs an unguarded write. Assumptions: Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (11 files)
Reviewed by deepseek-flash-latest · Input: 349.7K · Output: 56K · Cached: 6M Review guidance: REVIEW.md from base branch |
iscekic
left a comment
There was a problem hiding this comment.
bot: Automated code review for this pull request, requested by @eshurakov.
Code Review Summary
Status: 3 Issues Found | Recommendation: Address before merge
Executive Summary
The durable (session_id, kilo_session_id) cost total with an id-keyed resumable backfill reproduces the removed sum(json_extract(...)) scan faithfully; the remaining risk is on the backfill read path, where the per-read byte budget only counts message rows and the metadata poll now runs an unguarded write transaction.
Assumptions: [INFERENCE] No production replay was performed, so the CPU headroom claim for the specific large session is unverified. The partial total reported while a pre-existing session's backfill converges is intentional per the PR description; for a very large session that is one 2048-row window per read, so the displayed total can stay low across many polls before converging.
Overview
| Severity | Count |
|---|---|
| CRITICAL | 0 |
| WARNING | 2 |
| SUGGESTION | 1 |
Issue Details (click to expand)
WARNING
| File | Line | Issue |
|---|---|---|
services/cloud-agent-next/src/control-plane/session/cost-total.ts |
194 | The 4 MiB payload budget accumulates only message/ candidates, so the window is bounded by 2048 rows regardless of size; a heavy window that cannot finish rolls back and is retried forever from the same cursor. |
services/cloud-agent-next/src/control-plane/session/cost-total.ts |
261 | totalCostMicrodollars runs an unguarded backfillStep() write from getSession, getSessionMetadataSummary and both page readers, so a failing step fails a read that previously only flushed part text through error-isolated onFlushError. |
SUGGESTION
| File | Line | Issue |
|---|---|---|
services/cloud-agent-next/test/integration/control-plane-session-cost-total.test.ts |
225 | The byte-budget break and its cutoff handling are never exercised by the seeded history, because every seeded payload is small and the first window is all message rows. |
What I checked
- Semantics parity with the removed scan.
messageCostContributionmatches the old SQL predicate for predicate:kilocoderows only, themessage/…message0range (equivalent tostartsWith('message/')under BINARY collation), bothmessage.updated/infoandconversation.message.updated/messageshapes,assistantrole only, attribution to the payload's ownsessionID, and the sametypeof cost in ('integer','real')→max(0, cost)coercion (including JSON booleans contributing one dollar and non-numeric costs contributing zero). Overflow still yieldsInfinity, whichtotalCostMicrodollarsreports as0exactly as the oldNumber.isFiniteguard did. - The incremental total. Every production write to
eventsgoes through the control-planewrite()wrapper (insert,upsert,insertUnique), including native events, kilo2 conversation events and the part-text-buffer flush, andmessage.removedmaps to the samemessage/<id>entity so the before/after delta subtracts the removed cost. Raw inserts intoeventsexist nowhere insrc/, so nothing bypasses the total after the backfill completes. Rows written ahead of the cursor are skipped and counted once when the window reaches them;sessionCostTotalsandcostBackfillStateare cleared in both wipe paths, anddeleteAllre-runsmigrateso the new tables exist again. - Overflow recovery. A poisoned non-finite total is replaced by
recomputeScopeon the next cost-bearing write, scoped to the cursor while the backfill is running, so it does not double count the uncounted range. - Migration artifacts.
0007_session_cost_totals.sql,migrations.js,_journal.jsonand the0007snapshot are consistent, the snapshotprevIdchains from0006, and the schema defaults match the generated SQL. - Tests. The new suites would fail without the change; the integration cases cover bounding, eviction resume, write-while-backfilling, both payload shapes, child-scope isolation, clamping, removal, replay/history import, failed-transaction rollback and both wipe paths.
Not verified here
Nothing is installed in this checkout, so I ran no test, typecheck or lint. The Workers-runtime suite in test/integration/control-plane-session-cost-total.test.ts is the right place to prove the backfill behaviour, and it was not executed by this review.
Act on first
Line 194: count row.bytes for every row in the backfill window, not just message/ candidates, so one metadata poll cannot pull an arbitrary payload volume — and so a window that cannot finish cannot wedge the cursor.
E2E: not run.
No end-to-end or integration check was executed. Nothing is installed in this checkout and this job installs no dependencies, so the Workers-runtime suite (test/integration/control-plane-session-cost-total.test.ts), the unit suite and typecheck could not be run here. I reviewed the change statically against the removed SQL and the call paths. For context only, the checks reported at head 9b1704a show verify and format-check as SUCCESS; that is the PR's own CI record, not a run performed by this review.
Every remark is attached to the changed line where GitHub accepts one.
This service never pushes, rebases, forces, closes, labels, or merges, and it posts nothing else on this pull request.
| for (const row of window) { | ||
| if (row.stream_event_type === 'kilocode' && row.entity_id?.startsWith('message/')) { | ||
| if (bytes + row.bytes > COST_BACKFILL_BYTE_BUDGET && cutoff !== state.cursor) break; | ||
| bytes += row.bytes; |
There was a problem hiding this comment.
WARNING: The 4 MiB payload byte budget only counts message/ rows, so a backfill step is bounded by row count alone, not by bytes.
bytes is incremented only inside the candidate branch (lines 192-196), while the window select at lines 172-183 still measures and reads length(cast(payload as blob)) for every row it returns. part/ rows are the bulk of a large session's bytes and are never candidates, so a window of 2048 rows can touch far more than COST_BACKFILL_BYTE_BUDGET on a single metadata poll. That matters because backfillStep() is invoked from getSession / getSessionMetadataSummary (session-do.ts:2875 readSessionSummary) and from both history page readers, i.e. the exact polling path this PR exists to make cheap.
The failure mode is not just a slow read: the step runs inside storage.transactionSync. If one step cannot finish (CPU limit, storage error), the transaction rolls back, cost_backfill_state.cursor never advances, and the next poll retries the same window. A session whose window is too heavy therefore never converges.
Smallest fix: accumulate row.bytes for every row in the window (before the candidate check) and break on the budget, keeping the existing "always take at least one row" rule so the cursor still strictly advances; then the per-read payload bound matches the 4 MiB the PR describes.
| addDollars(afterContribution, afterContribution.dollars, state); | ||
| }, | ||
| totalCostMicrodollars(sessionId: string, kiloSessionId: string): number { | ||
| backfillStep(); |
There was a problem hiding this comment.
WARNING: A previously read-only metadata read now performs an unguarded write transaction.
totalCostMicrodollars calls backfillStep() (line 167), which opens storage.transactionSync and writes session_cost_totals + cost_backfill_state. The callers are getSession, getSessionMetadataSummary and both history page readers (history.ts:180, conversation-history.ts:100). While the backfill is incomplete, any throw out of that transaction (storage failure, a step over the DO's CPU budget) propagates and fails the whole read — the caller gets no snapshot at all instead of the counted-so-far total.
The file already has the right precedent for a write driven by a read: readSessionSummary flushes buffered part text, and createPartTextBuffer takes onFlushError (session-do.ts:849-855) so a failed flush is logged and the read still returns.
Smallest fix: wrap the backfillStep() call in a try/catch, log at error level with the session id, and continue to readDollars(...); the next read retries the step from the unchanged cursor. That keeps the metadata read serving the explicit partial total the PR documents instead of turning it into a hard failure.
| it('reads large stored history through a bounded backfill, then without rescanning rows', async () => { | ||
| const { stub, sessionId, kiloSessionId } = await fixture(); | ||
| const assistants = 5000; | ||
| await seedStoredHistory(stub, sessionId, kiloSessionId, assistants, 1000); |
There was a problem hiding this comment.
SUGGESTION: The byte-budget break in backfillStep is the one branch with no test, and it is the branch that can silently lose cost.
Every seeded payload here is small (to a 2048-byte part at line 136, ~250-byte assistant payloads), and the first window is entirely message/ rows, so COST_BACKFILL_BYTE_BUDGET is never reached and the break at cost-total.ts:193 never runs. That branch decides cutoff, and cutoff decides both which row the next step resumes from and which id range recomputeScope counts. A one-row-off mistake there drops a message's cost with no other test noticing, because the parity assertions all run after the backfill is complete.
A case worth adding: seed a few byte-heavy non-message rows (or one assistant payload above the 4 MiB budget) ahead of the assistant rows, then assert (a) the first step's cursor stops before the un-counted row, (b) the total after convergence still equals referenceTotalMicrodollars. That is the missing case the change's own risk demands, and it also pins down the behaviour I raised on line 194.
What
Fixes Sentry KILO-CLOUD-AGENT-M:
getSessionMetadataSummary/getSessionrescanned everymessage event with
json_extract+sum()on every read, so UI polling against a largesession repeatedly exceeded the Durable Object CPU limit and reset the DO
(
operation:getSessionMetadataSummary … "Durable Object exceeded its CPU time limit and was reset.").The same full scan also ran in
getSessionand in the v1/v2 history page readers.How
A single owner for the total —
src/control-plane/session/cost-total.ts:(sessionId, kiloSessionId)in a newsession_cost_totalstable,updated in the same
transactionSyncas every cost-bearing row write (insert / upsert /replace / replay / import). Because the total commits atomically with the row, a DO reset can
never leave it stale against committed rows — no per-read reconciliation scan is needed.
over the whole message range.
getSession,getSessionMetadataSummary, and both history readersdelegate to the owner; the public response shape is unchanged.
one row window (2048 rows) plus a payload byte budget (4 MiB) per read, persisting progress
durably. The first read after deploy does not rescan unbounded history; it converges over a
few reads and survives eviction/reset mid-backfill. Writes made while the backfill runs are
counted exactly once (rows ahead of the cursor are counted when reached; counted rows apply a
delta).
message.updated(v1info)and
conversation.message.updated(nativemessage) shapes, attribution to the payload's ownsessionID, nonnegative filtering,MAX_SAFE_INTEGERmicrodollar clamping, and SQLite'sJSON-boolean→integer cost coercion. A pathological overflowing cost poisons/recovers the sum the
same way the old scan did.
deleteAlland transactional-locator paths) clears the total and thebackfill watermark. History replay, delete, and reset recovery are otherwise untouched.
Schema + migration
0007_session_cost_totals(Drizzle DO SQLite, generated artifacts committed).Verification
Workers-runtime (Miniflare/workerd) integration tests drive the real Session DO and SQLite
storage — not mocks:
test/integration/control-plane-session-cost-total.test.ts:steady-state read of only single-row lookups with no aggregate cost scan. Confirmed to fail
on the old full-scan behavior (reads all ~5000 rows per read) and pass with the fix.
edge costs stay exactly equal to a verbatim copy of the old full scan (parity reference).
a failed write transaction rolls the total back.
src/control-plane/session/cost-total.test.tscovers the contribution semantics.transport (replay), rpc, worktree-deletion, scope-handle, and the migration-bundle test.
Checks run (scoped to
services/cloud-agent-next):pnpm test— 8007 unit tests pass (325 files).pnpm typecheck— clean.oxlinton changedsrc— clean;oxfmtapplied to changed files.pnpm check:duplication— no new duplication.Limitations
runtime, but they cannot prove production CPU behavior for the specific large session in the
Sentry report (no production replay was performed). The fix removes the per-read full scan that
caused the reset; actual production CPU headroom should be confirmed after deploy.
sandbox/wrapper E2E matrix, which does not exercise this storage path differently.
the counted-so-far sum and reaches the exact value once the cursor completes; new sessions are
exact from the first read.