Skip to content

perf(lark,native_chat): back off noop recovery retries and cache recovery lookups - #9096

Merged
loopx-agent merged 1 commit into
mainfrom
codex/fix-chat-recovery-cpu-spin-20261120
Nov 19, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/fix-chat-recovery-cpu-spin-20261120

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Goal & Root Cause

Live profiling of a running personal com.loopx.chat daemon (~39% average CPU across ~51m uptime without active turns) traced four interacting background scan/retry loops:

  1. Historical delivered requests whose recovery planner returned a no-op were reset to a flat 60s _context_recovery_probe_at deadline every attempt, repeatedly invoking recover_failed_request.
  2. _source_requests(row) called self.pending() (full-directory JSON reads across all request files) before and after native probes.
  3. _RecoveryDiscoveryReads.user_messages(session_id) was reconstructed per 5s discovery tick and reparsed multi-MB messages.jsonl history (mostly base64 attachments) on every pass.
  4. PrivateDeliveryDiscovery._observe skipped caching all status != "delivered" records, re-reading retired triage card captures every second.

Changes

  • Bounded noop recovery backoff + fingerprint invalidation (loopx/extensions/lark/private_conversations.py): track consecutive unchanged-fingerprint noop recoveries on delivered requests with exponential backoff (60s first retry, then 120 -> 240 -> 480 -> 960 -> 1800s + CRC32 filename jitter) while immediately clearing backoff if either the delivery receipt or external-request file fingerprint changes, or when recovery actually admits a continuation.
  • Cached source request digest & lightweight user-message history (loopx/capabilities/native_chat/external_conversations.py): replace _source_requests(row) in recovery/metadata concurrency checks with FileChatDiscovery-backed _source_request_digests(row) (full-JSON SHA-256 per request) and cache fingerprint-validated user-only message projections (role, turn_id, text, origin, created_at, excluding attachments) on real ChatSessionStore instances.
  • Cache retired triage captures (loopx/extensions/lark/private_delivery_discovery.py): cache minimal {status, triage: {retired}} hints when file_change_fingerprint verifies an unchanged file.
  • Regression tests (tests/test_lark_steward_directory_scan_perf.py, tests/test_lark_private_conversations.py): added 4 targeted unit/perf regression tests and aligned the test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks clock increment with the [60, 120)s ineligible stagger contract.

Verification

  • ruff check across all 5 touched files: passed.
  • pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py: 32 passed.
  • pytest -q tests/test_lark_private_conversations.py -k test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks: 1 passed (after updating line 8181 clock step; prior 4-file suite passed 711/712 before that line change).
  • loopx canary premerge --from-git-diff --git-diff-base origin/main: ok: true, 10/10 selected checks passed (2/2 catalog canaries, 8/8 risk profile smokes), failures: 0.

…very lookups

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google (self_reported)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

  • affected_caller_or_operator: 启用了 Lark 私聊与原生外部会话自动恢复轮询(LarkPrivateConversations、ChatExternalConversations、PrivateDeliveryDiscovery)的守护进程维护者及个人私聊用户。
  • before_after_scenario: 修改前:LarkPrivateConversations.pending_delivery_paths 对 historically delivered 且 recovery_candidate=True 但 recover_failed_request 实际无副作用的旧请求每 60s 固定重试一次;_source_requests(row) 在并发状态校验时两次全量扫描外部请求 JSON 目录(约 1650 个文件);_RecoveryDiscoveryReads.user_messages(session_id) 每次发现循环都会重读并全量 JSONL 解析携带大图 base64 attachments 的多 MB messages.jsonl;PrivateDeliveryDiscovery._observe 跳过了所有非 delivered 文件的缓存,导致已退场(triage.retired)卡片记录每秒被重读。修改后:_record_context_recovery_noop_backoff 对未发生变化的连续 no-op 恢复按 60s -> 120 -> 240 -> 480 -> 960 -> 1800s + CRC32 抖动指数退避并在文件指纹变化或进入恢复时清零;_source_request_digests 以 FileChatDiscovery 缓存整条请求规范 SHA-256 摘要;_cached_user_messages 在真实 ChatSessionStore 上按 messages.jsonl 指纹缓存剥离 attachments 后的用户消息视图;PrivateDeliveryDiscovery._observe 缓存最小 {status, triage: {retired}} 视图;并将 test_lark_private_conversations.py:8181 的时钟步进对齐 [60, 120)s 探测间隔。
  • observable_outcome: unchanged 历史 no-op 恢复记录在首次 60s 后逐步指数退避至 ~30m+抖动且任一侧 JSON 更新立即唤醒;未变 external-requests、真实 ChatSessionStore.messages 和 retired triage 卡片在稳态轮询下实现零重复读盘与零附件 JSON 解析(32/32 快测、ruff 5 文件全绿、711+1 关联单测通过、canary premerge 10/10 ok: true)。
  • non_goals: 不改变任何外部请求恢复准入条件、不自动重试历史失败对话本身、不跳过真实文件的并发变更检测、不修改 TypeScript 控制面策略或远端 CI 等待规则。

在线上 com.loopx.chat 守护进程采样中发现:尽管 PR #9093 已缓存历史修复目录与散列无效探测,空载稳态下仍存在约 39% 平均 CPU 占用与每秒锁文件/全目录扫描脉冲。本次在提交 ffa441fd128bf7db05e150d8c7bff67c1dd69161 沿着采样调用栈把剩余 4 个稳态轮询/解析放大器收敛在既有模块内部,不改变任何业务语义与恢复边界。

改动思路 / Approach

  1. 投递侧 no-op 指数退避(private_conversations.py):在保留首次 60s 探测窗口的前提下,以 file_change_fingerprint 绑定投递记录与 Core 请求文件,对连续指纹未变且 recover_failed_request 未推进状态的记录按 min(1800, 60 * 2**(k-1)) + crc32 % 60 指数退避;任一方指纹变化或成功进入 accepted/交付时立即清除退避。
  2. Core 请求并发摘要与用户历史缓存(external_conversations.py):保留 _source_requests 向后兼容,新增基于 FileChatDiscovery 的 _source_request_digests 替代热路径调用;并仅在未覆盖 ChatSessionStore.messages 方法的真实存储实例上按 messages.jsonl 指纹缓存剥离 attachments 的轻量用户消息投影。
  3. 已退场 Triage 卡片发现缓存(private_delivery_discovery.py):对 status != "delivered" 且带非空 triage.retired 的记录缓存最小 {"status": ..., "triage": {"retired": ...}} 视图。
  4. 测试对齐与新增(tests/test_lark_steward_directory_scan_perf.py、tests/test_lark_private_conversations.py):补充 4 个性能/缓存单测,并将预存 test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks 时钟步进从 +61 对齐到 [60, 120)s 错峰契约。

当前边界(current_pr_boundary):本 PR 仅收口 4 个稳态轮询 CPU 热点及 1 处预存的 test_lark_private_conversations.py 时钟步进冲突;尚未完成线上 release 重新构建激活与激活后稳态闲置 CPU 实测(需合并后在本机守护进程侧完成)。。
架构保留原因(architecture_assessment.reason):4 处优化均留在原有模块边界内:指数退避与文件指纹校验在 Lark 投递调度层(private_conversations.py),请求摘要与会话用户消息缓存留在外部请求归档/发现层(external_conversations.py),已退场卡片视图缓存留在投递发现层(private_delivery_discovery.py),不新增控制面状态或第二权限源。

具体改动 / Changes

关键代码讲解

  1. _record_context_recovery_noop_backoff & pending_delivery_paths(private_conversations.py#L1562-L1630 / #L1762-L1790)
    pending_delivery_paths 对已处于 _context_recovery_noop_backoff 的条目先比对 (delivery_fp, native_fp),指纹未变才受限于退避截止时间;reconcile_request 在 delivered 记录成功推进为 accepted 或完成原生结果交付时清空退避,否则记录指纹与连续次数并按 60s -> 120s+jitter -> ... -> 1800s+jitter 设置下次探测时刻。
  2. _source_request_digests & _cached_user_messages(external_conversations.py#L303-L370)
    _project_source_request_digest 对单条请求规范 JSON 取 SHA-256,与原 _source_requests(row) 等价捕获同源任意字段变化;_cached_user_messages 通过 isinstance(store, ChatSessionStore) and getattr(store.messages, "__func__", None) is ChatSessionStore.messages 严格绕过测试 mock,仅缓存真实磁盘存储下的只读字段元组(去除 attachments)。
  3. PrivateDeliveryDiscovery._observe(private_delivery_discovery.py#L230-L239)
    对 triage.retired 为真的历史捕获缓存轻量只读视图并受限于 file_change_fingerprint 写后一致性校验,未退场 captured 记录仍保持每次实时读取。

对主干的风险 / Risks

  • 回归与一致性边界:需防止 _cached_user_messages 在测试 mock store.messages 时绕过 mock、或退避缓存在外部请求变更时未能及时唤醒。前者已通过 getattr(store.messages, "__func__", None) is ChatSessionStore.messages 规避,后者已由 pending_delivery_paths 的指纹比对逻辑及新增单测覆盖。
  • 文件行数监控:private_conversations.py 现为 1,999 行,external_conversations.py 为 1,934 行,均通过 control-plane-maintainability-ratchet-smoke.py 门禁。
  • 视觉与 UI:零 UI 改动,visual_review 为 not_applicable。
  • 本地验证记录(未依赖远端 GitHub CI):
    • ruff check(5 个改动 Python 文件):PASS,All checks passed!
    • pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py:PASS,32 passed in 17.66s
    • pytest -q tests/test_lark_private_conversations.py -k test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks:PASS,1 passed(修改第 8181 行前已跑 4 文件完整测试 711 passed, 1 failed,唯一失败即该行时钟步进问题,修复后复验通过)
    • loopx canary premerge --from-git-diff --git-diff-base origin/main:PASS,ok: true,direct_checks 6/6、catalog_canaries 2/2、risk_profile_smokes 8/8,0 失败。

我的整体评价 / Verdict

本次改动直击实际运行时的 4 个高频空载 CPU 放大点,实现克制且每条缓存均以真实文件指纹校验为基础并保留原方法向后兼容,经本地单测、Ratchet 与 canary premerge 全量验证通过,满足合入要求。视觉评审说明:Runtime caching and backoff optimization only; verified diff touches zero UI, HTML, or CSS files.(unverified_scope: none)。

English Verdict: APPROVE

@loopx-agent
loopx-agent merged commit 3a1f5ea into main Nov 19, 2026
28 of 36 checks passed
@loopx-agent
loopx-agent deleted the codex/fix-chat-recovery-cpu-spin-20261120 branch November 19, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant