Repository navigation
perf(lark,native_chat): back off noop recovery retries and cache recovery lookups - #9096
Conversation
…very lookups Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google (self_reported)
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
- affected_caller_or_operator: 启用了 Lark 私聊与原生外部会话自动恢复轮询(
LarkPrivateConversations、ChatExternalConversations、PrivateDeliveryDiscovery)的守护进程维护者及个人私聊用户。 - before_after_scenario: 修改前:LarkPrivateConversations.pending_delivery_paths 对 historically delivered 且 recovery_candidate=True 但 recover_failed_request 实际无副作用的旧请求每 60s 固定重试一次;_source_requests(row) 在并发状态校验时两次全量扫描外部请求 JSON 目录(约 1650 个文件);_RecoveryDiscoveryReads.user_messages(session_id) 每次发现循环都会重读并全量 JSONL 解析携带大图 base64 attachments 的多 MB messages.jsonl;PrivateDeliveryDiscovery._observe 跳过了所有非 delivered 文件的缓存,导致已退场(triage.retired)卡片记录每秒被重读。修改后:_record_context_recovery_noop_backoff 对未发生变化的连续 no-op 恢复按 60s -> 120 -> 240 -> 480 -> 960 -> 1800s + CRC32 抖动指数退避并在文件指纹变化或进入恢复时清零;_source_request_digests 以 FileChatDiscovery 缓存整条请求规范 SHA-256 摘要;_cached_user_messages 在真实 ChatSessionStore 上按 messages.jsonl 指纹缓存剥离 attachments 后的用户消息视图;PrivateDeliveryDiscovery._observe 缓存最小 {status, triage: {retired}} 视图;并将 test_lark_private_conversations.py:8181 的时钟步进对齐 [60, 120)s 探测间隔。
- observable_outcome: unchanged 历史 no-op 恢复记录在首次 60s 后逐步指数退避至 ~30m+抖动且任一侧 JSON 更新立即唤醒;未变 external-requests、真实 ChatSessionStore.messages 和 retired triage 卡片在稳态轮询下实现零重复读盘与零附件 JSON 解析(32/32 快测、ruff 5 文件全绿、711+1 关联单测通过、canary premerge 10/10 ok: true)。
- non_goals: 不改变任何外部请求恢复准入条件、不自动重试历史失败对话本身、不跳过真实文件的并发变更检测、不修改 TypeScript 控制面策略或远端 CI 等待规则。
在线上 com.loopx.chat 守护进程采样中发现:尽管 PR #9093 已缓存历史修复目录与散列无效探测,空载稳态下仍存在约 39% 平均 CPU 占用与每秒锁文件/全目录扫描脉冲。本次在提交 ffa441fd128bf7db05e150d8c7bff67c1dd69161 沿着采样调用栈把剩余 4 个稳态轮询/解析放大器收敛在既有模块内部,不改变任何业务语义与恢复边界。
改动思路 / Approach
- 投递侧 no-op 指数退避(private_conversations.py):在保留首次 60s 探测窗口的前提下,以
file_change_fingerprint绑定投递记录与 Core 请求文件,对连续指纹未变且recover_failed_request未推进状态的记录按min(1800, 60 * 2**(k-1)) + crc32 % 60指数退避;任一方指纹变化或成功进入accepted/交付时立即清除退避。 - Core 请求并发摘要与用户历史缓存(external_conversations.py):保留
_source_requests向后兼容,新增基于FileChatDiscovery的_source_request_digests替代热路径调用;并仅在未覆盖ChatSessionStore.messages方法的真实存储实例上按messages.jsonl指纹缓存剥离attachments的轻量用户消息投影。 - 已退场 Triage 卡片发现缓存(private_delivery_discovery.py):对
status != "delivered"且带非空triage.retired的记录缓存最小{"status": ..., "triage": {"retired": ...}}视图。 - 测试对齐与新增(tests/test_lark_steward_directory_scan_perf.py、tests/test_lark_private_conversations.py):补充 4 个性能/缓存单测,并将预存
test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks时钟步进从+61对齐到[60, 120)s错峰契约。
当前边界(current_pr_boundary):本 PR 仅收口 4 个稳态轮询 CPU 热点及 1 处预存的 test_lark_private_conversations.py 时钟步进冲突;尚未完成线上 release 重新构建激活与激活后稳态闲置 CPU 实测(需合并后在本机守护进程侧完成)。。
架构保留原因(architecture_assessment.reason):4 处优化均留在原有模块边界内:指数退避与文件指纹校验在 Lark 投递调度层(private_conversations.py),请求摘要与会话用户消息缓存留在外部请求归档/发现层(external_conversations.py),已退场卡片视图缓存留在投递发现层(private_delivery_discovery.py),不新增控制面状态或第二权限源。
具体改动 / Changes
关键代码讲解
_record_context_recovery_noop_backoff&pending_delivery_paths(private_conversations.py#L1562-L1630 / #L1762-L1790)
pending_delivery_paths对已处于_context_recovery_noop_backoff的条目先比对(delivery_fp, native_fp),指纹未变才受限于退避截止时间;reconcile_request在delivered记录成功推进为accepted或完成原生结果交付时清空退避,否则记录指纹与连续次数并按60s -> 120s+jitter -> ... -> 1800s+jitter设置下次探测时刻。_source_request_digests&_cached_user_messages(external_conversations.py#L303-L370)
_project_source_request_digest对单条请求规范 JSON 取 SHA-256,与原_source_requests(row)等价捕获同源任意字段变化;_cached_user_messages通过isinstance(store, ChatSessionStore) and getattr(store.messages, "__func__", None) is ChatSessionStore.messages严格绕过测试 mock,仅缓存真实磁盘存储下的只读字段元组(去除attachments)。PrivateDeliveryDiscovery._observe(private_delivery_discovery.py#L230-L239)
对triage.retired为真的历史捕获缓存轻量只读视图并受限于file_change_fingerprint写后一致性校验,未退场captured记录仍保持每次实时读取。
对主干的风险 / Risks
- 回归与一致性边界:需防止
_cached_user_messages在测试 mockstore.messages时绕过 mock、或退避缓存在外部请求变更时未能及时唤醒。前者已通过getattr(store.messages, "__func__", None) is ChatSessionStore.messages规避,后者已由pending_delivery_paths的指纹比对逻辑及新增单测覆盖。 - 文件行数监控:
private_conversations.py现为 1,999 行,external_conversations.py为 1,934 行,均通过control-plane-maintainability-ratchet-smoke.py门禁。 - 视觉与 UI:零 UI 改动,
visual_review为not_applicable。 - 本地验证记录(未依赖远端 GitHub CI):
ruff check(5 个改动 Python 文件):PASS,All checks passed!pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py:PASS,32 passed in 17.66spytest -q tests/test_lark_private_conversations.py -k test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks:PASS,1 passed(修改第 8181 行前已跑 4 文件完整测试711 passed, 1 failed,唯一失败即该行时钟步进问题,修复后复验通过)loopx canary premerge --from-git-diff --git-diff-base origin/main:PASS,ok: true,direct_checks6/6、catalog_canaries2/2、risk_profile_smokes8/8,0 失败。
我的整体评价 / Verdict
本次改动直击实际运行时的 4 个高频空载 CPU 放大点,实现克制且每条缓存均以真实文件指纹校验为基础并保留原方法向后兼容,经本地单测、Ratchet 与 canary premerge 全量验证通过,满足合入要求。视觉评审说明:Runtime caching and backoff optimization only; verified diff touches zero UI, HTML, or CSS files.(unverified_scope: none)。
English Verdict: APPROVE
Goal & Root Cause
Live profiling of a running personal
com.loopx.chatdaemon (~39% average CPU across ~51m uptime without active turns) traced four interacting background scan/retry loops:_context_recovery_probe_atdeadline every attempt, repeatedly invokingrecover_failed_request._source_requests(row)calledself.pending()(full-directory JSON reads across all request files) before and after native probes._RecoveryDiscoveryReads.user_messages(session_id)was reconstructed per 5s discovery tick and reparsed multi-MBmessages.jsonlhistory (mostly base64 attachments) on every pass.PrivateDeliveryDiscovery._observeskipped caching allstatus != "delivered"records, re-reading retiredtriagecard captures every second.Changes
loopx/extensions/lark/private_conversations.py): track consecutive unchanged-fingerprint noop recoveries ondeliveredrequests with exponential backoff (60sfirst retry, then120 -> 240 -> 480 -> 960 -> 1800s+ CRC32 filename jitter) while immediately clearing backoff if either the delivery receipt or external-request file fingerprint changes, or when recovery actually admits a continuation.loopx/capabilities/native_chat/external_conversations.py): replace_source_requests(row)in recovery/metadata concurrency checks withFileChatDiscovery-backed_source_request_digests(row)(full-JSON SHA-256 per request) and cache fingerprint-validated user-only message projections (role,turn_id,text,origin,created_at, excluding attachments) on realChatSessionStoreinstances.loopx/extensions/lark/private_delivery_discovery.py): cache minimal{status, triage: {retired}}hints whenfile_change_fingerprintverifies an unchanged file.tests/test_lark_steward_directory_scan_perf.py,tests/test_lark_private_conversations.py): added 4 targeted unit/perf regression tests and aligned thetest_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checksclock increment with the[60, 120)sineligible stagger contract.Verification
ruff checkacross all 5 touched files: passed.pytest -q tests/test_lark_steward_directory_scan_perf.py tests/test_chat_file_discovery.py tests/test_context_recovery_discovery.py: 32 passed.pytest -q tests/test_lark_private_conversations.py -k test_private_discovery_skips_unchanged_bodies_but_not_new_work_or_core_checks: 1 passed (after updating line 8181 clock step; prior 4-file suite passed 711/712 before that line change).loopx canary premerge --from-git-diff --git-diff-base origin/main:ok: true, 10/10 selected checks passed (2/2 catalog canaries, 8/8 risk profile smokes),failures: 0.