Repository navigation
fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans - #9093
Conversation
…le monitor recovery, and cache private delivery scans Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · Gemini · Google (self_reported)
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
动机 / Motivation
- affected_caller_or_operator: 启用了 Steward 原生自定义外部信号监控(如 GitHub PR 队列监听)与 Lark 私聊投递轮询的守护进程维护者及终端用户。
- before_after_scenario: 修改前:observe_github_repository_prs 单次请求 100 条 PR 与 reviews/statusCheckRollup,频繁触发 GitHub GraphQL 502/504 超时使信号变为 unavailable;一旦某个 custom signal monitor 在 manager_context 之后抛出 signal-unavailable runtime_error,admitMonitorRestartRecovery 拒绝恢复,且 pump_steward_monitors 只取 due[0] 导致同一绑定下的其余到期监控永久饥饿;若仅剩纯 monitor 队列(obligation=attempt_due_monitor),dueMonitorAllowed 又因 MONITOR_DUE_ITEM_LIMIT=1 只放行 due[0] 而拒绝后续 due[1:] 提交。同时 pending_delivery_paths 每次轮询全量 glob historical-answer-images 并将数千条已投递文件锁死在同一 60s 到期秒产生冷启动/周期脉冲。修改后:GraphQL 按每页 20 条最多 5 页分页并在任意页异常时 fail-closed;signal-unavailable 允许满足严格零副作用证明时最多 1 次重启恢复;pump_steward_monitors 遍历所有到期项且 attempt_due_monitor 在无用户 gate 时允许经由 canonical authority 验证的 registry_due_monitor 放行;_planned_historical_repairs 缓存目录 (dir_exists, mtime_ns),并对无资格历史文件将下次探测分散到 [60, 120)s。
- observable_outcome: 多页开放 PR 列表采集可在分页与节点归一化单测及 5 页实测基准中稳定完成且故障仍 fail-closed;因外部信号不可用失败的自定义 monitor 在下一轮宿主泵中可完成最多 1 次零副作用恢复,连续失败时不会饿死同绑定其余到期 monitor(端到端本地单测通过,TS 控制面 80/80 通过,canary premerge 18/18 通过);冷启动后的无效历史重扫从全目录同步过期分散为文件名 CRC32 取模后的 60s 平滑窗口。
- non_goals: 不增大 MONITOR_DUE_ITEM_LIMIT 膨胀 quota decision 体积;不放宽存在 requires_user_action 时的门控边界;不修改远程 GitHub CI 等待策略或绕过 DCO/Review 门禁。
在实际长期运行的 com.loopx.chat 个人管家守护进程中,三个相互放大的运维问题叠加成了持续卡死与无效唤醒:一方面单次拉取 100 条 open PR 及其审查和检查摘要容易命中上游超时,另一方面信号偶发不可用会同时触发恢复白名单缺失与 due[0] 队头阻塞。PR #9093 在提交 ba033be4077eebaa224137fe51854c9827c5514a 将这三个热点在各自权威边界内一并收口,不改动外部用户交互格式与远端 CI 等待契约。
改动思路 / Approach
整体方案严格遵循 TypeScript 决策权威与 Python 执行壳分层,各模块只负责自身既有职责,不引入第二状态源或全局扫描开关:
- 在 GraphQL 数据采集层(github_pr_evidence.py)用
gh api graphql按每页 20 条、最多 5 页游标分页替代单次--limit 100全量查询,保持返回值 schema 与 fail-closed 语义完全一致。 - 在监控恢复准入层(steward_managed_mode.ts)与宿主调度层(steward_monitor.py + monitor_poll_admission.ts)协同打通恢复与轮转路径:前者在严格零副作用前提下放行单次
signalUnavailable恢复,后者将due[0]硬编码改成带单条异常隔离的for todo in due:循环并在控制面准入中对齐attempt_due_monitor与due_monitor_context对registry_due_monitor的一致性语义,同时严守requires_user_action门控隔离。 - 在 Lark 投递目录发现层(private_conversations.py)保留冷启动即时发现与有资格恢复候选的精确 60s 重试节奏,仅用
(dir_exists, mtime_ns)缓存静态清单并按文件名 CRC32 错峰无资格历史项的稳态再探测。
当前边界(current_pr_boundary):本 PR 仅修复 Steward 外部信号分页采集、单次零副作用恢复与多到期监控防饥饿,以及 Lark 私聊目录探测缓存/错峰;不包含未迁移旧 canary Goal 归档数据清理与服务端本地发布包激活(需在合并后于运行环境单独执行)。。
架构保留原因(architecture_assessment.reason):每项改动均落在既有职责边界内:GitHub GraphQL 分页采集保留在 python IO 辅助函数,信号恢复准入与纯 monitor 队列 registry 回退保持在 TypeScript 控制面,宿主泵循环与目录探测缓存保留在对应 runtime 适配器。
具体改动 / Changes
无独立书面 RFC,本改动直接对照既有 steward_managed_mode 零副作用重启恢复契约与 monitor_poll_admission 准入契约实施:
关键代码讲解
observe_github_repository_prs&_normalize_graphql_pr_node(github_pr_evidence.py#L452-L567)
使用gh api graphql对 open PR 按UPDATED_AT DESC每次拉取 20 条、最多 5 页并设单页timeout=20。_normalize_graphql_pr_node将 GraphQL 嵌套的reviews.nodes与commits.nodes[0].commit.statusCheckRollup.contexts.nodes(含CheckRun与StatusContext)展平为summarize_github_pr_signal既有输入结构;若跨页翻页期间 PR 更新导致重复,按number字典去重并排序。任意一页返回非零退出码、JSON 异常、存在errors列表或hasNextPage与endCursor矛盾时立刻 fail-closed 返回{"status": "unavailable"}。admitMonitorRestartRecovery(steward_managed_mode.ts#L24-L100)
在保留原有server_restarted、scopeRefused、identityRefused之外新增signalUnavailable分支:严格限定error_code === "runtime_error"、error === "steward monitor external signal source temporarily unavailable"、contexts.length === 1(manager_turn_context_v1)、单个manager_contextphase、合法四类 event_kinds、poll_absent === true、lease_available === true、upstream_turn_id === null且predecessor === null(继承既有限制,每个截止窗口至多恢复 1 次)。pump_steward_monitors(steward_monitor.py#L446-L492)
从只取due[0]改为遍历for todo in due:,单条submit(...)抛出运行异常时记日志并continue到下一条;只有新创建 turn(created为真)或已有在途 turn(status in {"queued", "running"})才置dispatched = True并退出当前绑定调度。同时仅当remaining > 0时缩减 60s 退避缓存,避免失败监控在过去截止时间上出现remaining == 0忙等轮询。dueMonitorAllowed(monitor_poll_admission.ts#L65-L78)
此前MONITOR_DUE_ITEM_LIMIT = 1使attempt_due_monitor场景的due_monitor_candidates只含due[0];本修改在不扩大状态包体积的前提下,当!decision.requires_user_action时允许经 canonical authority 解析且due === true && claimed_by === decision.agent_id的registry_due_monitor命中,与已有due_monitor_context分支保持对称,同时通过!decision.requires_user_action锁死用户门控隔离。_planned_historical_repairs&pending_delivery_paths(private_conversations.py#L1531-L1589)
以(dir_exists, mtime_ns)缓存historical-answer-images文件名集合(兼容测试中__new__构造实例的getattr兜底);并将无资格历史回补探测间隔分散到60 + (zlib.crc32(path.name) % 60)秒,而保留repairs或命中recovery_candidate条目的精确 60s 重试周期。
对主干的风险 / Risks
- 边界与回归剖析:唯一需要注意的安全边界是
dueMonitorAllowed对attempt_due_monitor放宽registry_due_monitor回退时是否可能绕过scoped_user_gate_fallback。由于条件显式要求!decision.requires_user_action(存在任意 user gate 时仍严格走due_monitor_candidates+monitorGateAllowsObservation)且要求registry.claimed_by === decision.agent_id,门控与跨 Agent 隔离均由 TS 原生准入测试正反例覆盖(quota_monitor_poll_commit.test.ts#L294)。 - 视觉与 UI:本 PR 不涉及前端展示组件、布局或 CSS 改动,
visual_review为not_applicable。 - 本地验证与基线失败归因(未参考远端 GitHub CI):
- TypeScript 控制面单测(
node --experimental-strip-types --test tests/control_plane_ts/steward_managed_mode.test.ts tests/control_plane_ts/quota_monitor_poll_commit.test.ts tests/control_plane_ts/auxiliary_monitor_settlement.test.ts):PASS,80 passed, 0 failed。 ruff check(涉及 6 个 Python 文件):PASS,All checks passed!。- 聚焦 pytest 定向新增测试(
observe_github_repository_prs/signal_unavailable/caches_historical_repairs/rediscovered/negative_and_attempt_backoff):PASS,9 passed(验证通过delivered-priority_read60s 精确时钟断言及 1 次恢复后切到due[1]提交完成)。 - 完整聚焦回归套件(共 8 个测试文件):
248 passed, 10 skipped,另有 2 个用例 FAIL,即tests/control_plane/test_dependency_dispatch_cli.py::test_local_wait_and_user_gates_preserve_only_verified_independent_work[file]与[sqlite],均在行 155 执行todo update --unblocks-todo-id时因user_gate_link_target_invalid失败;该路径位于todo update参数校验,本 PR 10 个改动文件完全不触及 Todo 更新规则,同时其余test_auxiliary_scoped_gate.py、test_monitor_unavailable_closeout_cli.py、test_monitor_poll_shell.py、test_steward_*、test_lark_steward_directory_scan_perf.py、test_native_steward_monitor.py全部 PASS,因此归类为pre_existing_unrelated。 canary premerge --from-git-diff --git-diff-base origin/main:PASS,ok: true(direct_checks4/4、catalog_canaries10/10 含 semantic-vocabulary 与 budget 检查、risk_profile_smokes8/8 全过,0 失败)。
- TypeScript 控制面单测(
我的整体评价 / Verdict
从长程执行价值看,本次提交从根因消除了 GitHub GraphQL 超时、恢复死锁与多监控队头饥饿三条串联故障链,在保持 MONITOR_DUE_ITEM_LIMIT = 1 紧致预算与既有门控隔离的同时恢复了多监控自主推进能力;从用户体验看,减少守护进程无效醒来与全目录扫描脉冲可降低本机 CPU 抖动且无需新增人工干预。各项控制面规则均在现有词表与 TS/Python 分层边界内收敛,无默认策略静默膨胀或并行决策副本,验证证据充足,达到合入标准。视觉评审说明:Runtime/control-plane fix only; inspected changed paths and confirmed zero UI/HTML/CSS files modified.(unverified_scope: none)。
English Verdict: APPROVE
Summary
github_pr_evidence.py): Replace single unpaginatedgh pr list --limit 100 --json ...,reviews,statusCheckRollupcalls (which repeatedly hit GitHub GraphQL 502/504 timeouts on larger open PR sets) with boundedgh api graphqlpagination (first: 20, up to 5 pages = 100 open PRs, 20s timeout per page), normalizingreviews.nodesandCheckRun/StatusContextrollup items, deduplicating across page shifts, and failing closed on GraphQL/HTTP errors.steward_managed_mode.ts,steward_monitor.py,monitor_poll_admission.ts):predecessor === null) when a custom Steward signal monitor fails withruntime_error("steward monitor external signal source temporarily unavailable") aftermanager_contextand before any poll receipt.duemonitors with per-todo exception handling inpump_steward_monitorsso one failing or unrecoverabledue[0]does not starvedue[1:]or runnable advancement todos, and preserve the 60s backoff cache whenremaining <= 0.dueMonitorAllowedinmonitor_poll_admission.tsforobligation === "attempt_due_monitor"with the existingdue_monitor_contextpath so a verifiedregistry_due_monitorclaimed bydecision.agent_idis admitted whenrequires_user_actionisfalseeven whenMONITOR_DUE_ITEM_LIMIT = 1keepsdue_monitor_candidatestodue[0].pending_delivery_pathscold-start burst & globbing (private_conversations.py): Cachehistorical-answer-images(dir_exists, mtime_ns)in_planned_historical_repairs()and stagger steady-state ineligible probe deadlines across60 + (zlib.crc32(path.name) % 60)seconds while keeping eligible recovery candidates on the exact 60s cadence.Local Verification
node --experimental-strip-types --test tests/control_plane_ts/steward_managed_mode.test.ts tests/control_plane_ts/quota_monitor_poll_commit.test.ts tests/control_plane_ts/auxiliary_monitor_settlement.test.ts:80 passed, 0 failed.uv run --extra test ruff check ...across touched Python files:All checks passed!.uv run --extra test pytest tests/test_native_steward_monitor.py tests/test_steward_pr_merge_and_review_signal.py tests/test_lark_steward_directory_scan_perf.py tests/test_lark_private_conversations.py -k "signal_unavailable or observe_github_repository_prs or caches_historical_repairs or rediscovered or negative_and_attempt_backoff" -q:9 passed, 749 deselected.tests/control_plane/test_dependency_dispatch_cli.py tests/control_plane/test_auxiliary_scoped_gate.py tests/control_plane/test_monitor_unavailable_closeout_cli.py tests/control_plane/test_monitor_poll_shell.py tests/test_steward_pr_merge_and_review_signal.py tests/test_steward_generic_monitor_todo.py tests/test_lark_steward_directory_scan_perf.py tests/test_native_steward_monitor.py):248 passed, 10 skipped; only 2 pre-existing unrelated baseline failures intest_dependency_dispatch_cli.py::test_local_wait_and_user_gates_preserve_only_verified_independent_work(todo update --unblocks-todo-idreturninguser_gate_link_target_invalid).uv run --extra test loopx --format json canary premerge --from-git-diff --git-diff-base origin/main:ok: true(direct_checks4/4,catalog_canaries10/10,risk_profile_smokes8/8).