Skip to content

fix(steward): paginate open PR signal GraphQL, allow signal-unavailable monitor recovery, and cache private delivery scans - #9093

Merged
loopx-agent merged 1 commit into
mainfrom
codex/fix-steward-monitor-graphql-recovery-20261120
Nov 19, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/fix-steward-monitor-graphql-recovery-20261120

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

  • Paginate open PR signal collection (github_pr_evidence.py): Replace single unpaginated gh pr list --limit 100 --json ...,reviews,statusCheckRollup calls (which repeatedly hit GitHub GraphQL 502/504 timeouts on larger open PR sets) with bounded gh api graphql pagination (first: 20, up to 5 pages = 100 open PRs, 20s timeout per page), normalizing reviews.nodes and CheckRun/StatusContext rollup items, deduplicating across page shifts, and failing closed on GraphQL/HTTP errors.
  • Unblock stuck signal monitors & prevent due-queue starvation (steward_managed_mode.ts, steward_monitor.py, monitor_poll_admission.ts):
    • Admit at most one bounded zero-effect restart recovery (predecessor === null) when a custom Steward signal monitor fails with runtime_error ("steward monitor external signal source temporarily unavailable") after manager_context and before any poll receipt.
    • Iterate through all due monitors with per-todo exception handling in pump_steward_monitors so one failing or unrecoverable due[0] does not starve due[1:] or runnable advancement todos, and preserve the 60s backoff cache when remaining <= 0.
    • Align dueMonitorAllowed in monitor_poll_admission.ts for obligation === "attempt_due_monitor" with the existing due_monitor_context path so a verified registry_due_monitor claimed by decision.agent_id is admitted when requires_user_action is false even when MONITOR_DUE_ITEM_LIMIT = 1 keeps due_monitor_candidates to due[0].
  • Reduce pending_delivery_paths cold-start burst & globbing (private_conversations.py): Cache historical-answer-images (dir_exists, mtime_ns) in _planned_historical_repairs() and stagger steady-state ineligible probe deadlines across 60 + (zlib.crc32(path.name) % 60) seconds while keeping eligible recovery candidates on the exact 60s cadence.

Local Verification

  • node --experimental-strip-types --test tests/control_plane_ts/steward_managed_mode.test.ts tests/control_plane_ts/quota_monitor_poll_commit.test.ts tests/control_plane_ts/auxiliary_monitor_settlement.test.ts: 80 passed, 0 failed.
  • uv run --extra test ruff check ... across touched Python files: All checks passed!.
  • uv run --extra test pytest tests/test_native_steward_monitor.py tests/test_steward_pr_merge_and_review_signal.py tests/test_lark_steward_directory_scan_perf.py tests/test_lark_private_conversations.py -k "signal_unavailable or observe_github_repository_prs or caches_historical_repairs or rediscovered or negative_and_attempt_backoff" -q: 9 passed, 749 deselected.
  • Full focused suite (tests/control_plane/test_dependency_dispatch_cli.py tests/control_plane/test_auxiliary_scoped_gate.py tests/control_plane/test_monitor_unavailable_closeout_cli.py tests/control_plane/test_monitor_poll_shell.py tests/test_steward_pr_merge_and_review_signal.py tests/test_steward_generic_monitor_todo.py tests/test_lark_steward_directory_scan_perf.py tests/test_native_steward_monitor.py): 248 passed, 10 skipped; only 2 pre-existing unrelated baseline failures in test_dependency_dispatch_cli.py::test_local_wait_and_user_gates_preserve_only_verified_independent_work (todo update --unblocks-todo-id returning user_gate_link_target_invalid).
  • uv run --extra test loopx --format json canary premerge --from-git-diff --git-diff-base origin/main: ok: true (direct_checks 4/4, catalog_canaries 10/10, risk_profile_smokes 8/8).

…le monitor recovery, and cache private delivery scans

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · Gemini · Google (self_reported)

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

  • affected_caller_or_operator: 启用了 Steward 原生自定义外部信号监控(如 GitHub PR 队列监听)与 Lark 私聊投递轮询的守护进程维护者及终端用户。
  • before_after_scenario: 修改前:observe_github_repository_prs 单次请求 100 条 PR 与 reviews/statusCheckRollup,频繁触发 GitHub GraphQL 502/504 超时使信号变为 unavailable;一旦某个 custom signal monitor 在 manager_context 之后抛出 signal-unavailable runtime_error,admitMonitorRestartRecovery 拒绝恢复,且 pump_steward_monitors 只取 due[0] 导致同一绑定下的其余到期监控永久饥饿;若仅剩纯 monitor 队列(obligation=attempt_due_monitor),dueMonitorAllowed 又因 MONITOR_DUE_ITEM_LIMIT=1 只放行 due[0] 而拒绝后续 due[1:] 提交。同时 pending_delivery_paths 每次轮询全量 glob historical-answer-images 并将数千条已投递文件锁死在同一 60s 到期秒产生冷启动/周期脉冲。修改后:GraphQL 按每页 20 条最多 5 页分页并在任意页异常时 fail-closed;signal-unavailable 允许满足严格零副作用证明时最多 1 次重启恢复;pump_steward_monitors 遍历所有到期项且 attempt_due_monitor 在无用户 gate 时允许经由 canonical authority 验证的 registry_due_monitor 放行;_planned_historical_repairs 缓存目录 (dir_exists, mtime_ns),并对无资格历史文件将下次探测分散到 [60, 120)s。
  • observable_outcome: 多页开放 PR 列表采集可在分页与节点归一化单测及 5 页实测基准中稳定完成且故障仍 fail-closed;因外部信号不可用失败的自定义 monitor 在下一轮宿主泵中可完成最多 1 次零副作用恢复,连续失败时不会饿死同绑定其余到期 monitor(端到端本地单测通过,TS 控制面 80/80 通过,canary premerge 18/18 通过);冷启动后的无效历史重扫从全目录同步过期分散为文件名 CRC32 取模后的 60s 平滑窗口。
  • non_goals: 不增大 MONITOR_DUE_ITEM_LIMIT 膨胀 quota decision 体积;不放宽存在 requires_user_action 时的门控边界;不修改远程 GitHub CI 等待策略或绕过 DCO/Review 门禁。

在实际长期运行的 com.loopx.chat 个人管家守护进程中,三个相互放大的运维问题叠加成了持续卡死与无效唤醒:一方面单次拉取 100 条 open PR 及其审查和检查摘要容易命中上游超时,另一方面信号偶发不可用会同时触发恢复白名单缺失与 due[0] 队头阻塞。PR #9093 在提交 ba033be4077eebaa224137fe51854c9827c5514a 将这三个热点在各自权威边界内一并收口,不改动外部用户交互格式与远端 CI 等待契约。

改动思路 / Approach

整体方案严格遵循 TypeScript 决策权威与 Python 执行壳分层,各模块只负责自身既有职责,不引入第二状态源或全局扫描开关:

  1. 在 GraphQL 数据采集层(github_pr_evidence.py)用 gh api graphql 按每页 20 条、最多 5 页游标分页替代单次 --limit 100 全量查询,保持返回值 schema 与 fail-closed 语义完全一致。
  2. 在监控恢复准入层(steward_managed_mode.ts)与宿主调度层(steward_monitor.py + monitor_poll_admission.ts)协同打通恢复与轮转路径:前者在严格零副作用前提下放行单次 signalUnavailable 恢复,后者将 due[0] 硬编码改成带单条异常隔离的 for todo in due: 循环并在控制面准入中对齐 attempt_due_monitor 与 due_monitor_context 对 registry_due_monitor 的一致性语义,同时严守 requires_user_action 门控隔离。
  3. 在 Lark 投递目录发现层(private_conversations.py)保留冷启动即时发现与有资格恢复候选的精确 60s 重试节奏,仅用 (dir_exists, mtime_ns) 缓存静态清单并按文件名 CRC32 错峰无资格历史项的稳态再探测。

当前边界(current_pr_boundary):本 PR 仅修复 Steward 外部信号分页采集、单次零副作用恢复与多到期监控防饥饿,以及 Lark 私聊目录探测缓存/错峰;不包含未迁移旧 canary Goal 归档数据清理与服务端本地发布包激活(需在合并后于运行环境单独执行)。。
架构保留原因(architecture_assessment.reason):每项改动均落在既有职责边界内:GitHub GraphQL 分页采集保留在 python IO 辅助函数,信号恢复准入与纯 monitor 队列 registry 回退保持在 TypeScript 控制面,宿主泵循环与目录探测缓存保留在对应 runtime 适配器。

具体改动 / Changes

无独立书面 RFC,本改动直接对照既有 steward_managed_mode 零副作用重启恢复契约与 monitor_poll_admission 准入契约实施:

关键代码讲解

  1. observe_github_repository_prs & _normalize_graphql_pr_node(github_pr_evidence.py#L452-L567)
    使用 gh api graphql 对 open PR 按 UPDATED_AT DESC 每次拉取 20 条、最多 5 页并设单页 timeout=20。_normalize_graphql_pr_node 将 GraphQL 嵌套的 reviews.nodes 与 commits.nodes[0].commit.statusCheckRollup.contexts.nodes(含 CheckRun 与 StatusContext)展平为 summarize_github_pr_signal 既有输入结构;若跨页翻页期间 PR 更新导致重复,按 number 字典去重并排序。任意一页返回非零退出码、JSON 异常、存在 errors 列表或 hasNextPage 与 endCursor 矛盾时立刻 fail-closed 返回 {"status": "unavailable"}。
  2. admitMonitorRestartRecovery(steward_managed_mode.ts#L24-L100)
    在保留原有 server_restarted、scopeRefused、identityRefused 之外新增 signalUnavailable 分支:严格限定 error_code === "runtime_error"、error === "steward monitor external signal source temporarily unavailable"、contexts.length === 1(manager_turn_context_v1)、单个 manager_context phase、合法四类 event_kinds、poll_absent === true、lease_available === true、upstream_turn_id === null 且 predecessor === null(继承既有限制,每个截止窗口至多恢复 1 次)。
  3. pump_steward_monitors(steward_monitor.py#L446-L492)
    从只取 due[0] 改为遍历 for todo in due:,单条 submit(...) 抛出运行异常时记日志并 continue 到下一条;只有新创建 turn(created 为真)或已有在途 turn(status in {"queued", "running"})才置 dispatched = True 并退出当前绑定调度。同时仅当 remaining > 0 时缩减 60s 退避缓存,避免失败监控在过去截止时间上出现 remaining == 0 忙等轮询。
  4. dueMonitorAllowed(monitor_poll_admission.ts#L65-L78)
    此前 MONITOR_DUE_ITEM_LIMIT = 1 使 attempt_due_monitor 场景的 due_monitor_candidates 只含 due[0];本修改在不扩大状态包体积的前提下,当 !decision.requires_user_action 时允许经 canonical authority 解析且 due === true && claimed_by === decision.agent_id 的 registry_due_monitor 命中,与已有 due_monitor_context 分支保持对称,同时通过 !decision.requires_user_action 锁死用户门控隔离。
  5. _planned_historical_repairs & pending_delivery_paths(private_conversations.py#L1531-L1589)
    以 (dir_exists, mtime_ns) 缓存 historical-answer-images 文件名集合(兼容测试中 __new__ 构造实例的 getattr 兜底);并将无资格历史回补探测间隔分散到 60 + (zlib.crc32(path.name) % 60) 秒,而保留 repairs 或命中 recovery_candidate 条目的精确 60s 重试周期。

对主干的风险 / Risks

  • 边界与回归剖析:唯一需要注意的安全边界是 dueMonitorAllowed 对 attempt_due_monitor 放宽 registry_due_monitor 回退时是否可能绕过 scoped_user_gate_fallback。由于条件显式要求 !decision.requires_user_action(存在任意 user gate 时仍严格走 due_monitor_candidates + monitorGateAllowsObservation)且要求 registry.claimed_by === decision.agent_id,门控与跨 Agent 隔离均由 TS 原生准入测试正反例覆盖(quota_monitor_poll_commit.test.ts#L294)。
  • 视觉与 UI:本 PR 不涉及前端展示组件、布局或 CSS 改动,visual_review 为 not_applicable。
  • 本地验证与基线失败归因(未参考远端 GitHub CI):
    • TypeScript 控制面单测(node --experimental-strip-types --test tests/control_plane_ts/steward_managed_mode.test.ts tests/control_plane_ts/quota_monitor_poll_commit.test.ts tests/control_plane_ts/auxiliary_monitor_settlement.test.ts):PASS,80 passed, 0 failed。
    • ruff check(涉及 6 个 Python 文件):PASS,All checks passed!。
    • 聚焦 pytest 定向新增测试(observe_github_repository_prs / signal_unavailable / caches_historical_repairs / rediscovered / negative_and_attempt_backoff):PASS,9 passed(验证通过 delivered-priority_read 60s 精确时钟断言及 1 次恢复后切到 due[1] 提交完成)。
    • 完整聚焦回归套件(共 8 个测试文件):248 passed, 10 skipped,另有 2 个用例 FAIL,即 tests/control_plane/test_dependency_dispatch_cli.py::test_local_wait_and_user_gates_preserve_only_verified_independent_work[file] 与 [sqlite],均在行 155 执行 todo update --unblocks-todo-id 时因 user_gate_link_target_invalid 失败;该路径位于 todo update 参数校验,本 PR 10 个改动文件完全不触及 Todo 更新规则,同时其余 test_auxiliary_scoped_gate.py、test_monitor_unavailable_closeout_cli.py、test_monitor_poll_shell.py、test_steward_*、test_lark_steward_directory_scan_perf.py、test_native_steward_monitor.py 全部 PASS,因此归类为 pre_existing_unrelated。
    • canary premerge --from-git-diff --git-diff-base origin/main:PASS,ok: true(direct_checks 4/4、catalog_canaries 10/10 含 semantic-vocabulary 与 budget 检查、risk_profile_smokes 8/8 全过,0 失败)。

我的整体评价 / Verdict

从长程执行价值看,本次提交从根因消除了 GitHub GraphQL 超时、恢复死锁与多监控队头饥饿三条串联故障链,在保持 MONITOR_DUE_ITEM_LIMIT = 1 紧致预算与既有门控隔离的同时恢复了多监控自主推进能力;从用户体验看,减少守护进程无效醒来与全目录扫描脉冲可降低本机 CPU 抖动且无需新增人工干预。各项控制面规则均在现有词表与 TS/Python 分层边界内收敛,无默认策略静默膨胀或并行决策副本,验证证据充足,达到合入标准。视觉评审说明:Runtime/control-plane fix only; inspected changed paths and confirmed zero UI/HTML/CSS files modified.(unverified_scope: none)。

English Verdict: APPROVE

@loopx-agent
loopx-agent merged commit 45c015f into main Nov 19, 2026
31 of 39 checks passed
@loopx-agent
loopx-agent deleted the codex/fix-steward-monitor-graphql-recovery-20261120 branch November 19, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant