Skip to content

Make Auto Reviewer closing lines deterministic and retry invalid drafts once - #12

Merged
ihororlovskyi merged 3 commits into
mainfrom
261112T1021-dev-session
Nov 12, 2026
Merged

ihororlovskyi merged 3 commits into
mainfrom
261112T1021-dev-session

Conversation

@ihororlovskyi

@ihororlovskyi ihororlovskyi commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Fixes #11

  1. The review job writes the closing lines itself: Add review duration drops any trailing Reviewed with: and Next step: lines the model wrote and appends the canonical signature plus a Next step (the model's when valid, else derived from the Summary table counts). It also closes a Details block the model left open. The published format is unchanged.
  2. Pre-validate review draft runs the publish job's own review-actions validator (one definition, shared by YAML anchor) and a Details/readiness/Summary table check. On any error it writes .auto-reviewer/retry-error.txt and one more claude-code-action run (same prompt, anchored) fixes only the listed errors; closing lines, usage (summed over both runs) and inline comments are produced again. Publish validation is unchanged and still decides.

Covers the failures in the issue: missing trailer (storypoker #81, this repo #10), "inline finding mismatch" and missing </details> (agilecharts #859).

Tests: 164 OK (new DeterministicTrailerTest, RetryTest), actionlint and contract checker clean. CHANGELOG ## v0.0.7; the own caller stays on v0.0.6.

Drop any model-written signature and Next step lines and append the canonical
ones in Add review duration, so a missing or malformed trailer no longer fails
Publish. Fixes #11.

@yngwald yngwald left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Readiness: Ready with fixes

Intent System Craft Critical Important Minor
PASS FINDINGS PASS 0 0 1
Details

Findings

R1-F1 - Minor, confidence medium, .github/workflows/auto-reviewer-shared.yaml:1251.
Impact: when the model's Next step is missing or invalid, the fallback is chosen only from the three finding counts in the Summary table. With all counts at zero it always writes Next step: \/verify -> /finalize squash`. Zero findings does not always mean a clean result. A fresh review pass can leave a previous finding not addressed, and the workflow prompt then requires /cr-received PR #. A review can also be Not ready, or UNASSESSED, with zero findings. In those cases the draft gets a merge-forward recommendation that contradicts the report. The same path drops the plan path the prompt asks for in the clean case. Evidence: lines 1248-1254 read only countsfrom the table. TheReadiness:line is in the body and is not consulted. The tests inDeterministicTrailerTestuse PASS verdicts only, so they do not cover this case. Suggestion: in the fallback, parse theReadiness:line and use/verify -> /finalize squashonly for a plainReady. Otherwise use /cr-received PR #, or none` with a reason for UNASSESSED. Add a test with a Not ready body and zero counts.

Target

Target: PR #12 (261112T1021-dev-session -> main)

Boundary: committed range. Base 57aeb53, head d123f4d, Tree 074603c8c0326f95532d94571a9c25b6754bda72. Uncommitted state: none. Excluded paths: none.
Range: 57aeb53...d123f4d - 5 files changed, 104 insertions(+), 22 deletions(-)
Commits:

  • d123f4d Make the Auto Reviewer closing lines deterministic

Coverage: a single self-review from the diff, not independent: the harness had no isolated reviewer, so I read the whole diff and the Add review duration step and its env in the workflow file. I did not read the Publish validator (Validate review body) in full. I checked the grammar match only by comparing the regex at line 1544 with the one at line 1240. Skipped: the rest of the workflow and the tests beyond the diff. Intent sources: the PR title and description (Fixes #11), AGENTS.md, and CHANGELOG ## v0.0.7. There is no plan file or spec.

Axis verdicts

The change does what the PR says, and the only issue is a weak fallback in an edge case.

  • Intent: PASS. The step drops the model's trailing signature and Next step lines and appends the canonical ones. A missing or malformed trailer no longer fails Publish. The CHANGELOG, README and transport doc are updated. The own caller stays on the released tag, and no workflow other than the shared one changes (AGENTS.md rule 3).
  • System: FINDINGS. R1-F1: the fallback Next step is chosen from counts only. Otherwise the inline python3 -I is consistent with the repo rule. Its heredoc indentation is dedented by YAML. NUMBER is provided by the job-level env (line 800), and the tests supply it. The kept-Next-step regex matches the Publish validator's grammar at line 1544.
  • Craft: PASS. The comment is updated. The new DeterministicTrailerTest covers missing, malformed, reversed and CRLF trailers, validator acceptance, the fallback, and a kept agent Next step.

No previous yngwald review exists on this PR (pr.json has no reviews), so this is the first pass and there are no previous findings to report.

Set aside, open questions, and review gaps

Set aside:

  • The code strips all \r characters in the body, not only line endings. This is harmless for Markdown.
  • Only trailing lines are dropped, so a signature quoted higher up in the body stays as written. This is intended.
  • Usage and cost parsing is unchanged by this PR.

Open questions: none.

Review gaps:

  • Tests were not run in this job. CI (validate) is green on the head SHA, as reported by the caller.
  • This was a self-review, not independent.
  • The follow-up model retry is out of scope per the PR description.

Evidence attribution: all of the above was read by me from the diff and the workflow file. The CI result is taken from the caller. No strengths beyond those listed under Craft.

Used: cr-request 0.0.17 · auto-reviewer-shared 0.0.4 · claude-sonnet-5-5 medium · 0 min 57 sec · 210,252 tokens · $0.1903

Pre-validate the draft in the review job with the publish validators and
run the review once more with the errors when it fails. Fixes #11.
@ihororlovskyi ihororlovskyi changed the title Make the Auto Reviewer closing lines deterministic Make Auto Reviewer closing lines deterministic and retry invalid drafts once Nov 12, 2026
@ihororlovskyi
ihororlovskyi merged commit ae5991a into main Nov 12, 2026
2 checks passed
@ihororlovskyi
ihororlovskyi deleted the 261112T1021-dev-session branch November 12, 2026 08:44
@yngwald

yngwald commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Review cycle summary

Critical Important Minor Fixed Rejected Deferred Undetermined
0 0 1 0 + unknown 0 + unknown 0 + unknown 1 + unknown

Review usage

Review Model Effort Duration Tokens Cost
1 claude-sonnet-5-5 medium 57 sec 210,252 $0.19
Total 57 sec 210,252 $0.19
Mean 57.00 sec
Measured 1/1 1/1 1/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto Reviewer fails when the model omits the signature or Next step trailer

2 participants