diff --git a/Makefile b/Makefile index b4d081d..ff737f3 100644 --- a/Makefile +++ b/Makefile @@ -7,10 +7,12 @@ # make release SHA= plan, then apply at once (no prompt), read back # make release-rule-test RULE= SHA= Console-only edit of a scratch rule # rollback: make release SHA= +# make self-review-plan read-only: what the R-self setup would create +# make self-review-setup create the R-self workspace and rule, write its ID into analyze.py TOOL = cd review && uv run --frozen python tools/release.py -.PHONY: release-plan release release-rule-test +.PHONY: release-plan release release-rule-test self-review-plan self-review-setup release-plan: $(TOOL) plan --sha "$(SHA)" $(if $(NO_CONSOLE),--no-console) @@ -20,3 +22,9 @@ release: release-rule-test: $(TOOL) rule-test --rule "$(RULE)" --sha "$(SHA)" + +self-review-plan: + cd review && uv run --frozen python tools/self_review_setup.py plan + +self-review-setup: + cd review && uv run --frozen python tools/self_review_setup.py setup diff --git a/docs/claude-review-design.md b/docs/claude-review-design.md index 63c1587..2a44429 100644 --- a/docs/claude-review-design.md +++ b/docs/claude-review-design.md @@ -86,7 +86,7 @@ Two org rulesets enforce the review on every supported repo: the review ruleset - The review ruleset is the active pilot ruleset. Org-wide enforcement adds the supported repos to its targets and keeps the pilot bypass: for maintainers the review is an advisory signal, not a hard gate. Setting its bypass list to org admins only is follow-up work, done only when someone asks the owner for a gate (P6-5). The baseline ruleset was enabled on a canary repo (infra#192), then widened to all repos (decision P5-1). Each release moves the pin of the review ruleset (see "Rollout"). - Before each target change, bypass change and pin move, an org owner reads the review ruleset back through the [org rules API](https://docs.github.com/en/rest/orgs/rules#get-an-organization-repository-ruleset). The workflow entry must have the `repository_id` of `OperationCode/.github`, the path above and the release `sha`, and no `ref`, so no mutable ref can stand in for the pin. Before each review target change the baseline is read back too, and every review target must get the baseline rules from a ruleset whose only bypass is org admins in pull_request mode. - Ruleset Evaluate mode would report without blocking, but the [org rules API](https://docs.github.com/en/rest/orgs/rules#create-an-organization-repository-ruleset) offers it on GitHub Enterprise only ("evaluate is only available with GitHub Enterprise"). On Team the workflows rule is either enforced or off. So the pilot uses an enforced ruleset with a wide bypass list instead (see "Pilot and enforcement"). -- Never target `.github`. A ruleset that requires a workflow from `.github` and also targets `.github` can deadlock ([example](https://github.com/ContextualWisdomLab/.github/issues/732)). The file also runs as a normal workflow on PRs to `.github` itself. Nothing requires it there, so that result does not gate merges. That normal run uses the file from the default branch of `.github`, not the PR's copy (see "Workflow details"). It also fails auth by design: the production federation rule binds `job_workflow_ref` to `@`, and a normal run has `@refs/heads/main`, so each pass is `failed` and the run shows a red result. That result does not gate a merge. The file has no event filter to hide it, because a ruleset ignores filters and no test shows that a filter would help. +- Never target `.github`. A ruleset that requires a workflow from `.github` and also targets `.github` can deadlock ([example](https://github.com/ContextualWisdomLab/.github/issues/732)). The file also runs as a normal workflow on PRs to `.github` itself. Nothing requires it there, so that result does not gate merges. That normal run uses the file from the default branch of `.github`, not the PR's copy (see "Workflow details"). R1 binds `job_workflow_ref` to `@` and a normal run has `@refs/heads/main`, so R1 refuses its token. Analyze uses the self-review rule R-self for these runs instead (see "Self-review rule" in "Auth and cost"), so a PR to `.github` gets a real model review and a real summary comment. Until the owner creates R-self, its ID is all zeros and each pass is `failed` with "federation not configured", a red result. Either result does not gate a merge. The file has no event filter to hide it, because a ruleset ignores filters and no test shows that a filter would help. ### Supported repos @@ -857,6 +857,36 @@ token lifetime: 10 minutes **Scratch rule test.** `make release-rule-test RULE= SHA=` runs only step 4b on another rule, for example the scratch rule R2 `fdrl_01AfACzupW3U4Bx361ZUbsyR` on the test workspace. It refuses R1. Use it to check the tool after a change to it or to `ant`. History: phases 3 and 4 used a two-rule flow (a new rule R2 per release at the placeholder SHA, its ID in the candidate, two Console edits, then delete R1; V26, P0-15). Every phase 4 release instead edited R1 in place at the merge SHA (cc710fb, cf19216, 96f4966), and P4-9 makes that the spec. The trade-off: no overlap window in which runs at the old SHA still get tokens, and a rollback is one more rule edit instead of a switch back to the old rule. +- **Self-review rule (R-self, issue #36).** PRs to `.github` run `claude-review.yml` as a normal `pull_request_target` workflow from `main`, never from the PR's copy, so the code that holds the token is merged code, the same trust as a push to `main`. R1 refuses their token (`@refs/heads/main`, not `@`), so they use a second rule: + +``` +.github PR --pull_request_target--> claude-review.yml @ main (merged code only) + | | + | OIDC: repository_id=1432342572 (.github), + | job_workflow_ref=...@refs/heads/main, + | event_name=pull_request_target + v v + PR content read via REST only R-self --> workspace claude-review-self-review + (own low spend limit) +issuer: the issuer of R1 +match mode: pattern (not CEL) +subject prefix: repo:OperationCode@10479364/* +claims (exact string match): + repository_id: "1432342572" + event_name: "pull_request_target" + job_workflow_ref: "OperationCode/.github/.github/workflows/claude-review.yml@refs/heads/main" +workspace: claude-review-self-review only (never "all workspaces") +target: the service account of R1 (`claude-review`, role workspace_developer in that workspace) +scope: workspace:developer +token lifetime: 10 minutes +``` + + - **Selection in code.** `analyze.self_review_federation` picks R-self only when the runner variables `GITHUB_REPOSITORY_ID` equals `1432342572` and `GITHUB_EVENT_NAME` equals `pull_request_target`, both exact. A workflow cannot override a `GITHUB_*` default variable, and no PR content takes part. Every other run, including a `workflow_dispatch` of the same file in `.github`, uses R1. The constant `SELF_REVIEW_FEDERATION_RULE_ID` in `analyze.py` holds the ID; the organization and the service account are R1's. An all-zero ID fails each pass with "federation not configured" before any token is minted, never a pass. The Console claims bind the same three facts, so a `workflow_dispatch` run or a run in another repo gets no R-self token even if the code chose wrong. + - **Claims of a direct run (evidence, 2026-11-14).** GitHub's OIDC reference describes `job_workflow_ref` and `job_workflow_sha` under reusable workflows, but a direct run carries them too, so R-self and the local `job_workflow_sha` check use them. (1) V3 run [40259047420](https://github.com/OperationCode/claude-review-p0-source/actions/runs/40259047420), a direct `workflow_dispatch` run in claude-review-p0-source (not ruleset-required, not reusable), printed its decoded claims: `job_workflow_ref` `.../p0-required.yml@refs/heads/main` and `job_workflow_sha` `c2390ae`, equal to `workflow_ref` and `workflow_sha`. The claim set, actor fields removed, is the unit test fixture `review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json`. (2) Run [43631842435](https://github.com/OperationCode/.github/actions/runs/43631842435), a direct `pull_request_target` run on `.github` PR #33 at `c129aa9`, ended each pass with "authentication failed", the server's deny, not "OIDC claim mismatch": at that SHA `claims_match` requires `job_workflow_sha` to equal `job.workflow_sha` before the exchange, so the token of a direct `pull_request_target` run in `.github` has `job_workflow_sha`. Not yet seen: the `job_workflow_ref` value of such a run; the first R-self run shows it (a deny is `match_claim_value_mismatch` or `match_claim_absent` in the Console history). + - **The release tool never edits R-self.** `make release` reads and edits only `PROD_RULE_ID` (R1), and `make release-rule-test` refuses any rule whose `job_workflow_ref` does not end in a 40-hex SHA, which includes R-self. Unit tests check both. A release does not move R-self: it always names `refs/heads/main`. + - **Setup (owner, once).** `make self-review-plan` (read-only) shows what `make self-review-setup` would do. Both use the release tool's ephemeral `ant` login (org:admin, private temp dir, logged out and deleted on every exit path). All reads and checks run before any write. The setup only creates what is missing; it never edits or archives an object, so running it again is safe. (1) Read R1 for the issuer, and check its service account is the one in `analyze.py`. (2) Read workspace `claude-review-self-review` by name, the service account's role in it, and rule `claude-review-self-review` by name. Each list must have a known shape (one item per JSONL line, or a complete `data` page) and each item its required fields (`id` and `name`; for a member `service_account_id` and `workspace_role`), else the tool refuses. A member with a role other than `workspace_developer`, or a rule with that name and any other field, is refused too. (3) Setup only: create the workspace, add the service account as `workspace_developer`, create the rule with the claims above, each only when missing, and read each back (the member must list `workspace_developer`). (4) Write the rule ID into `SELF_REVIEW_FEDERATION_RULE_ID` in the working tree, no commit: atomically, and only when `analyze.py` did not change during the run; after a change the tool refuses, and a re-run writes only the ID. The owner reviews `git diff` and opens a PR. One step stays manual, because `ant` has no command for a workspace spend limit: in the Console, set a low monthly spend limit on `claude-review-self-review` before that PR merges; the tool prints this step. The amount stays out of this public repo. `ant` 2.9.0 lists every other command the tool calls (checked with `--help` only); the response shapes of the workspace and member lists are not yet confirmed by a live run, so run the plan first, and a shape the tool does not expect fails closed. + - **Spend.** R-self has its own workspace and spend limit, so `.github` reviews cannot use up the production budget, and a spent self-review limit stops only `.github` reviews. + - **Residuals.** (1) The run uses `main` HEAD, which can be ahead of the release SHA (merged but unreleased code). Accepted: only maintainers can push to `main`, and a merge needs review. (2) A change to the reviewer itself is reviewed by the code it replaces, so a second human review on `.github` still matters. (3) No ruleset targets `.github`, so the result does not gate a merge. - **Only analyze can mint a token.** Collect and publish have no `id-token` permission. This is how "the GitHub-writing job has no Anthropic credential" holds, because the federation rule cannot tell jobs in the same workflow apart. - **Fixture eval credential.** `eval.yml` runs on `workflow_dispatch`, so the review federation rule rejects its token. Evals use their own federation rules, one per eval environment (P6-4): the evals job uses the environment `eval` for a run on `refs/heads/main` and `eval-test` for any other ref, and sets `EVAL_ENVIRONMENT` to the same value. A select job with no permissions makes the choice with an exact shell test (`[ "$GITHUB_REF" = "refs/heads/main" ]`) and exports it as its one output; the evals job takes that output (`${{ needs.select-environment.outputs.environment }}`) for both. The choice is not an Actions expression: expressions compare strings without case, so `github.ref == 'refs/heads/main'` is also true for a branch named `MAIN` or `Main` (review AC, I1). If the select job fails, the evals job does not run. `evals.py` holds the IDs per environment (separate from the production IDs in `analyze.py`): the same organization; for each environment its own rule and a service account (`eval-test` uses the `eval` service account until the owner chooses another). The runner accepts only `eval` and `eval-test` in `EVAL_ENVIRONMENT`; any other value, or an environment whose rule ID is not set, fails closed before any token is minted ("eval-test federation rule not configured"; no entry is unset now, and a unit test unsets one to cover the path). The `eval` rule is described below; the `eval-test` rule, `eval-test-federation-rule` (created by the owner on 2026-11-06, ID in `evals.py`), is a copy of it with the exact claim `environment: "eval-test"`. It is bound to the test workspace `claude-review-p0-test`, which is not production and has its own spend limit (P2-1). It uses pattern mode, the subject prefix `repo:OperationCode@10479364/*`, the scope `workspace:developer` and the test workspace only. The scratch rules of phase 3 (V26 c) are on the same workspace, so evals and those tests share its spend limit: when it is spent, evals and those tests stop, and production reviews do not. Exact claims: `repository_owner_id`, `repository_id` of `.github`, `event_name: "workflow_dispatch"` and `environment: "eval"` ([OIDC](https://docs.github.com/en/actions/reference/security/oidc)). In a dispatch run `job_workflow_ref` ends in the branch ref (V3), and evals run on any branch, so an exact claim cannot name `eval.yml`. V27 passed on 2026-10-27 (UTC), on the throwaway branch `v27-probe`, which is never merged. P1 (approved `workflow_dispatch` run, job in environment `eval`): token issued (HTTP 200, req_011CgS2vx6h7nwH6wi3dAFNz, 03:30:02Z). N1 (no environment): 401 (req_011CgS2ticG9pbbKTffkfMeK, 03:29:32Z). N2 (approval rejected): the job never started and made no exchange. N3 (`push` event, approved): 401 (req_011CgS2r8BXRmeR8EPtPd9si, 03:28:57Z). N4 (`push` in claude-review-test, no environment): 401 (req_011CgS34fZVr9eZEVx8z8xZw, 03:31:47Z); the job has no environment and ran on `push`, so the missing environment or the `push` event alone explains the 401, and the `repository_id` check is not tested separately. The dispatch run with its own `eval` environment in claude-review-test was not run. U1 (unit tests, `test_workflows.py -k eval`, on a clean tree): pass. An independent check of the run logs agreed with these results and found no secret. Every deny returns the same 401 body, so the owner read the reasons in the Console federation authentication history on 2026-10-27 (all rows: issuer `github-actions`, rule `eval-federation-rule`, service account `claude-review-p0`; times in UTC). 03:27:59Z Failure `match_claim_value_mismatch` (the no-environment job in the push run). 03:28:57Z Failure `match_claim_value_mismatch` (N3, `push` event). 03:29:32Z Failure `match_claim_absent` (N1, no environment). 03:30:03Z Success (P1, approved dispatch; the probe log has 03:30:02Z, so the two clocks differ by one second). 03:30:34Z Failure `match_claim_absent` (the no-environment job in the rejected-approval run; the Console showed no row for N2 itself, the rejected job, as expected). 03:31:47Z Failure `match_claim_value_mismatch` (N4, other repo). The list view does not name the claim and does not show the scope of the P1 token: claim name and P1 scope not shown in the list view. The next sentence is inference from the probe token claims, not read from the Console: each failure code fits the expected cause, an absent `environment` claim or a mismatched `event_name` or `repository` claim, but the list view names no claim. The no-environment job in the push run got `match_claim_value_mismatch`, not `match_claim_absent`; this fits the `event_name` check failing first (also inference). - **Claim boundary.** Each eval rule accepts a dispatch run of any workflow file in `.github` whose job uses its environment, not only `eval.yml`. That only `eval.yml` uses `eval` and `eval-test` is a repo convention; the rules cannot enforce it. Three controls hold the credential separation: the `eval` branch policy (only `main`, which holds reviewed code, can deploy to it), the owner's approval of each `eval-test` run, and a workflow unit test. The unit test reads every workflow file in `.github` and fails if a workflow other than `eval.yml` uses `eval` or `eval-test`, if `eval.yml` lacks the select job (no permissions, no environment, the one output, the fixed exact shell test), if a job in `eval.yml` with an environment does not need that job, uses anything but its output, or sets `EVAL_ENVIRONMENT` to anything else, or if such a job sets no explicit job-level `permissions` or has any `write` permission other than `id-token: write`. Environment names are not case sensitive, so the test treats `EVAL` as `eval`, and in any other workflow an environment that is not a literal string (an expression) counts as a use. Job-level `permissions` is required on every job that uses `eval`, even when the workflow level sets it, and it holds exactly `contents: read` and `id-token: write`. `eval.yml` triggers on `workflow_dispatch` only. The test handles the string forms: `write-all` at either level fails the test, and `read-all` or a missing job-level block also fails it, because the job must state `contents: read` and `id-token: write` itself. The eval job has `contents: read` and `id-token: write` only, so the job that holds the Anthropic credential has no GitHub write permission. @@ -916,12 +946,13 @@ token lifetime: 10 minutes schema/envelope.schema.json tools/dep_repro.py developer tool: collect's dependency stage on two real commits, live registries tools/release.py owner-run release: R1 rule edit, ruleset pin moves, read-backs (never in CI) + tools/self_review_setup.py owner-run, once: R-self workspace and rule, ID into analyze.py (never in CI) tests/ unit/ no network, no API calls (unit/fixtures/eval-smoke: runner smoke case) fixtures/ the live eval cases: bundle.json, expected.json, source/ eval_fixtures.py fixture builder: source files -> bundle.json through collect docs/claude-review-design.md this file - Makefile owner-run release targets (release-plan, release, release-rule-test) + Makefile owner-run targets (release-plan, release, release-rule-test, self-review-plan, self-review-setup) .gitignore release-records/ (local before and after JSON of releases) CODEOWNERS README.md @@ -929,7 +960,7 @@ token lifetime: 10 minutes ## Testing -- **Unit tests (CI on every PR to this repo, no API calls):** prefilter checks, coverage rules, lockfile parsing per supported version, citation check (head and base side, title and body paths, marker-only quotes, inline eligibility, empty quotes, a quote or file that analyze cut to its limit), gate reasons, override checks, envelope checks, sanitizer and escaping, redaction, the summary comment format (one row per cause, the repo setting label, plain paths, the token line, and the unsigned-commit row with the gate result unchanged, P6-1), nonce handling, "every prompt has ``", schema validation of good and bad fixtures, and a workflow file check. It parses every workflow with duplicate keys rejected. For the central workflow it fails on: another trigger than `pull_request_target`, a workflow-level permission, a `concurrency` key, any `continue-on-error`, job names or job scopes that differ from "Architecture", `id-token` outside analyze, a `checks` scope, a job `if:` other than `always()` on the gate job, a step `if:` other than the exact guard on a gate download, a `uses:` that is not a full SHA with a tag comment, a checkout of anything but `OperationCode/.github` at `job.workflow_sha`, a PR ref or `github.sha`, an `actions/cache` step, a `setup-uv` step without `enable-cache: false`, a `${{` in a `run:`, the `secrets` context, a download by name, pattern or run, an upload without `if-no-files-found: error` or without the attempt in its name, a harden-runner step that is not first, not in `audit` mode, without a non-empty `host:port` allowlist, with `disable-telemetry: true` (block mode only) or with `disable-sudo-and-containers`, an `ANTHROPIC_*` variable, a missing or wrong timeout, a publish step that is not the fixed script, and a dependency-review job that does not export the step `outcome`. Two tests are marked `release` and skipped in the default run: `test_release_federation_ids_are_not_placeholders` (the production IDs in `analyze.py` are valid and not placeholders) and `test_harden_runner_egress_lists_are_the_v10_lists`. CI runs them on every PR in a second step of the `unit-tests` job (`pytest tests/unit -m release`; `test_test_workflow_runs_the_release_tests_in_the_unit_tests_job` checks the step), so a candidate that names a placeholder cannot merge. The second test checks that each central job's allowlist equals the list that V10 measured, in `audit` mode (K-14). It also reads every workflow file in this repo, including `test.yml` (no environment, no `id-token`, no secrets), and fails if a workflow other than `eval.yml` uses the `eval` or `eval-test` environment, if `eval.yml` picks its environment or sets `EVAL_ENVIRONMENT` other than from the select job's exact shell test (the test runs that script for `refs/heads/main`, `MAIN`, `Main`, tags, PR refs and others), or if a job that uses an eval environment sets no explicit `permissions` or has any `write` permission other than `id-token: write` (see "Fixture eval credential"). The runner tests check that `evals.py` accepts only `eval` and `eval-test` and fails closed on a missing value, another value, or an environment whose rule is not configured. +- **Unit tests (CI on every PR to this repo, no API calls):** prefilter checks, coverage rules, lockfile parsing per supported version, citation check (head and base side, title and body paths, marker-only quotes, inline eligibility, empty quotes, a quote or file that analyze cut to its limit), gate reasons, override checks, envelope checks, sanitizer and escaping, redaction, the summary comment format (one row per cause, the repo setting label, plain paths, the token line, and the unsigned-commit row with the gate result unchanged, P6-1), nonce handling, "every prompt has ``", schema validation of good and bad fixtures, and a workflow file check. It parses every workflow with duplicate keys rejected. For the central workflow it fails on: another trigger than `pull_request_target`, a workflow-level permission, a `concurrency` key, any `continue-on-error`, job names or job scopes that differ from "Architecture", `id-token` outside analyze, a `checks` scope, a job `if:` other than `always()` on the gate job, a step `if:` other than the exact guard on a gate download, a `uses:` that is not a full SHA with a tag comment, a checkout of anything but `OperationCode/.github` at `job.workflow_sha`, a PR ref or `github.sha`, an `actions/cache` step, a `setup-uv` step without `enable-cache: false`, a `${{` in a `run:`, the `secrets` context, a download by name, pattern or run, an upload without `if-no-files-found: error` or without the attempt in its name, a harden-runner step that is not first, not in `audit` mode, without a non-empty `host:port` allowlist, with `disable-telemetry: true` (block mode only) or with `disable-sudo-and-containers`, an `ANTHROPIC_*` variable, a missing or wrong timeout, a publish step that is not the fixed script, and a dependency-review job that does not export the step `outcome`. Two tests are marked `release` and skipped in the default run: `test_release_federation_ids_are_not_placeholders` (the production IDs in `analyze.py` are valid and not placeholders) and `test_harden_runner_egress_lists_are_the_v10_lists`. CI runs them on every PR in a second step of the `unit-tests` job (`pytest tests/unit -m release`; `test_test_workflow_runs_the_release_tests_in_the_unit_tests_job` checks the step), so a candidate that names a placeholder cannot merge. The second test checks that each central job's allowlist equals the list that V10 measured, in `audit` mode (K-14). It also reads every workflow file in this repo, including `test.yml` (no environment, no `id-token`, no secrets), and fails if a workflow other than `eval.yml` uses the `eval` or `eval-test` environment, if `eval.yml` picks its environment or sets `EVAL_ENVIRONMENT` other than from the select job's exact shell test (the test runs that script for `refs/heads/main`, `MAIN`, `Main`, tags, PR refs and others), or if a job that uses an eval environment sets no explicit `permissions` or has any `write` permission other than `id-token: write` (see "Fixture eval credential"). The runner tests check that `evals.py` accepts only `eval` and `eval-test` and fails closed on a missing value, another value, or an environment whose rule is not configured. The R-self tests check that analyze picks R-self only for `GITHUB_REPOSITORY_ID` `1432342572` with `GITHUB_EVENT_NAME` `pull_request_target` (exact), and R1 for every other repo or event, `.github` with `workflow_dispatch` included; that an all-zero R-self fails every pass with "federation not configured" and mints no token; that the release tool touches only R1 and its rule test refuses a rule bound to a branch ref; that a recorded direct-run claim set passes the local check and reaches R-self; and, with a fake `ant`, that the self-review setup creates only what is missing, refuses before any write on an unknown list shape, a member with another role or a same-name rule with any other field, checks the role after an add, writes the ID only over the placeholder and only when `analyze.py` did not change during the run, and logs out on every exit path. No workflow may run either owner tool. - **Gate matrix (unit):** every row of the reasons table on its own and in combination, plus: missing pass record, duplicate pass record, extra file, wrong `run_attempt`, digest mismatch, envelope for another PR, each `needs..result` value (`failure`, `cancelled`, `skipped`), stale head, override with a wrong SHA, edited override, override by the PR author, override by a user with `read`, override against a non-waivable reason, override older than the push epoch, override older than a reopened event, override posted after a reopened event and then a re-run (accepted), unreadable timeline, several valid overrides, spoofed sticky marker from another author, base ref not the default branch, merge base changed, a newer run at the final check (`superseded`), a grant deleted before the final check, a dependency-review step `outcome` that is `failure` or missing while `needs.dependency-review.result` is `success` (not-waivable fail), and the exit code for every gate result (0 only for `pass`). - **Size limit, lockfiles and commands (unit, P6-6):** the estimate is `ceil(chars x 10 / 17)` and fits at exactly the limit; order (a) follows the tiers, largest first within a tier, and stops as soon as it fits; every patch stays in a diff-only bundle; order (b) follows the tiers, never cuts inside a patch, skips a patch that does not fit and still adds a later smaller one; boundary case: the block lengths admit patch A at the limit and skip a later, smaller patch B, the full estimate is over, the correction evicts A, and B is retried and admitted (mode `partial`, A `not-reviewed`), not `skipped-size`; an evicted patch is never admitted again; a base over the limit and a bundle where no patch fits are `skipped-size` with their reasons; an envelope over 16 MiB in any mode is `skipped-size`; each path tier, including `conftest.py`, `__init__.py`, `jest.config.js` and `docs/conf.py` never in the tests or docs tier, and `requirements-dev.txt` never docs; `model_view` and `model_view_reason` for every row of the table; render's coverage note per view and `SIZE_KEYS_SHOWN`; a lockfile has no `content` at 10 lines and at 3,000 lines, and its `model_view_reason` is `lockfile`, never `long-file`; `mask_hashes` masks poetry and uv `sha256:`, npm, pnpm and bun `sha512-`, `sha1-`, go.sum `h1:=`, Cargo `checksum = ""`, Gemfile `sha256=`, pip `--hash=sha256:` and poetry `content-hash = ""`; it never masks a 40-hex git SHA in a lockfile, a workflow `uses: x/y@<40hex>` or a Dockerfile `FROM x@sha256:` (not lockfile paths); M1 boundaries, never masked: `-r https://x.example/sha256-<44 base64>/evil/path` (left `/`, right `/`), a hash-shaped URL query value (`-r https://x.example/?q=sha384-<64 base64>`, `https://host/?ref=sha1:<40 hex>`, also on an added line with no removed line to pair with, with JSON-escaped slashes `https:\/\/`, with a `'` or `"` inside the URL, and after `file:`, `data:` or a token-start `//`), while the hash after a URL and a space is still masked (uv `{ url = "...", hash = "sha256:" }`), `node_modules/sha1-<28 base64>`, an npm name `sha384-<64 lower-case alnum>`, `sha1-` followed by 29 or more base64 characters (wrong length), a bare 40-hex or 64-hex value under `reference`, `commit` or `rev`, and a bare hex token on a line with no known hash key; masked: `sha1:<40 hex>`, Cargo `checksum = "<64 hex>"`, yarn berry `checksum: 10c0/<128 hex>`; for each lockfile fixture the masked patch has the same line count and the same `+`/`-` positions as the raw patch; M3: a `-` and a `+` line that differ only after the 8th hash character stay raw when they are in the same hunk, in two different hunks, and in two gaps of an anchored patch (K-17); attack cases from review T1 finding 1 show as changed lines in the rendered patch: `requirements.txt` `-r safe.txt` to `-r https://attacker.example/r.txt`, `--trusted-host` added, `--require-hashes` removed, an npm entry whose only change is `hasInstallScript`, and one whose only change is `integrity`; M4 commute: `model_lines(mask_hashes(x)) == [mask_hashes(y) for y in model_lines(x)]` for lockfile lines, including a hash next to a delimiter name and next to an invisible character; M4: a quote copied from a masked line verifies, a quote of the same line from the raw blob verifies, a quote that is only a hash prefix on a wrong line does not, a masked quote in a file that is not a lockfile does not, and the quote Claude saw on a lockfile line with a delimiter name and an invisible character next to a hash verifies (N1); a lockfile patch keeps `patch_redaction` and render refuses it without the mark; credential redaction runs before the mask (a `_authToken=` line in a lockfile shows `[REDACTED:...]`); `removed` rows (name gone, version change is not removed, empty base); raise parsing (`2x`, `3x`, `4x` valid; `5x`, `1x`, `4X`, extra text, two spaces are `malformed`), `edited`, `no-write-permission` (no write, unreadable permission, PR author without write), PR author with write accepted, bot comment skipped, newest valid wins (created_at, then id; a later lower raise wins; a later rejected raise does not cancel; a valid 4x then a valid 2x, with the 2x deleted or edited, gives 4x again; with both invalidated, 1x), comments unreadable gives multiplier 1 and `commands_read: false`; `verdict.evaluate_overrides` skips size-limit comments; the gate reason `partial-review` from `size.mode` and from a `not-reviewed` file alone, waivable, with the 10-path detail; the size record schema (skipped-size needs mode `skipped` and a reason, ok never has mode `skipped`); the summary lines, the row, the callout, the shared ignored-command list and its cap of 3, and the JSON line keys `size`, `input_total` and `estimate_ratio` (null rules included). The eval fixture builder synthesizes the PR comments from an optional `comments` list in a case's `source/pr.json` (each with its commenter's collaborator permission; empty by default), so a raise goes through collect's real checks, and every committed fixture bundle is rebuilt, because the size record and the estimate change. - **First real check of the masked patch:** [OperationCode/.github#33](https://github.com/OperationCode/.github/pull/33), a Dependabot `uv.lock` PR whose review fails today. Run collect only on it (GitHub REST reads as data, no Messages API call, no cost) and check: `uv.lock` has no `content`, its patch is masked with every changed line in place, `dependency_delta` holds its rows, and the estimate and the size mode it gets. @@ -1050,6 +1081,7 @@ The tasks, owner actions and exit criteria for each step are in [docs/build-plan - [ ] Also on `.github` main: dismiss stale approvals on push, and require the `test.yml` check (the job `unit-tests`). - [ ] Anthropic Console: dedicated workspace with a hard monthly spend limit, service account, federation rule as in "Auth and cost" (pattern mode, ID-form subject prefix, scope `workspace:developer`, the review workspace only, the placeholder SHA until phase 3; V26). Done 2026-10-27: workspace `claude-review`, service account `claude-review`, rule R1 `claude-review-prod-r1` created at the placeholder SHA and accepted by the Console (V26 a partial); no read-back is recorded, so this item stays open. - [ ] Anthropic Console, eval part: the eval federation rule (V27) on the test workspace `claude-review-p0-test`, separate from the review workspace (P2-1). GitHub environments on `.github` (P6-4): `eval` with a main-only branch policy and no required reviewer (since 2026-11-04); `eval-test` with the owner as required reviewer and "prevent self-review" off (sole reviewer = run actor), for every other ref. [x] The `eval-test` federation rule `eval-test-federation-rule` (2026-11-06), like the eval rule, with the exact claim `environment: "eval-test"`, and its ID in `evals.py`. [ ] Test it as V27. No API key. +- [ ] Anthropic Console, self-review part (P6-7): `make self-review-plan`, then `make self-review-setup`; set a low monthly spend limit on the workspace `claude-review-self-review` in the Console; review the `analyze.py` diff and open the PR that sets the R-self ID. - [ ] Org Actions setting: default `GITHUB_TOKEN` permissions read-only. - [ ] Org setting: require 2FA for all members. A stolen maintainer account bypasses everything in this design. - [ ] Enable Dependabot alerts with npm malware detection on active repos ([changelog](https://github.blog/changelog/2026-03-17-dependabot-now-detects-malware-in-npm-dependencies/)). @@ -1261,3 +1293,4 @@ P6: phase 6, org-wide enforcement, from 2026-11-04 (owner and coordinator). Wher - P6-4. Eval environment split (owner and coordinator, 2026-11-04). Release evals run on `main`, which holds only reviewed code, so they need no approval; evals of other refs can run unreviewed code with the eval credential, so they need the owner's approval. Step 1 (owner, 2026-11-04): the `eval` environment got a main-only branch policy and lost its required reviewer and "prevent self-review"; admin bypass stays on. Until step 2, branch evals are refused by that policy. Step 2 (this change): `eval.yml` picks `eval` on `refs/heads/main` and `eval-test` on any other ref, with an exact shell test in a select job with no permissions (an Actions expression compares without case, review AC I1), and passes the choice to `evals.py` in `EVAL_ENVIRONMENT`; `evals.py` holds the federation IDs per environment and fails closed on any other value and on an environment whose rule is not set. Step 3 (owner, 2026-11-06): the owner created the federation rule `eval-test-federation-rule` on the test workspace as a copy of the `eval` rule (same issuer, subject prefix, owner, repository and event claims, scope `workspace:developer`, service account, workspace and 600 s lifetime) with the exact claim `environment: "eval-test"`; its ID is set in `evals.py`. Open: test it as V27 (an approved run gets a token, a run in another environment or with no environment gets none). - P6-5. Phase 6 scope (owner, 2026-11-05). (1) Phase 4 exit evidence: 5 real PRs reviewed (was 10; owner and Dependabot PRs count), 2 weekly metric reports, and 7 days with no unexplained false-positive hard block, run failure or maintainer bypass without a reason. (2) `START_HERE` is not a review target: it will be archived after its content moves into `.github` (issue #27 in this repo). The review targets stay the 4 pilot repos. (3) The bypass switch is deferred to follow-up work: the review ruleset keeps the pilot bypass, and the review stays an advisory signal for maintainers, not a hard gate, until someone asks the owner for a gate. The name change to `claude-review` waits for that switch. (4) Code security configurations: the configuration for public repos (enforced on the active public repos, default for new public repos) and the dependency configuration (enforced, default for new private and internal repos) are live and read back; one planned change to the dependency configuration is skipped by owner decision. Details are kept in the owner's records. (5) The phase 6 final acceptance test checks this shape (build plan, phase 6 exit); the gate-shape checks (org-admin-only bypass, the renamed ruleset, a live check that a Maintain user cannot bypass) move to the follow-up. - P6-6. Review PRs larger than the size gate (issue #29 and the owner's lockfile comment on it; owner and coordinator, 2026-11-14). PR #28 in this repo merged with no model review: about 239k estimated tokens, 177k of them unchanged full-file content, while its patches were about 63k. On operationcode-pybot#466 about 75% of the input was `poetry.lock` hashes, sent as `content` next to its patch and the dependency delta. (1) Estimate: chars / 1.7, labelled real tokens; base limit 200,000 real tokens (was 150k at chars / 4). (2) Diff-only fallback: drop `content` in a fixed order (added, docs, tests and fixtures, then largest first); every patch stays; files are classified by path, and files that run on import or at build or test time stay out of the tests and docs tiers. (3) Partial fallback: whole patches in priority order (sensitive, dependency files, source, tests, docs); a patch that does not fit is skipped and filling goes on; new waivable reason "partial model review" that names the files not seen. Never a silent pass. When no patch fits, the bundle is `skipped-size` as before. The 16 MiB envelope rule does not change. (4) `/oc-review size-limit Nx`, N in 2, 3, 4; read by collect; bound to the PR, not the push epoch, so it survives force pushes while its author keeps write or admin and the comment is unedited; the newest valid raise wins; the PR author may raise only with write; applied by the same re-run call as an override; the workflow does not change. (5) Rejected raises and overrides share one list of ignored commands in the summary. (6) Lockfiles, `requirements*.txt` included, never send `content`, and always send their patch with hash values cut to the first 8 characters and `~` (M1 to M4: exact-length prefixed hashes and keyed hex only, lockfile paths only, never a bare git commit ID). This adds the first path rule to the patch itself: until now (K-17) the patch read only line text, never the path. It is safe because the path selects only the mask; the diff, the hunks and every changed line are the same for every path, and M3 unmasks any pair the mask would make equal; `dependency_delta` stays next to the patch, and `deps` adds `removed` rows as context. Why a masked patch and not a summary in place of the patch: a draft replaced a proven lockfile patch with a pointer to the delta, but review T1 finding 1 showed that a summary misses every change its model does not cover (`-r `, `--trusted-host`, a removed `--require-hashes`, `integrity`-only and `hasInstallScript`-only changes), and the proof needed raw entries and an allowlist per format. The mask hides no changed line, works on any format by path, and is small. Most of the saving comes from no `content`; the mask cuts the patch by about a half (Python) or a third (JS). The owner's comment on issue #29 lists the outside tools that were checked and rejected for this: the GitHub dependency-graph compare API (no `uv.lock`, no direct or transitive field), osv-scanner and osv-scalibr (no diff mode, a pinned binary and a second parser that can disagree with the gate), syft with an SBOM diff, Trivy, deps.dev, ecosyste.ms, and the small lockfile-diff tools (no checksummed releases). Residual: a package-manager migration can still be over the base limit when masked (about 242k tokens on a pnpm 12 migration), so it gets a partial review or needs a raise; a hash form the regex does not match costs tokens only. A raise ends only when every valid raise comment is invalidated: deleting the newest can bring back an older one. (7) The publisher's JSON line holds the estimate next to the real input per pass. Window arithmetic at 4x: 800,000 + 10,000 prompts + 32,000 `max_tokens` = 842,000, under 1M. A long-context eval at about 800k runs before the release, with owner approval (it costs money). Not in scope: chunked reviews, exact token counting through the API, the Batch API, another model, per-pass bundles. +- P6-7. Self-review rule R-self (issue #36; owner and coordinator, 2026-11-14). PRs to `.github` got no model review since 2026-10-31: the file runs there from `main`, so R1 (bound to the release SHA) refused every token (#33). A second rule, R-self, binds `repository_id` of `.github`, `event_name: pull_request_target` and `job_workflow_ref` at `refs/heads/main`, in its own workspace with a low spend limit. Analyze picks it from the runner variables `GITHUB_REPOSITORY_ID` and `GITHUB_EVENT_NAME` only. The release tool never edits it. Owner request: the Console steps are scripted (`make self-review-plan`, `make self-review-setup`), reusing the release tool's ephemeral login; the spend limit stays manual because `ant` has no command for it. The ID is all zeros until the owner runs the setup and merges the PR that sets it; until then `.github` runs fail as before, with "federation not configured". See "Self-review rule" in "Auth and cost". diff --git a/review/src/oc_review/analyze.py b/review/src/oc_review/analyze.py index 587fd40..1220ffb 100644 --- a/review/src/oc_review/analyze.py +++ b/review/src/oc_review/analyze.py @@ -82,6 +82,15 @@ ANTHROPIC_ORGANIZATION_ID = "2bc920f9-977b-43b2-870a-f840e4ce809c" ANTHROPIC_SERVICE_ACCOUNT_ID = "svac_01PiMgfb6qJgNpLZUV8hprCC" +# Self-review rule R-self (spec "Auth and cost", "Self-review rule"): only for a +# `pull_request_target` run in OperationCode/.github, which runs this file from +# `main`, so R1 (bound to the release SHA) refuses its token. Own workspace, +# same service account. The release tool never edits it. All zeros until the +# owner creates it: such a run fails with "federation not configured". +SELF_REVIEW_FEDERATION_RULE_ID = "fdrl_000000000000000000000000" +SELF_REVIEW_REPOSITORY_ID = "1432342572" # OperationCode/.github +SELF_REVIEW_EVENT_NAME = "pull_request_target" + FEDERATION_RULE_ID_RE = re.compile(r"^fdrl_[A-Za-z0-9]{24}$") ORGANIZATION_ID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$") SERVICE_ACCOUNT_ID_RE = re.compile(r"^svac_[A-Za-z0-9]{24}$") @@ -453,6 +462,15 @@ def federation_ids() -> FederationIds: return FederationIds(ANTHROPIC_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID) +def self_review_federation(env: Mapping[str, str]) -> FederationIds | None: + """R-self for a `pull_request_target` run in `.github`, else None (R1). + Only runner-set variables decide; a workflow cannot override `GITHUB_*`.""" + if (env.get("GITHUB_REPOSITORY_ID") == SELF_REVIEW_REPOSITORY_ID + and env.get("GITHUB_EVENT_NAME") == SELF_REVIEW_EVENT_NAME): + return FederationIds(SELF_REVIEW_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID) + return None + + def _all_zero(s: str) -> bool: return set(s) <= {"0", "-"} @@ -796,7 +814,8 @@ def analyze( pass problem; every path gives three results. `federation` is the Anthropic federation IDs. None (the review path) means - the production constants of this module. The eval runner passes its own. + the production constants of this module (R1). `main` passes R-self for a + `.github` self-review run; the eval runner passes its own. `github` is None without `GITHUB_TOKEN`, and `runner_temp` is None without `RUNNER_TEMP`. Only a bundle with status `ok` needs them; without them its @@ -1016,6 +1035,7 @@ def main( client_factory=client_factory, clock=clock, deadline_seconds=deadline_seconds, + federation=self_review_federation(env), ) except Exception as exc: # noqa: BLE001 - fail closed with three records print(f"analyze: internal error: {type(exc).__name__}", file=sys.stderr) diff --git a/review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json b/review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json new file mode 100644 index 0000000..bfc1875 --- /dev/null +++ b/review/tests/unit/fixtures/oidc/v3-direct-dispatch-claims.json @@ -0,0 +1,34 @@ +{ + "_source": "decoded OIDC claims printed (no token) by V3 run 40259047420, OperationCode/claude-review-p0-source, a direct workflow_dispatch run (not ruleset-required, not reusable); actor fields removed", + "claims": { + "aud": "https://api.anthropic.com", + "base_ref": "", + "check_run_id": "121367205481", + "event_name": "workflow_dispatch", + "exp": 1792981857, + "head_ref": "", + "iat": 1792981557, + "iss": "https://token.actions.githubusercontent.com", + "job_workflow_ref": "OperationCode/claude-review-p0-source/.github/workflows/p0-required.yml@refs/heads/main", + "job_workflow_sha": "c2390aef5f2f9ae9501ad1365c549150a2488e4a", + "jti": "92171d42-e134-4465-b831-7df9240a43e1", + "nbf": 1792981257, + "ref": "refs/heads/main", + "ref_protected": "false", + "ref_type": "branch", + "repository": "OperationCode/claude-review-p0-source", + "repository_id": "1433591286", + "repository_owner": "OperationCode", + "repository_owner_id": "10479364", + "repository_visibility": "public", + "run_attempt": "1", + "run_id": "40259047420", + "run_number": "1", + "runner_environment": "github-hosted", + "sha": "c2390aef5f2f9ae9501ad1365c549150a2488e4a", + "sub": "repo:OperationCode@10479364/claude-review-p0-source@1433591286:ref:refs/heads/main", + "workflow": "p0-required", + "workflow_ref": "OperationCode/claude-review-p0-source/.github/workflows/p0-required.yml@refs/heads/main", + "workflow_sha": "c2390aef5f2f9ae9501ad1365c549150a2488e4a" + } +} diff --git a/review/tests/unit/test_analyze.py b/review/tests/unit/test_analyze.py index 15a64ff..dc30261 100644 --- a/review/tests/unit/test_analyze.py +++ b/review/tests/unit/test_analyze.py @@ -190,6 +190,7 @@ class Run: oidc: FakeOidc out_dir: Path token_files: list + factory_kwargs: list = field(default_factory=list) def run_main(tmp_path, *, client=None, bundle=None, digest=None, github=None, oidc=None, env=None, @@ -219,11 +220,12 @@ def run_main(tmp_path, *, client=None, bundle=None, digest=None, github=None, oi client = client or FakeClient() github = github or FakeGitHub() oidc = oidc or FakeOidc() - token_files = [] + token_files, factory_kwargs = [], [] - def factory(token_file, *, deadline, clock): + def factory(token_file, *, deadline, clock, **kw): mode = stat.S_IMODE(token_file.stat().st_mode) token_files.append((token_file, mode, token_file.read_text())) + factory_kwargs.append(kw) if factory_exc is not None: raise factory_exc if real_factory is not None: @@ -251,7 +253,7 @@ def factory(token_file, *, deadline, clock): for line in output.read_text().splitlines(): k, _, v = line.partition("=") outputs[k] = v - return Run(code, records, raw, outputs, client, github, oidc, out_dir, token_files) + return Run(code, records, raw, outputs, client, github, oidc, out_dir, token_files, factory_kwargs) def statuses(run): @@ -1429,6 +1431,95 @@ def factory(token_file, *, deadline, clock): # no `federation` kwarg: the defau assert calls == [1] and {r.status for r in results.values()} == {"ok"} +# --- the self-review rule R-self (PRs to .github) ------------------------------------ + +SELF_ENV = {"GITHUB_REPOSITORY_ID": analyze.SELF_REVIEW_REPOSITORY_ID, "GITHUB_EVENT_NAME": "pull_request_target"} +SELF_RULE = "fdrl_01SelfReviewRuleIdAbcdef" + + +@pytest.mark.parametrize("env, chosen", [ + (SELF_ENV, True), + ({**SELF_ENV, "GITHUB_EVENT_NAME": "workflow_dispatch"}, False), + ({**SELF_ENV, "GITHUB_EVENT_NAME": "push"}, False), + ({**SELF_ENV, "GITHUB_EVENT_NAME": "pull_request"}, False), + ({**SELF_ENV, "GITHUB_EVENT_NAME": "PULL_REQUEST_TARGET"}, False), + ({**SELF_ENV, "GITHUB_REPOSITORY_ID": "1432457406"}, False), # claude-review-test + ({**SELF_ENV, "GITHUB_REPOSITORY_ID": " 1432342572"}, False), + ({"GITHUB_EVENT_NAME": "pull_request_target"}, False), + ({"GITHUB_REPOSITORY_ID": analyze.SELF_REVIEW_REPOSITORY_ID}, False), + ({}, False), +]) +def test_r_self_only_for_pull_request_target_in_dotgithub(env, chosen): + ids = analyze.self_review_federation(env) + if chosen: + assert ids == analyze.FederationIds(analyze.SELF_REVIEW_FEDERATION_RULE_ID, analyze.ANTHROPIC_ORGANIZATION_ID, + analyze.ANTHROPIC_SERVICE_ACCOUNT_ID) + else: + assert ids is None + + +def test_r_self_is_its_own_rule_and_r1_is_the_release_rule(): + assert analyze.SELF_REVIEW_FEDERATION_RULE_ID != analyze.ANTHROPIC_FEDERATION_RULE_ID + assert analyze.FEDERATION_RULE_ID_RE.match(analyze.SELF_REVIEW_FEDERATION_RULE_ID) + + +def test_placeholder_r_self_fails_every_pass_and_never_mints(tmp_path, configured, monkeypatch): + monkeypatch.setattr(analyze, "SELF_REVIEW_FEDERATION_RULE_ID", "fdrl_000000000000000000000000") + run = run_main(tmp_path, env=SELF_ENV) + assert_three_records(run) + assert set(statuses(run).values()) == {"failed"} + assert set(reasons(run).values()) == {analyze.R_FEDERATION_NOT_CONFIGURED} + assert run.client.calls == {} and run.oidc.calls == [] and run.token_files == [] + + +def test_configured_r_self_reaches_the_client(tmp_path, configured, monkeypatch): + monkeypatch.setattr(analyze, "SELF_REVIEW_FEDERATION_RULE_ID", SELF_RULE) + run = run_main(tmp_path, env=SELF_ENV) + assert set(statuses(run).values()) == {"ok"} + assert run.factory_kwargs == [{"federation": analyze.FederationIds( + SELF_RULE, CONFIGURED_IDS["ANTHROPIC_ORGANIZATION_ID"], CONFIGURED_IDS["ANTHROPIC_SERVICE_ACCOUNT_ID"])}] + + +V3_DIRECT = json.loads((Path(__file__).parent / "fixtures" / "oidc" / "v3-direct-dispatch-claims.json").read_text()) + + +def dotgithub_direct_claims(): + """The recorded direct-run claim set (V3 run 40259047420), moved to a + `pull_request_target` run of claude-review.yml in .github on main.""" + ref = "OperationCode/.github/.github/workflows/claude-review.yml@refs/heads/main" + claims = {**V3_DIRECT["claims"], "event_name": "pull_request_target", "repository": "OperationCode/.github", + "repository_id": analyze.SELF_REVIEW_REPOSITORY_ID, "job_workflow_ref": ref, "workflow_ref": ref, + "job_workflow_sha": JOB_SHA, "workflow_sha": JOB_SHA, + "sub": f"repo:OperationCode@10479364/.github@{analyze.SELF_REVIEW_REPOSITORY_ID}:pull_request"} + return claims + + +def test_a_recorded_direct_run_token_has_the_job_workflow_claims(): + """A direct run (not reusable, not ruleset-required) carries job_workflow_ref + in branch form and job_workflow_sha, equal to workflow_ref / workflow_sha. + So the local check and R-self's job_workflow_ref claim hold for it.""" + c = V3_DIRECT["claims"] + assert c["job_workflow_ref"] == c["workflow_ref"] and c["job_workflow_ref"].endswith("@refs/heads/main") + assert c["job_workflow_sha"] == c["workflow_sha"] + assert analyze.claims_match(c, c["workflow_sha"]) + + +def test_configured_r_self_accepts_a_direct_run_claim_set(tmp_path, configured, monkeypatch): + monkeypatch.setattr(analyze, "SELF_REVIEW_FEDERATION_RULE_ID", SELF_RULE) + run = run_main(tmp_path, env=SELF_ENV, oidc=FakeOidc(make_jwt(**dotgithub_direct_claims()))) + assert set(statuses(run).values()) == {"ok"} + assert run.factory_kwargs[0]["federation"].rule_id == SELF_RULE + + +@pytest.mark.parametrize("env", [{}, {**SELF_ENV, "GITHUB_EVENT_NAME": "workflow_dispatch"}, + {**SELF_ENV, "GITHUB_REPOSITORY_ID": "107068150"}]) +def test_every_other_run_keeps_r1(tmp_path, configured, monkeypatch, env): + monkeypatch.setattr(analyze, "SELF_REVIEW_FEDERATION_RULE_ID", SELF_RULE) + run = run_main(tmp_path, env=env) + assert set(statuses(run).values()) == {"ok"} + assert run.factory_kwargs == [{}] # no `federation`: the module constants (R1) + + def test_default_client_factory_uses_the_federation_parameter(tmp_path, configured): token_file = tmp_path / "tok" token_file.write_text(GOOD_JWT) diff --git a/review/tests/unit/test_release_tool.py b/review/tests/unit/test_release_tool.py index 182a1da..27bada7 100644 --- a/review/tests/unit/test_release_tool.py +++ b/review/tests/unit/test_release_tool.py @@ -566,6 +566,24 @@ def test_rule_test_refuses_the_production_rule(): assert fake.calls == [] +def test_release_targets_r1_only_never_r_self(): + from oc_review import analyze + assert R1 == analyze.ANTHROPIC_FEDERATION_RULE_ID + assert R1 != analyze.SELF_REVIEW_FEDERATION_RULE_ID + fake = Fake() + assert release.main(["release", "--sha", SHA_NEW], run=fake) == 0 + touched = {c[c.index("--federation-rule-id") + 1] for c, _ in fake.calls if "--federation-rule-id" in c} + assert touched == {R1} + + +def test_rule_test_refuses_a_rule_bound_to_a_branch_ref_such_as_r_self(): + fake = Fake() + r_self = "fdrl_01SelfReviewRuleIdAbcdef" + fake.rules[r_self] = rule(r_self, sha="refs/heads/main") + assert release.main(["rule-test", "--rule", r_self, "--sha", SHA_NEW], run=fake) == 2 + assert fake.writes() == [] and fake.rules[r_self] == rule(r_self, sha="refs/heads/main") + + # ---------------------------------------------------------------- never in CI @@ -573,6 +591,7 @@ def test_no_workflow_runs_the_release_tool(): for path in (ROOT / ".github" / "workflows").glob("*.y*ml"): text = path.read_text(encoding="utf-8") assert "release.py" not in text and "make release" not in text and "ant " not in text, path.name + assert "self_review_setup" not in text and "make self-review" not in text, path.name def test_default_runner_is_guarded_in_tests(): diff --git a/review/tests/unit/test_self_review_setup.py b/review/tests/unit/test_self_review_setup.py new file mode 100644 index 0000000..5b8c90e --- /dev/null +++ b/review/tests/unit/test_self_review_setup.py @@ -0,0 +1,338 @@ +"""Owner-run R-self setup (review/tools/self_review_setup.py). No network: +every ant call goes to a fake runner.""" + +import copy +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(ROOT / "review" / "tools")) +import self_review_setup as setup # noqa: E402 + +release = setup.release +R1 = release.PROD_RULE_ID +SA = "svac_01PiMgfb6qJgNpLZUV8hprCC" +ISSUER = "fdis_x" +WID = "wrkspc_self" +NEW_RULE = "fdrl_01SelfReviewRuleIdAbcdef" +ANALYZE_TEXT = (f'ANTHROPIC_FEDERATION_RULE_ID = "{R1}"\n' + f'ANTHROPIC_SERVICE_ACCOUNT_ID = "{SA}"\n' + f'SELF_REVIEW_FEDERATION_RULE_ID = "{setup.PLACEHOLDER}"\n') +HELP = "retrieve update list create add" +MEMBER = {"service_account_id": SA, "workspace_role": "workspace_developer"} + + +def r_self(rule_id=NEW_RULE, **over): + rule = {"id": rule_id, "name": setup.RULE_NAME, "issuer_id": ISSUER, "archived_at": None, + "match": {"audience": None, "condition": None, **copy.deepcopy(setup.EXPECTED_MATCH)}, + "oauth_scope": "workspace:developer", "applies_to_all_workspaces": False, + "target": {"service_account_id": SA, "type": "service_account"}, + "token_lifetime_seconds": 600, "workspace_id": WID, "workspace_ids": [WID]} + rule.update(over) + return rule + + +class FakeAnt: + """ant stand-in. Records each call; keeps workspace, member and rule state.""" + + def __init__(self, *, workspaces=None, members=None, rules=None, help_text=HELP, raw=None, add_role=None, + on_login=None): + self.calls: list[list[str]] = [] + self.raw = raw or {} # resource -> list output text, to test unknown shapes + self.add_role, self.on_login = add_role, on_login + self.workspaces = workspaces if workspaces is not None else [] + self.members = members if members is not None else {} + self.rules = {R1: {"id": R1, "name": "claude-review-prod-r1", "issuer_id": ISSUER, + "target": {"service_account_id": SA, "type": "service_account"}}} + self.rules.update(rules or {}) + self.help_text = help_text + + @staticmethod + def out(obj=None, text=None): + return subprocess.CompletedProcess([], 0, text if text is not None else json.dumps(obj), "") + + def __call__(self, cmd, **kw): + assert cmd[0] == "ant" + assert not any(k.startswith("ANTHROPIC_") and k != "ANTHROPIC_CONFIG_DIR" for k in kw["env"]) + args = cmd[1:] + self.calls.append(args) + if args[-1] == "--help": + return self.out(text=self.help_text) + if args[0] == "auth": + if args[1] == "login" and self.on_login: + self.on_login() + return self.out(text="") + res, verb = args[0], args[1] + + def opt(name): + return args[args.index(name) + 1] + + def jsonl(items): + return self.out(text="".join(json.dumps(i) + "\n" for i in items)) + + if verb == "list" and res in self.raw: + return self.out(text=self.raw[res]) + if res == setup.WORKSPACES: + if verb == "list": + return jsonl(self.workspaces) + if verb == "create": + ws = {"id": WID, "name": opt("--name"), "archived_at": None} + self.workspaces.append(ws) + return self.out(ws) + if verb == "retrieve": + return self.out(next(w for w in self.workspaces if w["id"] == opt("--workspace-id"))) + if res == setup.MEMBERS: + wid = opt("--workspace-id") + if verb == "list": + return jsonl(self.members.get(wid, [])) + if verb == "add": + self.members.setdefault(wid, []).append({"service_account_id": opt("--service-account-id"), + "workspace_role": self.add_role or opt("--workspace-role")}) + return self.out({}) + if res == release.Console.RULES: + if verb == "list": + return jsonl(list(self.rules.values())) + if verb == "retrieve": + return self.out(self.rules[opt("--federation-rule-id")]) + if verb == "create": + rule = r_self(match=json.loads(opt("--match")), issuer_id=opt("--issuer-id"), + target=json.loads(opt("--target")), oauth_scope=opt("--oauth-scope"), + workspace_id=opt("--workspace-id"), workspace_ids=[opt("--workspace-id")], + token_lifetime_seconds=int(opt("--token-lifetime-seconds")), name=opt("--name")) + self.rules[NEW_RULE] = rule + return self.out(rule) + raise AssertionError(args) + + def writes(self): + return [c for c in self.calls if len(c) > 1 and c[1] in ("create", "add", "update", "archive")] + + +@pytest.fixture(autouse=True) +def _hermetic(tmp_path, monkeypatch): + """No real subprocess; records, temp dirs and analyze.py go under tmp_path.""" + def refuse(*a, **kw): + raise AssertionError(f"real subprocess call in a unit test: {a!r}") + monkeypatch.setattr(release.subprocess, "run", refuse) + monkeypatch.setattr(release.subprocess, "Popen", refuse) + monkeypatch.setattr(release, "RECORDS", tmp_path / "records") + monkeypatch.setattr(release.tempfile, "tempdir", str(tmp_path / "tmp")) + (tmp_path / "tmp").mkdir() + analyze = tmp_path / "analyze.py" + analyze.write_text(ANALYZE_TEXT) + monkeypatch.setattr(setup, "ANALYZE_FILE", analyze) + monkeypatch.setenv("ANTHROPIC_API_KEY", "must-not-reach-ant") + + +def analyze_text(): + return setup.ANALYZE_FILE.read_text() + + +def logged_out(fake): + return fake.calls[-1] == ["auth", "logout", "--all"] + + +def test_setup_from_nothing_creates_workspace_member_and_rule_and_writes_the_id(capsys): + fake = FakeAnt() + assert setup.main(["setup"], run=fake) == 0 + assert [c[:2] for c in fake.writes()] == [[setup.WORKSPACES, "create"], [setup.MEMBERS, "add"], + [release.Console.RULES, "create"]] + assert fake.members[WID] == [{"service_account_id": SA, "workspace_role": "workspace_developer"}] + assert setup.rule_problems(fake.rules[NEW_RULE], workspace_id=WID, issuer_id=ISSUER, service_account_id=SA) == [] + assert f'SELF_REVIEW_FEDERATION_RULE_ID = "{NEW_RULE}"' in analyze_text() + assert analyze_text().count(R1) == 1 # R1 untouched + out = capsys.readouterr().out + assert "MANUAL STEP" in out and "spend limit" in out + assert logged_out(fake) and not list((Path(release.tempfile.tempdir)).iterdir()) + + +def test_setup_creates_the_rule_with_the_exact_claims(): + fake = FakeAnt() + setup.main(["setup"], run=fake) + create = next(c for c in fake.writes() if c[0] == release.Console.RULES) + assert json.loads(create[create.index("--match") + 1]) == { + "subject_prefix": "repo:OperationCode@10479364/*", + "claims": {"repository_id": "1432342572", "event_name": "pull_request_target", + "job_workflow_ref": "OperationCode/.github/.github/workflows/claude-review.yml@refs/heads/main"}} + assert create[create.index("--oauth-scope") + 1] == "workspace:developer" + assert create[create.index("--workspace-id") + 1] == WID + assert "--applies-to-all-workspaces" not in create + + +def test_setup_again_is_idempotent_and_writes_nothing(): + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], + members={WID: [MEMBER]}, rules={NEW_RULE: r_self()}) + assert setup.main(["setup"], run=fake) == 0 + assert fake.writes() == [] + assert f'"{NEW_RULE}"' in analyze_text() + before = analyze_text() + assert setup.main(["setup"], run=fake) == 0 and analyze_text() == before + + +def test_plan_is_read_only(capsys): + fake = FakeAnt() + assert setup.main(["plan"], run=fake) == 0 + assert fake.writes() == [] and analyze_text() == ANALYZE_TEXT and logged_out(fake) + out = capsys.readouterr().out + assert "missing, setup creates it" in out and "Nothing was changed" in out + + +def test_plan_with_everything_in_place_says_what_analyze_would_get(capsys): + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], + members={WID: [MEMBER]}, rules={NEW_RULE: r_self()}) + assert setup.main(["plan"], run=fake) == 0 + assert fake.writes() == [] and analyze_text() == ANALYZE_TEXT + assert f"would be set to {NEW_RULE}" in capsys.readouterr().out + + +@pytest.mark.parametrize("over", [ + {"match": {"subject_prefix": "repo:OperationCode@10479364/*", + "claims": {"event_name": "pull_request_target", "repository_id": "1432342572"}}}, + {"oauth_scope": "org:admin"}, + {"applies_to_all_workspaces": True}, + {"workspace_ids": [WID, "wrkspc_other"]}, + {"target": {"service_account_id": "svac_other", "type": "service_account"}}, + {"token_lifetime_seconds": 3600}, +]) +def test_an_existing_rule_that_differs_is_refused_never_edited(over, capsys): + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], + members={WID: [MEMBER]}, rules={NEW_RULE: r_self(**over)}) + assert setup.main(["setup"], run=fake) == 2 + assert fake.writes() == [] and analyze_text() == ANALYZE_TEXT and logged_out(fake) + assert "never edits a rule" in capsys.readouterr().err + + +def test_two_workspaces_with_the_name_are_refused(): + ws = [{"id": "wrkspc_a", "name": setup.WORKSPACE_NAME}, {"id": "wrkspc_b", "name": setup.WORKSPACE_NAME}] + fake = FakeAnt(workspaces=ws) + assert setup.main(["setup"], run=fake) == 2 + assert fake.writes() == [] and logged_out(fake) + + +def test_an_archived_workspace_with_the_name_does_not_count(): + fake = FakeAnt(workspaces=[{"id": "wrkspc_old", "name": setup.WORKSPACE_NAME, "archived_at": "t"}]) + assert setup.main(["setup"], run=fake) == 0 + assert [c[:2] for c in fake.writes()][0] == [setup.WORKSPACES, "create"] + + +def test_r1_with_another_service_account_is_refused_before_any_write(): + fake = FakeAnt() + fake.rules[R1]["target"]["service_account_id"] = "svac_other" + assert setup.main(["setup"], run=fake) == 2 + assert fake.writes() == [] and logged_out(fake) + + +def test_analyze_with_another_real_r_self_is_not_overwritten(capsys): + setup.ANALYZE_FILE.write_text(ANALYZE_TEXT.replace(setup.PLACEHOLDER, "fdrl_01OtherRuleAbcdefghijklm")) + before = analyze_text() + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], + members={WID: [MEMBER]}, rules={NEW_RULE: r_self()}) + assert setup.main(["setup"], run=fake) == 2 + assert analyze_text() == before + assert "edit analyze.py by hand" in capsys.readouterr().err + + +def test_analyze_without_the_constant_refuses_before_login(): + setup.ANALYZE_FILE.write_text(ANALYZE_TEXT.replace("SELF_REVIEW_FEDERATION_RULE_ID", "OTHER")) + fake = FakeAnt() + assert setup.main(["setup"], run=fake) == 2 + assert fake.calls == [] + + +def test_an_ant_without_a_needed_command_refuses_before_login(): + fake = FakeAnt(help_text="retrieve update list") + assert setup.main(["setup"], run=fake) == 2 + assert all(c[-1] == "--help" for c in fake.calls) + + +def test_interrupt_logs_out_and_says_how_to_recover(monkeypatch, capsys): + fake = FakeAnt() + + def interrupt(*a, **k): + raise KeyboardInterrupt + monkeypatch.setattr(setup.Setup, "write", interrupt) + assert setup.main(["setup"], run=fake) == 130 + assert logged_out(fake) + assert "make self-review-plan" in capsys.readouterr().err + + +def test_parse_items_reads_jsonl_items_and_pages(): + text = '{"id": "a", "name": "n"}\n\n{"data": [{"id": "b", "name": "n"}], "has_more": false}\n' + assert [i["id"] for i in setup.parse_items(text, "workspace", "x")] == ["a", "b"] + + +@pytest.mark.parametrize("text", [ + "not json\n", + '{"items": [{"id": "w", "name": "claude-review-self-review"}]}\n', # unknown envelope + '{"data": [{"id": "w", "name": "n"}], "has_more": true}\n', # incomplete + '{"data": {"id": "w"}}\n', + '{"data": [], "cursor": "x"}\n', + '{"id": "w"}\n', # no name + '{"id": "", "name": "n"}\n', + '["w"]\n', +]) +def test_parse_items_refuses_an_unknown_shape(text): + with pytest.raises(release.ReleaseError): + setup.parse_items(text, "workspace", "x") + + +ENVELOPE = '{"items": [{"id": "existing", "name": "claude-review-self-review"}]}\n' + + +@pytest.mark.parametrize("raw", [ + {setup.WORKSPACES: ENVELOPE}, + {release.Console.RULES: ENVELOPE}, + {setup.MEMBERS: '{"items": [{"service_account_id": "%s"}]}\n' % SA}, + {setup.MEMBERS: '{"service_account_id": "%s"}\n' % SA}, # no role +]) +def test_an_unknown_list_shape_refuses_before_any_create_or_add(raw, capsys): + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], raw=raw) + assert setup.main(["setup"], run=fake) == 2 + assert fake.writes() == [] and analyze_text() == ANALYZE_TEXT and logged_out(fake) + assert "REFUSED" in capsys.readouterr().err + + +@pytest.mark.parametrize("role", ["workspace_user", "workspace_admin", "workspace_restricted_developer"]) +def test_an_existing_member_with_another_role_is_refused_before_any_write(role): + fake = FakeAnt(workspaces=[{"id": WID, "name": setup.WORKSPACE_NAME}], + members={WID: [{"service_account_id": SA, "workspace_role": role}]}) + assert setup.main(["setup"], run=fake) == 2 + assert fake.writes() == [] and analyze_text() == ANALYZE_TEXT + + +def test_a_member_added_with_another_role_fails_the_read_back(): + fake = FakeAnt(add_role="workspace_user") + assert setup.main(["setup"], run=fake) == 2 + assert [c[:2] for c in fake.writes()] == [[setup.WORKSPACES, "create"], [setup.MEMBERS, "add"]] + assert analyze_text() == ANALYZE_TEXT and logged_out(fake) + + +def test_an_edit_to_analyze_during_login_is_kept_and_the_id_not_written(capsys): + edited = ANALYZE_TEXT + "# local edit\n" + fake = FakeAnt(on_login=lambda: setup.ANALYZE_FILE.write_text(edited)) + assert setup.main(["setup"], run=fake) == 2 + assert analyze_text() == edited + assert "changed during the run" in capsys.readouterr().err + assert [p.name for p in setup.ANALYZE_FILE.parent.iterdir() if p.name.startswith(".analyze")] == [] + fake.on_login = None + assert setup.main(["setup"], run=fake) == 0 # a re-run finds everything and writes only the ID + assert analyze_text() == edited.replace(setup.PLACEHOLDER, NEW_RULE) + + +def test_the_analyze_write_keeps_the_file_mode_and_leaves_no_temp_file(): + setup.ANALYZE_FILE.chmod(0o640) + assert setup.main(["setup"], run=FakeAnt()) == 0 + assert setup.ANALYZE_FILE.stat().st_mode & 0o777 == 0o640 + assert sorted(p.name for p in setup.ANALYZE_FILE.parent.iterdir()) == ["analyze.py", "records", "tmp"] + + +def test_every_r_self_claim_is_in_a_recorded_direct_run_token(): + """The rule's claims exist in a direct run's token (V3 run 40259047420), + and its subject prefix matches that token's `sub`.""" + claims = json.loads((ROOT / "review" / "tests" / "unit" / "fixtures" / "oidc" / + "v3-direct-dispatch-claims.json").read_text())["claims"] + assert set(setup.EXPECTED_MATCH["claims"]) <= set(claims) + assert claims["sub"].startswith(setup.EXPECTED_MATCH["subject_prefix"].rstrip("*")) diff --git a/review/tools/release.py b/review/tools/release.py index 9a82f0f..5129daf 100644 --- a/review/tools/release.py +++ b/review/tools/release.py @@ -350,6 +350,10 @@ def logout(self) -> bool: return False return cp.returncode == 0 + def call(self, args: Sequence[str], what: str) -> str: + """Any other `ant` call in this session (self_review_setup.py); returns stdout.""" + return self._ant(args, what) + def retrieve(self, rule_id: str) -> dict: out = self._ant([self.RULES, "retrieve", "--federation-rule-id", rule_id, "--format", "json"], f"ant {self.RULES} retrieve {rule_id}") @@ -365,12 +369,12 @@ def update(self, rule_id: str, match: dict) -> dict: return _json(out, f"update of rule {rule_id}") -def check_ant(run: Runner) -> None: - """`ant` is installed and has the rule commands this tool calls. The +def check_ant(run: Runner, resource: str = Console.RULES, commands: Sequence[str] = ("retrieve", "update")) -> None: + """`ant` is installed and `resource` has the commands this tool calls. The getter name `retrieve` is not confirmed by the docs, so check it here.""" empty = tempfile.mkdtemp(prefix="oc-release-ant-help-") # no profile in it try: - cp = run(["ant", Console.RULES, "--help"], env=ant_env(empty), capture_output=True, text=True) + cp = run(["ant", resource, "--help"], env=ant_env(empty), capture_output=True, text=True) except FileNotFoundError: raise ReleaseError("ant is not installed, so the Console cannot be read. Install it with " "`brew install anthropics/tap/ant`, or run the plan with NO_CONSOLE=1 " @@ -378,9 +382,9 @@ def check_ant(run: Runner) -> None: finally: remove_dir(empty) text = (cp.stdout or "") + (cp.stderr or "") - missing = [c for c in ("retrieve", "update") if not re.search(rf"\b{c}\b", text)] + missing = [c for c in commands if not re.search(rf"\b{c}\b", text)] if cp.returncode != 0 or missing: - raise ReleaseError(f"`ant {Console.RULES} --help` does not list {missing or 'its commands'}; " + raise ReleaseError(f"`ant {resource} --help` does not list {missing or 'its commands'}; " "this ant build does not match the tool; read the help and fix the Console class") @@ -855,6 +859,20 @@ def main(argv: Sequence[str] | None = None, run: Runner | None = None) -> int: s.add_argument("--rule", required=True) s.add_argument("--sha", required=True) args = p.parse_args(argv) + + def interrupted() -> str: + if REPORTS_PRINTED: + return "Interrupted after a write was sent. Follow the RELEASE INCOMPLETE report above." + return "Interrupted before any write was sent. Nothing was changed." + + if args.cmd == "rule-test": + return run_guarded(lambda: cmd_rule_test(args, run), interrupted) + return run_guarded(lambda: cmd_plan(args, run, apply=args.cmd == "release"), interrupted) + + +def run_guarded(fn: Callable[[], int], interrupted: Callable[[], str]) -> int: + """Run one command with the exit rules of every owner tool: a refusal + exits 2, an interrupt 130 with `interrupted()`, a failed cleanup at least 3.""" if hasattr(sys.stdout, "reconfigure"): sys.stdout.reconfigure(line_buffering=True) # keep our lines in order with ant's output for sig in (signal.SIGTERM, signal.SIGHUP): @@ -862,19 +880,12 @@ def main(argv: Sequence[str] | None = None, run: Runner | None = None) -> int: CLEANUP_FAILURES.clear() REPORTS_PRINTED.clear() try: - if args.cmd == "rule-test": - rc = cmd_rule_test(args, run) - else: - rc = cmd_plan(args, run, apply=args.cmd == "release") + rc = fn() except ReleaseError as e: print(f"\nREFUSED: {e}", file=sys.stderr) rc = 2 except KeyboardInterrupt: - if REPORTS_PRINTED: - print("\nInterrupted after a write was sent. Follow the RELEASE INCOMPLETE report above.", - file=sys.stderr) - else: - print("\nInterrupted before any write was sent. Nothing was changed.", file=sys.stderr) + print(f"\n{interrupted()}", file=sys.stderr) rc = 130 if CLEANUP_FAILURES: print(f"\nCLEANUP FAILED ({len(CLEANUP_FAILURES)}): see the ERROR lines above.", file=sys.stderr) diff --git a/review/tools/self_review_setup.py b/review/tools/self_review_setup.py new file mode 100644 index 0000000..b625238 --- /dev/null +++ b/review/tools/self_review_setup.py @@ -0,0 +1,343 @@ +"""Owner-run setup of the self-review federation rule R-self. + + make self-review-plan read-only: shows what the setup would create + make self-review-setup create what is missing, read it back, write the + rule ID into analyze.py (working tree, no commit) + +Spec: docs/claude-review-design.md, "Self-review rule" in "Auth and cost". +All reads and checks run before any write. The tool creates only what is +missing; it never edits or archives an object, so running it again is safe: + + 1. read R1: R-self uses its issuer; its service account must be the one + analyze.py names (R-self uses the same service account) + 2. read: workspace WORKSPACE_NAME by name, the service account's role in + it (must be WORKSPACE_ROLE), rule RULE_NAME by name (every field must + be as the spec says). An unknown list shape, a wrong role or a wrong + rule refuses here, before any write + 3. setup only: create the workspace, add the service account as + WORKSPACE_ROLE, create the rule, each only when missing, and read each + back and check it + 4. write the rule ID into SELF_REVIEW_FEDERATION_RULE_ID in analyze.py, + atomically, only when the file did not change during the run + +The workspace spend limit has no `ant` command, so the tool prints the +Console steps for it. Login, logout, cleanup and exit codes are release.py's +(an ephemeral org:admin `ant` session, cleared on every exit path). + +Not run in CI or any workflow (it holds an Anthropic org:admin login). +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import stat +import subprocess +import sys +import tempfile +from collections.abc import Sequence +from pathlib import Path +import release # sibling module: tools/ is on sys.path when this file runs as a script +from release import ReleaseError + +WORKSPACE_NAME = "claude-review-self-review" +RULE_NAME = "claude-review-self-review" +WORKSPACE_ROLE = "workspace_developer" +OAUTH_SCOPE = "workspace:developer" +TOKEN_LIFETIME_S = 600 +# Pattern mode: the org subject prefix plus exact claims (spec "Self-review rule"). +EXPECTED_MATCH = { + "subject_prefix": "repo:OperationCode@10479364/*", + "claims": { + "repository_id": str(release.WORKFLOW_REPO_ID), + "event_name": "pull_request_target", + "job_workflow_ref": f"{release.JOB_WORKFLOW_REF_PREFIX}refs/heads/main", + }, +} +ANALYZE_FILE = Path(__file__).resolve().parents[1] / "src" / "oc_review" / "analyze.py" +CONSTANT = "SELF_REVIEW_FEDERATION_RULE_ID" +PLACEHOLDER = "fdrl_" + "0" * 24 +WORKSPACES = "organization:workspaces" +MEMBERS = "organization:workspaces:service-accounts" +ANT_COMMANDS = ((release.Console.RULES, ("retrieve", "list", "create")), + (WORKSPACES, ("retrieve", "list", "create")), + (MEMBERS, ("list", "add"))) + + +# ---------------------------------------------------------------- pure logic + +# The fields each list item must have, as non-empty strings. Any other shape +# (another envelope, an item without them) refuses before a create or add. +ITEM_FIELDS = {"workspace": ("id", "name"), "rule": ("id", "name"), + "member": ("service_account_id", "workspace_role")} +PAGE_KEYS = frozenset({"data", "has_more", "first_id", "last_id", "next_page"}) + + +def parse_items(text: str, kind: str, what: str) -> list[dict]: + """Items of a `--format jsonl` list: one item per line, or a whole page + (`{"data": [...]}` with no more pages). Every item must have ITEM_FIELDS.""" + items: list[dict] = [] + for line in text.splitlines(): + if not line.strip(): + continue + obj = release._json(line, what) + batch = [obj] + if isinstance(obj, dict) and "data" in obj: + if set(obj) - PAGE_KEYS or not isinstance(obj["data"], list): + raise ReleaseError(f"{what}: unknown page shape (keys {sorted(obj)}); fix the tool before a write") + if obj.get("has_more") or obj.get("next_page"): + raise ReleaseError(f"{what}: the list has more pages than were read") + batch = obj["data"] + for item in batch: + if not isinstance(item, dict) or not all(isinstance(item.get(f), str) and item[f] + for f in ITEM_FIELDS[kind]): + raise ReleaseError(f"{what}: an item is not a {kind} with {', '.join(ITEM_FIELDS[kind])}; " + "unknown response shape, fix the tool before a write") + items.append(item) + return items + + +def one_named(items: list[dict], name: str, what: str) -> dict | None: + """The one live item called `name`, None when there is none. Several refuse.""" + hits = [i for i in items if i["name"] == name and not i.get("archived_at")] + if len(hits) > 1: + raise ReleaseError(f"{len(hits)} {what}s are named {name!r}; archive the extra ones in the Console, then run again") + return hits[0] if hits else None + + +def member_role(items: list[dict], service_account_id: str) -> str | None: + """The workspace role of the service account, None when it is no member.""" + roles = [i["workspace_role"] for i in items if i["service_account_id"] == service_account_id] + if len(roles) > 1: + raise ReleaseError(f"service account {service_account_id} is listed {len(roles)} times") + return roles[0] if roles else None + + +def rule_problems(rule: dict, *, workspace_id: str, issuer_id: str, service_account_id: str) -> list[str]: + """Every field of R-self that is not as the spec says.""" + problems = [f"match.{p}" for p in release.diff_paths(EXPECTED_MATCH, release._strip_nulls(rule.get("match")))] + target = rule.get("target") or {} + workspaces = rule.get("workspace_ids") or [rule.get("workspace_id")] + expected = { + "issuer_id": (rule.get("issuer_id"), issuer_id), + "oauth_scope": (rule.get("oauth_scope"), OAUTH_SCOPE), + "target.type": (target.get("type"), "service_account"), + "target.service_account_id": (target.get("service_account_id"), service_account_id), + "applies_to_all_workspaces": (bool(rule.get("applies_to_all_workspaces")), False), + "workspace_ids": (workspaces, [workspace_id]), + "token_lifetime_seconds": (rule.get("token_lifetime_seconds"), TOKEN_LIFETIME_S), + "archived_at": (rule.get("archived_at"), None), + } + problems += [f"{k} is {got!r}, expected {want!r}" for k, (got, want) in expected.items() if got != want] + return problems + + +def analyze_constant(text: str, name: str) -> str: + hits = re.findall(rf'^{name}\s*=\s*"([^"]*)"', text, re.M) + if len(hits) != 1: + raise ReleaseError(f"analyze.py has {len(hits)} lines that set {name}, expected 1") + return hits[0] + + +def set_rule_id(text: str, rule_id: str) -> str: + """analyze.py text with R-self set to `rule_id`. Refuses to replace another + real rule ID: only the placeholder or the same ID is accepted.""" + current = analyze_constant(text, CONSTANT) + if current not in (PLACEHOLDER, rule_id): + raise ReleaseError(f"analyze.py already names R-self {current}, not {rule_id}; check the Console, " + "then edit analyze.py by hand") + return re.sub(rf'^({CONSTANT}\s*=\s*)"[^"]*"', rf'\g<1>"{rule_id}"', text, count=1, flags=re.M) + + +def spend_limit_steps() -> str: + return (f"MANUAL STEP (no `ant` command sets a workspace spend limit): in the Console, open the workspace\n" + f" {WORKSPACE_NAME!r}, Limits, and set a low monthly spend limit. Do it before you merge the PR\n" + " that sets the rule ID. Keep the amount out of this public repo.") + + +# ---------------------------------------------------------------- Console steps + + +class Setup: + """`cmd` runs every read and check (`r1`, `find_workspace`, `role`, + `find_rule`) before `write`, so an unknown response shape or a wrong + object refuses before any create or add.""" + + def __init__(self, console: release.Console, rec: release.Records): + self.console, self.rec = console, rec + + def obj(self, args: Sequence[str], what: str) -> dict: + got = release._json(self.console.call([*args, "--format", "json"], what), what) + if not isinstance(got, dict) or not got.get("id"): + raise ReleaseError(f"{what}: the response is not an object with an id") + return got + + def items(self, args: Sequence[str], kind: str, what: str) -> list[dict]: + out = self.console.call([*args, "--format", "jsonl", "--max-items", "-1"], what) + self.rec.save(what.replace(" ", "-"), out) + return parse_items(out, kind, what) + + def r1(self, service_account_id: str) -> str: + r1 = self.console.retrieve(release.PROD_RULE_ID) + self.rec.save("r1", r1) + target = (r1.get("target") or {}).get("service_account_id") + if target != service_account_id: + raise ReleaseError(f"R1 targets service account {target!r}, but analyze.py names {service_account_id}") + issuer = r1.get("issuer_id") + if not isinstance(issuer, str) or not issuer: + raise ReleaseError("R1 has no issuer_id") + print(f" [ok] R1 {release.PROD_RULE_ID}: issuer {issuer}, service account {target}") + return issuer + + def find_workspace(self) -> str | None: + found = one_named(self.items([WORKSPACES, "list"], "workspace", "workspace list"), WORKSPACE_NAME, "workspace") + if found is None: + return None + wid = found["id"] + back = self.obj([WORKSPACES, "retrieve", "--workspace-id", wid], f"workspace {wid}") + self.rec.save("workspace", back) + if back.get("id") != wid or back.get("name") != WORKSPACE_NAME or back.get("archived_at"): + raise ReleaseError(f"workspace {wid} read back as {back.get('name')!r}, archived_at {back.get('archived_at')!r}") + print(f" [ok] workspace {WORKSPACE_NAME!r}: {wid}") + return wid + + def role(self, wid: str, service_account_id: str) -> str | None: + """The service account's role in `wid`; refuses any role but WORKSPACE_ROLE.""" + role = member_role(self.items([MEMBERS, "list", "--workspace-id", wid], "member", f"members of {wid}"), + service_account_id) + if role not in (None, WORKSPACE_ROLE): + raise ReleaseError(f"service account {service_account_id} has role {role!r} in {wid}, not " + f"{WORKSPACE_ROLE}; this tool never edits a membership: fix it in the Console") + if role: + print(f" [ok] service account {service_account_id} is a member of {wid} as {role}") + return role + + def find_rule(self, wid: str | None, issuer_id: str, service_account_id: str) -> str | None: + found = one_named(self.items([release.Console.RULES, "list"], "rule", "rule list"), RULE_NAME, "rule") + if found is None: + return None + rid = release.validate_rule_id(found["id"]) + back = self.console.retrieve(rid) + self.rec.save("rule", back) + problems = rule_problems(back, workspace_id=wid or "", issuer_id=issuer_id, + service_account_id=service_account_id) + if problems: + raise ReleaseError(f"rule {rid} {RULE_NAME!r} is not as the spec says ({'; '.join(problems)}); this " + "tool never edits a rule: fix or archive it in the Console, then run again") + print(f" [ok] rule {RULE_NAME!r}: {rid}, every field as the spec says") + return rid + + def create_args(self, wid: str, issuer_id: str, service_account_id: str) -> list[str]: + return ["--name", RULE_NAME, "--issuer-id", issuer_id, + "--match", json.dumps(EXPECTED_MATCH, separators=(",", ":")), "--oauth-scope", OAUTH_SCOPE, + "--target", json.dumps({"type": "service_account", "service_account_id": service_account_id}, + separators=(",", ":")), + "--workspace-id", wid, "--token-lifetime-seconds", str(TOKEN_LIFETIME_S), + "--description", "Self-review of PRs to OperationCode/.github (spec: Self-review rule)"] + + def write(self, wid: str | None, role: str | None, rule_id: str | None, issuer_id: str, + service_account_id: str) -> str: + """Create what is missing, in order, and read each back. Returns the rule ID.""" + if wid is None: + print(f" workspace {WORKSPACE_NAME!r}: missing, creating it") + self.rec.save("workspace-created", self.obj([WORKSPACES, "create", "--name", WORKSPACE_NAME], + "workspace create")) + wid = self.find_workspace() + if wid is None: + raise ReleaseError(f"workspace {WORKSPACE_NAME!r} was created but the list does not show it") + if role is None: + print(f" service account {service_account_id}: not in {wid}, adding it as {WORKSPACE_ROLE}") + self.console.call([MEMBERS, "add", "--workspace-id", wid, "--service-account-id", service_account_id, + "--workspace-role", WORKSPACE_ROLE, "--format", "json"], "service account add") + if self.role(wid, service_account_id) != WORKSPACE_ROLE: + raise ReleaseError(f"service account {service_account_id} was added but {wid} does not list it " + f"as {WORKSPACE_ROLE}") + if rule_id is None: + print(f" rule {RULE_NAME!r}: missing, creating it") + self.rec.save("rule-created", self.obj([release.Console.RULES, "create", + *self.create_args(wid, issuer_id, service_account_id)], + "rule create")) + rule_id = self.find_rule(wid, issuer_id, service_account_id) + if rule_id is None: + raise ReleaseError(f"rule {RULE_NAME!r} was created but the list does not show it") + return rule_id + + +def write_analyze(path: Path, original: str, new: str) -> None: + """Replace analyze.py atomically, only when it still holds `original` + (the text read at the start): an edit made during the login is kept.""" + if path.read_text(encoding="utf-8") != original: + raise ReleaseError(f"{path} changed during the run, so it was not written. Run `make self-review-setup` " + "again: it finds the rule and only writes the ID") + fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}.") + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(new) + os.chmod(tmp, stat.S_IMODE(path.stat().st_mode)) + os.replace(tmp, path) + except BaseException: + Path(tmp).unlink(missing_ok=True) + raise + + +def cmd(run: release.Runner, *, apply: bool) -> int: + text = ANALYZE_FILE.read_text(encoding="utf-8") + service_account_id = analyze_constant(text, "ANTHROPIC_SERVICE_ACCOUNT_ID") + if analyze_constant(text, "ANTHROPIC_FEDERATION_RULE_ID") != release.PROD_RULE_ID: + raise ReleaseError("analyze.py does not name R1; this tool runs on a checkout of main") + analyze_constant(text, CONSTANT) # fail before any login when the line is missing + rec = release.Records(f"self-review-{'setup' if apply else 'plan'}") + print(f"{'Self-review setup (creates what is missing, no prompt)' if apply else 'Self-review plan (read-only)'}" + "\n\nConsole (reads and checks):") + for resource, commands in ANT_COMMANDS: + release.check_ant(run, resource, commands) + with release.ephemeral_console(run) as console: + s = Setup(console, rec) + issuer = s.r1(service_account_id) + wid = s.find_workspace() + role = s.role(wid, service_account_id) if wid else None + rule_id = s.find_rule(wid, issuer, service_account_id) + if rule_id and wid is None: + raise ReleaseError(f"rule {RULE_NAME!r} exists but workspace {WORKSPACE_NAME!r} does not") + if not apply: + if wid is None: + print(f" workspace {WORKSPACE_NAME!r}: missing, setup creates it") + if role is None: + print(f" service account {service_account_id}: setup adds it to the workspace as {WORKSPACE_ROLE}") + if rule_id is None: + print(f" rule {RULE_NAME!r}: missing, setup creates it with:\n ant {release.Console.RULES} " + "create " + " ".join(s.create_args(wid or "", issuer, service_account_id))) + else: + print("\nConsole (creates what is missing):") + rule_id = s.write(wid, role, rule_id, issuer, service_account_id) + print(f"\n{spend_limit_steps()}\nRecords: {rec.dir}") + if rule_id is None: + print("Plan OK. Nothing was changed.") + return 0 + new = set_rule_id(text, rule_id) + if not apply: + print(f"Plan OK. Nothing was changed. analyze.py: {CONSTANT} " + + ("already names it." if new == text else f"would be set to {rule_id}.")) + return 0 + if new != text: + write_analyze(ANALYZE_FILE, text, new) + print(f"\nR-self is {rule_id}. analyze.py {CONSTANT} {'set' if new != text else 'already named it'} " + "(working tree, not committed). Review `git diff`, then open a PR.") + return 0 + + +def main(argv: Sequence[str] | None = None, run: release.Runner | None = None) -> int: + run = run or subprocess.run + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("cmd", choices=("plan", "setup")) + args = p.parse_args(argv) + return release.run_guarded( + lambda: cmd(run, apply=args.cmd == "setup"), + lambda: "Interrupted. The setup only creates what is missing: run `make self-review-plan` " + "to see the state, then `make self-review-setup` again.") + + +if __name__ == "__main__": + sys.exit(main())