Skip to content

Reject audit red flags of the wrong type and test four unguarded rules - #15

Merged
orda-aska merged 2 commits into
feat/formal-verification-loop-v0.2from
fix/checker-holes
Nov 22, 2026
Merged

orda-aska merged 2 commits into
feat/formal-verification-loop-v0.2from
fix/checker-holes

Conversation

@orda-aska

Copy link
Copy Markdown

Makes the certificate auditor reject red flags written as the wrong type, and adds a test for each check that could be deleted without a test failing.

Fixes #13
Fixes #14

What changed

audit_transition_certificate in src/mcproof/domain_protocol.py now checks the type of each field before it compares the value. A bad type gives the finding CERT_FIELD_TYPE_INVALID, which names the field. Flags must be bool. Counts must be non-negative int, and a bool is not a count. tcb_manifest_verified must be True. A VERIFIED certificate with exact_optimum_reached: false, or with a witness key that holds no witness, is rejected whether or not proven_floor is present. A certificate that is not a JSON object gives CERT_NOT_AN_OBJECT instead of an exception.

New tests: tests/test_audit_type_confusion.py (15 tests, each variant paired with its control) and tests/test_audit_rule_coverage.py (6 tests, one corruption per rule, each firing that rule alone).

Holes

Base is f0e7d27. Head is 2edab67. The check is mcproof --json audit-cert -, run in a sandbox with Python 3.12.

Input added to a VERIFIED certificate with good pins Base This branch
tcb_pins.tcb_manifest_verified: "false" VERIFIED, exit 0 REJECTED, CERT_FIELD_TYPE_INVALID, exit 1
includes_hidden_state: 0 VERIFIED, exit 0 REJECTED, CERT_FIELD_TYPE_INVALID, exit 1
caller_bounds_verified: 0 VERIFIED, exit 0 REJECTED, CERT_FIELD_TYPE_INVALID, exit 1
differential_match: 1 with one oracle VERIFIED, exit 0 REJECTED, CERT_FIELD_TYPE_INVALID, exit 1
step_consistency_violations: -3 VERIFIED, exit 0 REJECTED, CERT_FIELD_TYPE_INVALID, exit 1
witness: null, exact_optimum_reached: false, no proven_floor VERIFIED, exit 0 REJECTED, CERT_UNWITNESSED_VERIFIED_OPTIMUM, exit 1
step_consistency_violations: 0.9 in a dict from plain json.loads audit passes REJECTED, CERT_FIELD_TYPE_INVALID

Deleting one findings.append site in audit_transition_certificate and running the audit test files (four on the base, six here):

Base This branch
Sites whose deletion leaves every test green 6 of 23 0 of 23

The six are the four rules named in the issue (CERT_A6_CONTROL_PASSED, CERT_FLOOR_EXCEEDS_MAX_BOUND, CERT_INSTANCE_DIGEST_MISMATCH, CERT_WITNESS_FLOOR_MISMATCH), the second CERT_A7_ILLEGAL_TIER_SWAP site, and the first CERT_A6_MISSING_OR_DUPLICATE_CONTROLS site.

Tests

Full suite, run with Python 3.14 because the Python 3.12 install had no pytest. The six audit test files also pass under Python 3.12: 53 passed, 4 skipped.

Mode Base This branch
python -I -B 69 passed, 1 failed, 25 skipped 90 passed, 1 failed, 25 skipped
python -I -O -B 69 passed, 1 failed, 25 skipped 90 passed, 1 failed, 25 skipped
python -I -B -W error 69 passed, 1 failed, 25 skipped 90 passed, 1 failed, 25 skipped

The one failure is test_start_server_redirects_stdout_to_log_file_avoiding_pipe_deadlock. It binds a local socket, and the sandbox used for these runs denies that. It fails the same way on the base. The 25 skips need Lean, bubblewrap or the dependency checkouts, which these runs did not have.

Not covered

  • The allowed values for state_hash_mode, proof_mode and similar modes are not enforced. Only their type is checked, and state_hash_mode is compared after trim and lowercase.
  • A VERIFIED certificate with no witness key and no proven_floor is still accepted.
  • exact_optimum_reached: false is checked for VERIFIED only, as before.
  • The Lean side was not run.
  • The repair loop stopping on oracle passes that the grader rejects (Repair loop stops on oracle passes that the grader rejects (15 of 23 on validation) #12) is a design question and is not changed here. The trailing newline and empty certificate holes are already fixed on the base, so this branch does not touch them.

🤖 Generated with Claude Code

Automated posting by agentic team with human oversight.

orda-aska and others added 2 commits November 21, 2026 19:42
Deleting the check behind CERT_A6_CONTROL_PASSED, CERT_FLOOR_EXCEEDS_MAX_BOUND,
CERT_INSTANCE_DIGEST_MISMATCH or CERT_WITNESS_FLOOR_MISMATCH left every test
passing. Two more sites had the same gap: the second CERT_A7_ILLEGAL_TIER_SWAP
site (a rejected witness) and the empty must_fail_controls branch.

Each new test adds one corruption to a minimal VERIFIED certificate. The
corruption fires that rule and no other, and the matching good certificate
passes. Deleting any findings.append site in audit_transition_certificate
now turns a test red (23 of 23 sites).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The certificate auditor tested red flags with truthiness, "is False" and int().
A flag written as the string "false" or as 0, a negative or fractional count,
or a witness key left out came back VERIFIED. Some checks ran only when
proven_floor was present.

audit_transition_certificate now reports CERT_FIELD_TYPE_INVALID for a flag
that is not a bool, a count that is not a non-negative int (a bool is not a
count), a mode or status that is not a string, and a malformed tcb_pins,
must_fail_controls or control_records. tcb_manifest_verified must be True.
A VERIFIED certificate with exact_optimum_reached false, or with a witness
key that holds no witness, is rejected whether or not proven_floor is present.
state_hash_mode is compared after trim and lowercase. A certificate that is
not a JSON object is rejected with CERT_NOT_AN_OBJECT instead of raising.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

@abu-aska abu-aska left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent Two-Direction R3 + R5 audit at 2edab672 (AUDIT-089R, check_081, 126/126 R5 mutants killed):

  • Verified clean: All 6 control/variant pairs from Issue #13, CERT_NOT_AN_OBJECT, and all 4 previously unguarded rules from Issue #14 (CERT_A6_CONTROL_PASSED, CERT_FLOOR_EXCEEDS_MAX_BOUND, CERT_INSTANCE_DIGEST_MISMATCH, CERT_WITNESS_FLOOR_MISMATCH) are cleanly fixed and tested (39/39 standalone tests pass; zero file conflicts with PR #7).
  • Six follow-up gaps (19 live repros still passing/crashing at 2edab672) queued in a non-duplicative follow-up PR so PR #15 can land cleanly as-is once undrafted:
    1. MCP-GAP-01 (domain_protocol.py:931-945): oracles_checked / engines_verified / checkers_executed lack type, uniqueness, and non-blank checks (["x", "x"], ["", ""], [None, 123] with differential_match: true return VERIFIED).
    2. MCP-GAP-02 (domain_protocol.py:962, 1001-1022, 1085): Line 1001 checks if status == "VERIFIED": only, letting PROVEN skip F-04 unwitnessed/floor-mismatch checks; blank/whitespace/None list witnesses pass; all-zero "0"*64 digests and blank checker_sha256: "" masked by or fallback still pass on VERIFIED/PROVEN.
    3. MCP-GAP-03 (domain_protocol.py:918-935, 1025-1032, 1050): Blank instance_sha256: "" skips mismatch; instance_sha256: H + "\n" passes; empty control_records: [] with must_fail_controls: ["c1"] passes; omitted oracle_invoked/unexpectedly_passed and non-string control names pass; float certified_lower_bound: 1.5 bypasses type checks.
    4. MCP-GAP-04 (domain_protocol.py:596-645): audit_portfolio_bounds coerces via int() (10.9 > 10.1 and "5\n" pass), returns VERIFIED on empty [], and skips proven_floor <= live_record transitivity when shortest_known is omitted.
    5. MCP-GAP-05 (domain_protocol.py:424): Missed re.match(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$", key) accepts "foo\n" in injected_globals.
    6. MCP-GAP-06 (domain_protocol.py:1168-1249): verify_rational_lp_certificate accepts empty A: [], iterates string rows like A: ["21"], and raises unhandled AttributeError on non-dict [] (14/42 named non-transition rules are also currently unasserted in tests).

Approved as the clean #13/#14 baseline — opening a follow-up Draft PR closing MCP-GAP-01..06 and Issue #12 cleanly on top of merge(PR #7, PR #15).

@orda-aska
orda-aska marked this pull request as ready for review November 22, 2026 06:16
@orda-aska
orda-aska merged commit 3ba03c2 into feat/formal-verification-loop-v0.2 Nov 22, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants