Skip to content

feat: a product's agent surface and operability evals (0.32.0) - #73

Merged
sshlg merged 2 commits into
mainfrom
feat/ai-native-1109
Nov 9, 2026
Merged

sshlg merged 2 commits into
mainfrom
feat/ai-native-1109

Conversation

@sshlg

@sshlg sshlg commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Packet P-AS of the sshlg-skills run 2026-11-09-ai-native
(plan,
design).
A product's agent surface gets a doctrine home inside agent-interop (no new skill), and operability evals get a home in agent-evals.

  • agent-interop/references/agent-surface.md: the canon index AN-1 to AN-10 (home and raw URL per rule); AN-3 one contract for the UI and the agents, its mechanical proxy and the internal: <reason> marker; the schema-first contract source, the generated snapshot (MCP tools/list, OpenAPI, --help --json) and the diff classes none / additive / breaking; AN-7 the gate protocol (typed refusal naming the gate, human_steps with URL and action digest, MCP 2026-07-28 input_required for values the caller can supply, confirm: true is not approval); AN-8 a safe surface (OAuth 2.1 / MCP authorization with per-client consent, Client ID Metadata Documents instead of DCR, the iss check, Fabric T0/T1/T2 as one binding, principal ceiling, effect classes per gateway.md:191, idempotency keys, dry-run, audit row pseudonymised on erasure, untrusted marking, limits naming their window, deprecation with sunset instead of renaming); outbound events and webhooks (citing sheleg-dev's inbound patterns); discovery (AGENTS.md, a proving call run with a real client; llms.txt stays seo-aeo-audit's); a CLI readiness checklist; profiles public, internal, library; notes for feature flags, LLM features, offline sync, SSO consent. Existing rules are cited by file:line, not restated.
  • agent-evals/references/operability.md (AN-9): the proving call and per-capability contract tests as the deterministic gate; the cold-agent run as judgment (one task per agent-channel scenario of a must story, panel personas, pass^k with k = 3, sandbox gate fixtures, state-change asserts, authorization before effect, transcripts as fixtures, spend cited); the eight per-step checks task-pipeline's agent-tester reports.
  • Ownership contradiction fixed (BN-21): tool shape is agent-harness/references/tools.md; the wire and a product's surface are agent-interop's. The description, the body's boundaries and mcp-ship.md now agree.
  • Description lengths (whitespace collapsed): agent-interop 283 → 281 (now names a product's agent surface; "шлюз для агентов" moved to the body); agent-evals unchanged at 294. agent-evals body 4742 → 4734 tokens (cl100k).
  • Triggers: agent-interop "agent surface", "агентная поверхность", "AI-native", "API для агентов", "MCP для продукта", "доступно агентам"; agent-evals "operability eval", "агент проходит сценарий".
  • Evals before prose: AI-OUT-006 to AI-OUT-008 (billing exposed to agents, a gated refund, a CLI made agent-ready) and AE-OUT-006 (operability). Baseline (installed 0.31.4) failed all four; current arm (candidate tree read first) passed 3 of 4, AI-OUT-006 missing only the literal subset (recorded in test/evals/RESULTS.md, checks not tuned after the run). n = 1 per arm.
  • test/audit_regressions/ai-native-agent-surface.py: the canon index has ten rows; every file:line the two references cite resolves and 15 load-bearing citations still hold their text; the description budget; the cases pin their prompts.
  • Version 0.32.0 in package.json, plugin.json, marketplace.json, SKILL-CARD.md, CHANGELOG; ledger and HANDOFF entries.

Checks run

Check Result
npm test (output read in full) exit 0: structural validator OK (19 checks, v0.32.0), family gate and self-test 3/3, plant guard 9, owner gates 16, reap-own 13, installer 11, all 28 audit regression files green
python3 test/evals_validate.py and --self-test OK
claude plugin validate . --strict, claude plugin validate plugins/agent-stack --strict both passed
make-skill 0.33.5 audit_skill.py --house on agent-interop and agent-evals 0 GAP, 19 PASS each
family harness outcome_harness.py on the four new cases each valid
git diff --check, npm pack --dry-run clean; both new references ship
em dash (U+2014) in added lines and new files none
SSHLG_HUB_ROOT=<hub> python3 test/validate.py (what CI runs) FAIL, expected: the hub's routed-trigger check reports the eight new phrases as advertised but routed nowhere

Blocker before merge

CI clones hub main and runs advertised_check.js; the eight new trigger phrases reach no route yet. The hub must add them to lib/triggers.js (agent-stack/agent-interop and agent-stack/agent-evals) or allow-list them with a reason in test/advertised_phrases.js (NOT_ROUTED) before this PR's CI can pass. Routing is the plan's wave 3; the hub's W1 node is the earliest place to allow-list them. Not merged, tagged or released.

🤖 Generated with Claude Code

Робот and others added 2 commits November 9, 2026 14:14
Packet P-AS of the sshlg-skills plan 2026-11-09-ai-native.

- agent-interop references/agent-surface.md: canon index AN-1..AN-10 with
  homes and raw URLs; AN-3 one contract for UI and agents (proxy and the
  internal marker); contract source, generated snapshot and diff classes;
  AN-7 gate protocol (typed refusal, human_steps, MCP 2026-07-28
  input_required, confirm: true is not approval); AN-8 safe surface
  (OAuth 2.1 / MCP authorization, Client ID Metadata Documents, iss check,
  Fabric T0/T1/T2 as one binding, idempotency, dry-run, audit row,
  deprecation instead of renaming); outbound events; discovery; CLI
  readiness; profiles public/internal/library.
- agent-evals references/operability.md: AN-9 proving call and contract
  tests as the gate, cold-agent pass^k as judgment, sandbox gate fixtures.
- Tool shape is agent-harness's, the wire and a product's surface are
  agent-interop's (description, body and mcp-ship.md agree). Description
  283 -> 281 characters.
- Triggers for both skills; four outcome cases written before the prose,
  each failing on the 0.31.4 baseline; a regression file holding the canon
  index, citations, description budget and cases.
- Version 0.32.0 across package.json, plugin.json, marketplace.json,
  SKILL-CARD.md, CHANGELOG; ledger and HANDOFF entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sshlg
sshlg marked this pull request as draft November 9, 2026 13:54
@sshlg
sshlg marked this pull request as ready for review November 9, 2026 13:54
@sshlg
sshlg merged commit b0c0642 into main Nov 9, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant