Repository navigation
feat: a product's agent surface and operability evals (0.32.0) - #73
Merged
Merged
Conversation
Packet P-AS of the sshlg-skills plan 2026-11-09-ai-native. - agent-interop references/agent-surface.md: canon index AN-1..AN-10 with homes and raw URLs; AN-3 one contract for UI and agents (proxy and the internal marker); contract source, generated snapshot and diff classes; AN-7 gate protocol (typed refusal, human_steps, MCP 2026-07-28 input_required, confirm: true is not approval); AN-8 safe surface (OAuth 2.1 / MCP authorization, Client ID Metadata Documents, iss check, Fabric T0/T1/T2 as one binding, idempotency, dry-run, audit row, deprecation instead of renaming); outbound events; discovery; CLI readiness; profiles public/internal/library. - agent-evals references/operability.md: AN-9 proving call and contract tests as the gate, cold-agent pass^k as judgment, sandbox gate fixtures. - Tool shape is agent-harness's, the wire and a product's surface are agent-interop's (description, body and mcp-ship.md agree). Description 283 -> 281 characters. - Triggers for both skills; four outcome cases written before the prose, each failing on the 0.31.4 baseline; a regression file holding the canon index, citations, description budget and cases. - Version 0.32.0 across package.json, plugin.json, marketplace.json, SKILL-CARD.md, CHANGELOG; ledger and HANDOFF entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sshlg
marked this pull request as draft
November 9, 2026 13:54
sshlg
marked this pull request as ready for review
November 9, 2026 13:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Packet P-AS of the sshlg-skills run
2026-11-09-ai-native(plan,
design).
A product's agent surface gets a doctrine home inside agent-interop (no new skill), and operability evals get a home in agent-evals.
agent-interop/references/agent-surface.md: the canon index AN-1 to AN-10 (home and raw URL per rule); AN-3 one contract for the UI and the agents, its mechanical proxy and theinternal: <reason>marker; the schema-first contract source, the generated snapshot (MCPtools/list, OpenAPI,--help --json) and the diff classesnone/additive/breaking; AN-7 the gate protocol (typed refusal naming the gate,human_stepswith URL and action digest, MCP2026-07-28input_requiredfor values the caller can supply,confirm: trueis not approval); AN-8 a safe surface (OAuth 2.1 / MCP authorization with per-client consent, Client ID Metadata Documents instead of DCR, theisscheck, Fabric T0/T1/T2 as one binding, principal ceiling, effect classes pergateway.md:191, idempotency keys, dry-run, audit row pseudonymised on erasure, untrusted marking, limits naming their window, deprecation with sunset instead of renaming); outbound events and webhooks (citing sheleg-dev's inbound patterns); discovery (AGENTS.md, a proving call run with a real client;llms.txtstays seo-aeo-audit's); a CLI readiness checklist; profilespublic,internal,library; notes for feature flags, LLM features, offline sync, SSO consent. Existing rules are cited byfile:line, not restated.agent-evals/references/operability.md(AN-9): the proving call and per-capability contract tests as the deterministic gate; the cold-agent run as judgment (one task per agent-channel scenario of amuststory, panel personas,pass^kwith k = 3, sandbox gate fixtures, state-change asserts, authorization before effect, transcripts as fixtures, spend cited); the eight per-step checks task-pipeline's agent-tester reports.agent-harness/references/tools.md; the wire and a product's surface are agent-interop's. The description, the body's boundaries andmcp-ship.mdnow agree.AI-OUT-006toAI-OUT-008(billing exposed to agents, a gated refund, a CLI made agent-ready) andAE-OUT-006(operability). Baseline (installed 0.31.4) failed all four; current arm (candidate tree read first) passed 3 of 4, AI-OUT-006 missing only the literalsubset(recorded intest/evals/RESULTS.md, checks not tuned after the run). n = 1 per arm.test/audit_regressions/ai-native-agent-surface.py: the canon index has ten rows; everyfile:linethe two references cite resolves and 15 load-bearing citations still hold their text; the description budget; the cases pin their prompts.package.json,plugin.json,marketplace.json,SKILL-CARD.md, CHANGELOG; ledger and HANDOFF entries.Checks run
npm test(output read in full)python3 test/evals_validate.pyand--self-testclaude plugin validate . --strict,claude plugin validate plugins/agent-stack --strictaudit_skill.py --houseon agent-interop and agent-evalsoutcome_harness.pyon the four new casesgit diff --check,npm pack --dry-runSSHLG_HUB_ROOT=<hub> python3 test/validate.py(what CI runs)Blocker before merge
CI clones hub
mainand runsadvertised_check.js; the eight new trigger phrases reach no route yet. The hub must add them tolib/triggers.js(agent-stack/agent-interop and agent-stack/agent-evals) or allow-list them with a reason intest/advertised_phrases.js(NOT_ROUTED) before this PR's CI can pass. Routing is the plan's wave 3; the hub's W1 node is the earliest place to allow-list them. Not merged, tagged or released.🤖 Generated with Claude Code