Skip to content

feat(api): preview the winning operator rule without delivery - #60

Open
johnnyhuy wants to merge 1 commit into
mainfrom
agent/issue-59-operator-rule-preview
Open

johnnyhuy wants to merge 1 commit into
mainfrom
agent/issue-59-operator-rule-preview

Conversation

@johnnyhuy

Copy link
Copy Markdown
Contributor

Add POST /api/rules/preview to inspect the winning loaded operator rule and its exact response without retrieval, generation or Slack delivery. It uses the existing RAG singleton's matcher, returns a null match when no rule fires, validates the question before matching and hides matcher errors behind HTTP 503.

Part of #6. Closes #59.

Verification

Commands ran with inherited variables cleared using env -i HOME="$HOME" PATH="$PATH". This worktree has no root, backend or frontend environment files. Only fixture Slack values and local SQLite were used.

  • UV_PYTHON=3.13 MISE_YES=1 mise run install passed. The incidental lock-file refresh was discarded; this PR changes no dependencies.
  • UV_PYTHON=3.13 UV_FROZEN=1 MISE_YES=1 SLACK_BOT_TOKEN=xoxb-fixture-only SLACK_SIGNING_SECRET=fixture-only DATABASE_URL=sqlite:///.tmp-tests.db mise run test-backend passed, 239 tests with 1033 existing library and fixture warnings. The 18 new cases cover contains/exact/regex matching, response formatting, priority and config-order ties, disabled rules, empty config, no match, input validation, the shared matcher, API/Slack payload parity and safe failure responses. The fixture asserts that preview never calls Slack, Socket Mode, retrieval, Pi or database sessions.
  • MISE_YES=1 mise run test-frontend passed, 51 tests and 370 assertions.
  • MISE_YES=1 mise run test-runtime passed, 7 offline tests.
  • UV_PYTHON=3.13 UV_FROZEN=1 MISE_YES=1 PYTHONPATH="$PWD/.tmp/offline-rule-preview:$PWD/backend" SLACK_BOT_TOKEN=xoxb-fixture-only SLACK_SIGNING_SECRET=fixture-only DATABASE_URL="sqlite:///$PWD/.tmp/offline-rule-preview/proof.db" OPERATOR_RULES_FILE="$PWD/.tmp/proof/rule-preview-config.json" mise run dev-backend -- --host 127.0.0.1 --port 18059 reached Application startup complete, served the fixture requests, then reached Application shutdown complete after SIGINT.
  • git diff --cached --check passed. Captured text logs have trailing whitespace removed. No migration is required.

Proof

Capture method (a) succeeded. The curl transcript includes requests and responses for the health check, all three match types, a null match, HTTP 422 for blank and non-string questions, and a matching /api/ask answer. It demonstrates that a higher-priority disabled rule is skipped and the approved enabled rule wins. Retrieval chunks and scores do not apply because this endpoint never retrieves; the retrieval policy is unchanged.

Backend output with the new test names, frontend output, runtime output, installation output and startup/shutdown output are committed alongside the fixture.

To reproduce, create .tmp/offline-rule-preview/, copy the proof hook there as sitecustomize.py, start the backend with the command above, then run python3 .tmp/proof/rule-preview-capture.py. The capture script asserts the curl results using fixture configuration. The hook mocks Slack authentication and makes Slack API calls, Socket Mode, retrieval and Pi calls raise.

No dashboard UI changed, so screenshots are not applicable. No capture path was blocked. No live Slack verification is required for this slice, and no Slack message, real token, remote database, model request or paid embedding request was used.

Review notes

Preview exposes the matched pattern and full approved response. As with the existing management API, it needs trusted access controls and has no built-in authentication. It inspects configuration loaded by the backend process; restart backend and bot after configuration changes, since preview cannot confirm what another process has loaded. It neither reloads files nor checks answer freshness, permissions or shadowed rules.

The guide also corrects its stale claim that Slack rule wiring was still pending. Mutable-source rules and the remaining work under parent #6 stay open. Child #59 closes only when this PR merges.

Use the shared RAG matcher to inspect approved answers without retrieval or Pi jobs. Cover matching policy, validation, failure handling and path parity with offline tests and a local curl transcript.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preview the winning operator rule without generating or delivering an answer

1 participant