Skip to content

FIX: Refuse saved converters and scorers that would use this server's credentials (part 11) - #4050

Open
varunj-msft wants to merge 315 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Refuse-Saved-Converter-Scorer-Credentials
Open

varunj-msft wants to merge 315 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Refuse-Saved-Converter-Scorer-Credentials

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 11 of saving Registry entries across restarts. Stacked on part 10 (#4028); part 12 (#4051) builds on it. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Show-Who-Saved...varunj-msft/13178-Refuse-Saved-Converter-Scorer-Credentials

Design: 4.4 Credentials. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4028. Until the earlier PRs merge, the diff here also shows their changes.

A saved converter or scorer is rebuilt at every start and shared by everyone using the app, so restore should refuse one that would sign in with this server's own credentials, as it already does for targets (#4025).

  • The Azure Speech converters (both directions) and AzureContentFilterScorer read this server's key when theirs is empty, or else sign in with its Azure identity, and no value turns that off. They now list that key in server_fallback_parameters (azure_speech_key, api_key), so the shared saved-entry check refuses them, as it does for targets.
  • AudioTrueFalseScorer and AudioFloatScaleScorer transcribe with this server's Speech settings. LocalRefusalClassifierScorer loads its model from a folder on this server, or downloads it with this server's Hugging Face sign-in. These get a not_saveable_reason, like WebSocketCopilotTarget and HuggingFaceChatTarget in FIX: Refuse saved targets that would authenticate with a key (part 7) #4025.
  • server_fallback_parameters is now on the Converter and Scorer bases, and not_saveable_reason on Scorer, as on PromptTarget.
  • A class that declares no auth modes now supports only api_key in the saved-entry check, as check_saveable_class already assumed. A recipe can no longer pass by naming an auth mode its class doesn't take.
  • Each refusal is a restore failure that names the class or parameter, never a value. pyrit_conf.md lists the converters and scorers that can't be saved yet.

Converters and scorers made without saving, and the ones the startup scripts register, aren't checked. RobloxPiiScorer and LocalViolenceClassifierScorer can still be saved: their models are fixed public ones and nothing a caller sets.

Not covered: saving converters and scorers through the API, which is #4051 (part 12). Picking a server credential, or keeping typed keys in a key vault, would let the Speech converters and the Content Safety and audio scorers be saved later.

The banner tied a retry to its failure only through the failed state, so a
retry from an earlier failure could still show its answer when the polls
missed the runtime becoming ready in between. The generation changes only
when the runtime becomes ready or the backend restarts, so a new one now
starts a new failure too.
A long failure list made the banner taller than a phone screen, and the app
doesn't scroll, so Retry could end up out of reach. The list now scrolls on
its own within a third of the screen, and can be scrolled from the keyboard.
When the list can't be loaded, the banner says so, keeps the last list it had,
and keeps trying.
Scripts a scan runs at launch register their targets, converters and scorers
with replace, so one that used a saved entry's name silently replaced it for
everyone, and scripts that skip names already taken (the technique and tools
scripts, round-robin grouping, the scorer script's preferred scorers) left the
scan with the saved entry instead of their own. Restore now marks the entries
it rebuilds, the registries and those scripts refuse to replace or stand in for
a marked entry, and the launch stops with a 409 naming the script and the entry,
even when the script caught the refusal itself. A suite shows the same message
for that entry.
A launch script that caught the refusal and raised its own error skipped the
recorded-refusal check, so the launch failed with that error instead of the
409 naming the saved entry. The clash is now reported on any error once a
refusal was recorded; other errors pass through as before.
Its text can carry a SAS URL. The 503 handler no longer logs it as a debug traceback, and the scenario
suite service no longer attaches it to its warning; both log the storage message, which has no credentials.
# Conflicts:
#	frontend/src/components/Config/CreateTargetDialog.tsx
A suite launch runs its launch scripts during preflight and sends its scans without them, so the clash is
raised there. Map it to 409 with its message, as a single launch does, and drop the per-entry message that
a clash can no longer reach.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants