Skip to content

ReActV2: remember a provider's refusal of a forced tool_choice - #388

Closed
deepfates wants to merge 1 commit into
mainfrom
forced-choice-remembers-refusal
Closed

deepfates wants to merge 1 commit into
mainfrom
forced-choice-remembers-refusal

Conversation

@deepfates

Copy link
Copy Markdown
Owner

What happens now

When a ReActV2 turn ends with a forced submit, Imp first names submit in tool_choice. If the provider refuses that, Imp asks again with tool_choice: "required" and submit as the only tool. If the provider refuses that as well, the tools-disabled extractor writes the answer.

An endpoint that accepts only "auto" (Meta's API, relayed by OpenRouter: "only \"auto\" is supported for tool_choice. \"none\", \"required\", and named function choices are not currently supported") refuses both requests on every forced submit. So every forced submit on such a model sends two requests that are certain to be refused before the extractor's request succeeds. test/react_v2_resume_extract_test.exs already pins that sequence for one resume.

Change

The provider's refusal is kept for the client that received it, for the life of the node (:persistent_term, written once per client and choice). A later forced submit on the same client starts at the first choice the provider has not refused:

  • named and "required" both refused: the request goes straight to the extractor, so the first request sent is one the provider accepts
  • only the named choice refused (endpoints that accept the none/auto/required subset): "required" is sent directly
  • nothing refused: no change

The model registry can't answer this question. LLMDB records whether a model takes tools, and OpenRouter's supported_parameters lists tool_choice for an endpoint that accepts only "auto", so the refusal is the only evidence available. The memory is keyed by the client's module, its model spec and a hash of the whole client. Two clients for the same model that differ in endpoint, options or transport are therefore remembered separately, and the client's API key is never stored. A client with no model spec, such as a static or scripted LM, is not remembered. A provider that starts accepting a forced choice will be forced again after a restart.

Tests

In test/react_v2_resume_extract_test.exs, each test runs a wire-level local HTTP server that returns the real refusal body:

  • refuses_both: a second resume on the same client sends only the extractor's request. Before this change it failed and sent named, then required, then the extractor.
  • refuses_named: a second resume sends only "required". This also failed before the change.
  • Guard: once one model has been refused, a different model on the same provider is still forced by name and ends :forced_submit.

test/react_v2_*.exs, react_contract_test.exs and run_request_record_test.exs pass under five seeds. Those runs check that the existing fallback tests, which share a model spec but use different stub clients, don't affect each other.

An endpoint that accepts only tool_choice "auto" refuses the named
submit and then "required" on every forced submit, so each one sent two
refused requests before the extractor answered. Keep those refusals per
client for the life of the node; a later forced submit on that client
starts at the first choice its provider has not refused, or goes
straight to the extractor.
@deepfates

Copy link
Copy Markdown
Owner Author

Closing without merging. The cost this removes is two refused requests per forced submit on an endpoint that accepts only "auto". The provider rejects them before running the model, and the extractor answers on the third request. Keeping a node-wide record of refusals is more state than that cost justifies. The tests pinning the current fallback order stay as they are.

@deepfates deepfates closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant