Repository navigation
fix: decode UTF-8 incrementally in proxy body, bash output and MCP stderr - #247
Merged
Merged
Conversation
…derr Multi-byte characters split across stream chunks were turned into U+FFFD by per-chunk toString()/string concatenation. Use stream setEncoding or a StringDecoder per stream so partial sequences carry over.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the proxy corrupting non-ASCII prompts when a multi-byte UTF-8 character is split across two request body chunks, and fixes the same bug class in the other places that decode a stream chunk by chunk (per CONTRIBUTING's quality bar).
Root cause.
src/proxy/server.tscollected the request withreq.on('data', (chunk: Buffer) => { body += chunk; }).body += chunkconverts each Buffer to a string independently, so a character such as€(3 bytes) cut between twodataevents becomesU+FFFDreplacement characters. The corrupted body is then parsed and forwarded upstream, e.g.Price is €20reaches the gateway asPrice is ���20.Fix. Call
req.setEncoding('utf8')before attaching the listener, so Node's streamStringDecodercarries partial sequences across chunks. Same class, fixed in this PR:src/tools/bash.tsdecoded each stdout/stderr chunk withchunk.toString(), so split characters reached the model as U+FFFD; it now uses oneStringDecoderper stream (theMAX_OUTPUT_BYTEScut is applied to the bytes before decoding).src/mcp/client.tsstderr drain now callssetEncoding('utf8'). The response-side decoder in the proxy already used a streamingTextDecoderand is untouched. No other behaviour changes;src/panel/server.tsalready accumulates Buffers and decodes once, so it is unaffected.Test. Two regression tests in
test/local.mjs: one sends a request whose body is cut inside the euro sign and asserts the backend receives the original text; the other runs a bash command that emits a euro sign split across writes on stdout and stderr. Both fail onmainand pass with the change.Reproduction: POST
{"messages":[{"role":"user","content":"Price is €20"}]}to the proxy in two TCP writes split inside the€bytes (0xE2 0x82 0xAC), 100 ms apart.This PR was prepared with the help of an AI assistant (Claude); the changes were tested as described above.