Skip to content

fix: decode UTF-8 incrementally in proxy body, bash output and MCP stderr - #247

Merged
VickyXAI merged 1 commit into
BlockRunAI:mainfrom
pcbeingused333:fix/proxy-utf8-body
Nov 18, 2026
Merged

VickyXAI merged 1 commit into
BlockRunAI:mainfrom
pcbeingused333:fix/proxy-utf8-body

Conversation

@pcbeingused333

Copy link
Copy Markdown
Contributor

Fixes the proxy corrupting non-ASCII prompts when a multi-byte UTF-8 character is split across two request body chunks, and fixes the same bug class in the other places that decode a stream chunk by chunk (per CONTRIBUTING's quality bar).

Root cause. src/proxy/server.ts collected the request with req.on('data', (chunk: Buffer) => { body += chunk; }). body += chunk converts each Buffer to a string independently, so a character such as € (3 bytes) cut between two data events becomes U+FFFD replacement characters. The corrupted body is then parsed and forwarded upstream, e.g. Price is €20 reaches the gateway as Price is ���20.

Fix. Call req.setEncoding('utf8') before attaching the listener, so Node's stream StringDecoder carries partial sequences across chunks. Same class, fixed in this PR: src/tools/bash.ts decoded each stdout/stderr chunk with chunk.toString(), so split characters reached the model as U+FFFD; it now uses one StringDecoder per stream (the MAX_OUTPUT_BYTES cut is applied to the bytes before decoding). src/mcp/client.ts stderr drain now calls setEncoding('utf8'). The response-side decoder in the proxy already used a streaming TextDecoder and is untouched. No other behaviour changes; src/panel/server.ts already accumulates Buffers and decodes once, so it is unaffected.

Test. Two regression tests in test/local.mjs: one sends a request whose body is cut inside the euro sign and asserts the backend receives the original text; the other runs a bash command that emits a euro sign split across writes on stdout and stderr. Both fail on main and pass with the change.

Reproduction: POST {"messages":[{"role":"user","content":"Price is €20"}]} to the proxy in two TCP writes split inside the € bytes (0xE2 0x82 0xAC), 100 ms apart.


This PR was prepared with the help of an AI assistant (Claude); the changes were tested as described above.

…derr

Multi-byte characters split across stream chunks were turned into U+FFFD
by per-chunk toString()/string concatenation. Use stream setEncoding or a
StringDecoder per stream so partial sequences carry over.
@VickyXAI
VickyXAI merged commit e6351a6 into BlockRunAI:main Nov 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants