Repository navigation
fix(sleep): skip transcript lines that are not readable JSON objects - #388
Merged
Yifan Yang (Yif-Yang) merged 2 commits intoNov 10, 2026
Merged
Yifan Yang (Yif-Yang) merged 2 commits into
Yifan Yang (Yif-Yang) merged 2 commits into
Conversation
_iter_jsonl skipped lines that failed json.loads, but two kinds of bad line still escaped it and aborted the whole harvest: - The file was decoded in text mode outside the per-line try, so a transcript that ends partway through a multi-byte character (a session still being written) raised UnicodeDecodeError out of the generator. - A line that is valid JSON but not an object (null, [], "text", 42) was yielded, and digest_transcript / digest_codex_archived_session then raised AttributeError on rec.get(). Read the file as bytes, decode each line on its own, and yield only dicts, as the Copilot reader (_read_jsonl_snapshot) already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep both CHANGELOG entries; no other change.
Contributor
|
Merged, thank you. A transcript line that is cut mid-character, or that is valid JSON but not an object, is now skipped instead of aborting the whole harvest. On |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #386
_iter_jsonl(used by the Claude Code, Codex and Cursor harvesters) skipped lines that failedjson.loads. Two other kinds of bad line still got past it and aborted the whole harvest:try. A transcript that ends partway through a multi-byte character (for example, a session still being written) raisedUnicodeDecodeErrorout of the generator, and nothing above it catches that.null,[],"text"and42were yielded, anddigest_transcript/digest_codex_archived_sessionthen raisedAttributeErroronrec.get(...).The fix reads the file as bytes, decodes each line on its own, and yields only dicts. That is what the Copilot reader (
_read_jsonl_snapshot) already does. A bad line is now skipped like any other malformed line, and the rest of the session is still harvested.Tests
tests/test_harvest_malformed_lines.pycovers a cut line and non-object lines for the Claude Code harvester, and both together for the Codex harvester. All 3 tests fail onmain(UnicodeDecodeError/AttributeError) and pass with this change.python -m pytest -q: All 3 new tests fail onmainand pass here. Full suite on Linux (python:3.11-slim): 2077 passed; the same 11 tests fail on unmodifiedmainin that container (Devin installer, run-sleep fallback, reproducibility CLI: environment-dependent, unrelated). On Windows the harvest and sleep-engine tests pass apart from failures that also occur onmain.This change was written with Claude Code.
🤖 Generated with Claude Code