Repository navigation
Speed up decoding of annotation files in rdann - #627
Open
TheNetherWatcher wants to merge 1 commit into
Open
TheNetherWatcher wants to merge 1 commit into
TheNetherWatcher wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
proc_ann_bytesand its helpers decode an annotation file by indexing the Nx2 NumPy byte array one element at a time (filebytes[bpi, 1],np.int64(...),.astype("i1")), and build each aux note with"".join(chr(c) for c in ...). Each of those makes a NumPy scalar, which is slow in a Python loop.This converts the byte pairs to a list once with
.tolist()and does the rest with plain ints. Signedsubtype/numbytes are converted with a small helper, and aux notes are decoded withbytes(...).decode("latin-1"), which gives the same characters aschr()for 0-255. Since no NumPy scalars are involved any more, the decoding no longer depends on NumPy's scalar type promotion rules, which caused theOverflowErrorin #493 and #522 (already worked around on main withnp.int64).The output is unchanged: I compared every field (
sample,symbol,subtype,chan,num,aux_note,fs,label_store, custom labels, and dtypes) for all 15 annotation files insample-data, with and withoutsampto, against main.Benchmark (median, Apple Silicon, alternating runs of main and this branch; the machine was busy, so the absolute numbers are noisy):
rdann100.atrrdann1003.atrrdann12726.wabp (many aux notes)rdann03700181.gqrslrdann, 100,000 beats with some aux notes (written withwrann)proc_ann_bytesonly, 12726.wabpTests: added a round-trip test for
subtype,chanandnumvalues (including negative subtypes).uv run pytest -m "not network"passes,tests/test_annotation.pypasses with network tests,ruff format/ruff checkare clean, and the docs build without warnings. Added a changelog entry.Open PRs #491 and #498 also touch
annotation.py, but only how the file is opened and read into bytes, not the decoding, so they shouldn't conflict beyond nearby lines.