Skip to content

Speed up decoding of annotation files in rdann - #627

Open
TheNetherWatcher wants to merge 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:perf/rdann-decode
Open

TheNetherWatcher wants to merge 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:perf/rdann-decode

Conversation

@TheNetherWatcher

Copy link
Copy Markdown
Contributor

proc_ann_bytes and its helpers decode an annotation file by indexing the Nx2 NumPy byte array one element at a time (filebytes[bpi, 1], np.int64(...), .astype("i1")), and build each aux note with "".join(chr(c) for c in ...). Each of those makes a NumPy scalar, which is slow in a Python loop.

This converts the byte pairs to a list once with .tolist() and does the rest with plain ints. Signed subtype/num bytes are converted with a small helper, and aux notes are decoded with bytes(...).decode("latin-1"), which gives the same characters as chr() for 0-255. Since no NumPy scalars are involved any more, the decoding no longer depends on NumPy's scalar type promotion rules, which caused the OverflowError in #493 and #522 (already worked around on main with np.int64).

The output is unchanged: I compared every field (sample, symbol, subtype, chan, num, aux_note, fs, label_store, custom labels, and dtypes) for all 15 annotation files in sample-data, with and without sampto, against main.

Benchmark (median, Apple Silicon, alternating runs of main and this branch; the machine was busy, so the absolute numbers are noisy):

file main this PR speedup
rdann 100.atr 4.10 ms 2.01 ms 2.0x
rdann 1003.atr 6.24 ms 3.31 ms 1.9x
rdann 12726.wabp (many aux notes) 16.5 ms 6.4 ms 2.6x
rdann 03700181.gqrsl 2.99 ms 1.64 ms 1.8x
rdann, 100,000 beats with some aux notes (written with wrann) 176 ms 76 ms 2.3x
proc_ann_bytes only, 12726.wabp 22.4 ms 7.6 ms 2.9x

Tests: added a round-trip test for subtype, chan and num values (including negative subtypes). uv run pytest -m "not network" passes, tests/test_annotation.py passes with network tests, ruff format/ruff check are clean, and the docs build without warnings. Added a changelog entry.

Open PRs #491 and #498 also touch annotation.py, but only how the file is opened and read into bytes, not the decoding, so they shouldn't conflict beyond nearby lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant