Repository navigation
Conversation
Decode chunks written by the HDF5 Blosc2 filter (H5Zblosc2.c in the HDF Group hdf5_plugins repository). Each HDF5 chunk is one Blosc2 frame: chunks of rank >= 2 carry a b2nd metalayer and are decoded with b2nd_to_cbuffer (which removes the block padding), rank 1 chunks with blosc2_schunk_decompress_chunk. The decoded size comes from cd_values[3]. - LibBlosc2: add schunk/b2nd bindings and decodeFrame() - Blosc2Filter + FilterProvider, registered alongside BloscFilter - Tests with a small netCDF-4 file written by netCDF-C 4.9.3 and the HDF5 Blosc2 plugin (rank 1/2/3 variables, bitshuffle/byte shuffle/none, zstd/lz4, partial edge chunks), single chunk frames, truncated and invalid input, and concurrent reads Refs Unidata#1620 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1620 (HDF5 Blosc2 filter, id 32026).
Summary
Adds a decoder for the HDF5 Blosc2 filter (id 32026) to
libblosc2-jna, so netCDF-4/HDF5 files written withH5Zblosc2(hdf5_plugins; also shipped in the Python hdf5plugin package) can be read.Blosc2Filter+Blosc2Filter.Provider, registered inMETA-INF/services/ucar.nc2.filter.FilterProvidernext to the existingBloscFilter(32001). Decode only;encodethrowsUnsupportedOperationException.LibBlosc2.decodeFrame(byte[] frame, int nbytes)follows the decompression branch ofH5Zblosc2.c:blosc2_schunk_from_buffer, no copy);b2nd_from_schunk+b2nd_to_cbuffer;blosc2_schunk_decompress_chunk.cd_values[3](chunk size in bytes), not the frame header, because the frame size can include padding.IOExceptioninstead of reaching native code: frame magic check, null schunk check, return-code checks, size check.libblosc2-native; the functions used are in the bundled C-Blosc2 2.23.1.Tests (
TestBlosc2Filter)Test files are small (about 37 KB each) and were made with netCDF-C + the hdf5_plugins Blosc2 filter via
ncgen(_Filter = "32026,...",_ChunkSizes); seesrc/test/data/README.md.time(rank 1),temp(rank 3, float),count(rank 2, int) compared with an uncompressed copycd_values[3], missingcd_values[3]./gradlew :libblosc2-jna:spotlessCheck :libblosc2-jna:testWithJdk25passes locally (Windows x86_64). I did not have a JDK 8 toolchain locally for the defaulttesttask.I also checked this against 8 files from our project: 7 were bit-identical to their uncompressed originals, and the 8th (an operational file, about 650 million values) read completely with no errors.
🤖 Generated with Claude Code