Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Fixed
- Parquet import reads Snappy-compressed files, the default of Spark, parquet-rs and DuckDB, instead of failing for lack of `snappy-java`. ([#645](https://github.com/dfa1/vortex-java/issues/645))
- A decimal column's values are narrowed and compressed under `vortex.decimal_byte_parts`, as Rust does, instead of stored as plain 64-bit integers: TPC-H's decimal columns are 2.5 to 16 times smaller. ([#644](https://github.com/dfa1/vortex-java/issues/644))
- A nullable timestamp column holding nulls is written as `vortex.datetimeparts` with the validity on its days part, as Rust writes it, instead of `vortex.ext` over a masked i64. ([#627](https://github.com/dfa1/vortex-java/issues/627))
- A timestamp column picks the encodings Rust picks: its `vortex.datetimeparts` parts are narrowed first, so runny days with nulls run-length encode (9% smaller files), and its zone-stats bounds stay plain storage when splitting them is no smaller. ([#634](https://github.com/dfa1/vortex-java/issues/634))
Expand Down
5 changes: 5 additions & 0 deletions cli/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
</dependency>
<dependency>
<!-- Same for SNAPPY-compressed Parquet files. -->
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</dependency>
<dependency>
<!-- ImportCommand calls {@code options.writeOptions().allowedCascading()} via the
csv / parquet importers' API surface; that transitively reaches
Expand Down
6 changes: 6 additions & 0 deletions integration/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@
<artifactId>zstd-jni</artifactId>
<scope>test</scope>
</dependency>
<!-- ... and snappy-java for SNAPPY-compressed pages -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.github.dfa1.vortex</groupId>
<artifactId>vortex-writer</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions parquet/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@
<groupId>com.github.luben</groupId>
<artifactId>zstd-jni</artifactId>
</dependency>
<dependency>
<!-- Required by hardwood-core for SNAPPY-compressed Parquet files (optional upstream). -->
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</dependency>
<!-- testing -->
<!-- reader declares the zstd bindings optional; pull it in so the test can decode ZSTD chunks -->
<dependency>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -721,6 +721,29 @@ void projection_unknownColumn_throws(@TempDir Path tmp) throws Exception {
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("does_not_exist");
}

@Test
void snappyCompressedFile_isImported(@TempDir Path tmp) throws Exception {
// Given — pages compressed with SNAPPY, the codec of Spark, parquet-rs and DuckDB (#645)
Path vortex = tmp.resolve("out.vortex");
Path parquet = Path.of(ParquetImporterTest.class
.getResource("/fixtures/snappy_compressed.parquet").toURI());

// When
ParquetImporter.importParquet(parquet, vortex);

// Then
try (VortexReader reader = VortexReader.open(vortex);
ScanIterator iter = reader.scan(ScanOptions.all())) {
long rows = 0;
while (iter.hasNext()) {
try (Chunk c = iter.next()) {
rows += c.rowCount();
}
}
assertThat(rows).isEqualTo(1000L);
}
}
}

private static Path fixture() throws Exception {
Expand Down
Binary file not shown.
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
<jmh.version>1.37</jmh.version>
<!-- testing -->
<zstd-jni.version>1.5.7-24</zstd-jni.version>
<snappy-java.version>1.1.10.8</snappy-java.version>
<h2.version>2.5.252</h2.version>
<junit.version>6.1.3</junit.version>
<jazzer.version>0.30.0</jazzer.version>
Expand Down Expand Up @@ -251,6 +252,11 @@
<artifactId>zstd-jni</artifactId>
<version>${zstd-jni.version}</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy-java.version}</version>
</dependency>

<dependency>
<groupId>org.openjdk.jmh</groupId>
Expand Down
Loading