Skip to content

Merge upstream develop into QC develop (2026-11-11) - #115

Merged
ChristopherWirt merged 14 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-11
Nov 11, 2026
Merged

ChristopherWirt merged 14 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-11

Conversation

@ChristopherWirt

Copy link
Copy Markdown
Collaborator

Summary

QC develop is missing 13 upstream commits. Merge vortex-data/vortex:develop at cb613ac1c0adc56cbe63dfc79e4bbc431ad0bc52 into QC develop at b2c429ac2830b883cb37c1ff9262dc9e3cb7e483.

Merge this PR with a merge commit to preserve upstream ancestry for future syncs.

Changes

  • Import typed Arrow extension plugins and round-trip metadata preservation, run-end export fixes, and DuckDB dictionary reuse across chunks (#11360, #11375, #11378).
  • Bring in improved string dictionary selection, spatial contains/within pruning, and safe PCO decoding (#9762, #11372, #11363).
  • Include CUDA Widen execution, bounded dictionary gathers, encoded-child host copies, and string dictionaries in the CUDA compression preset.
  • Reconcile the upstream Java I/O ports with their existing QC implementations, preserving read metrics, Kryo-safe Hadoop broadcasts, lifecycle handling, session defaults, and regression coverage. Correct the coalescing default's Javadoc to 1 MiB.

Testing

  • Pinned nightly-2026-09-10 Rust formatting for all affected crates, root/subproject ./gradlew spotlessApply, and clang-format for the changed CUDA kernels passed. Reviewed and included the formatter diff.
  • git diff --check passed; no unresolved merge entries or conflict markers remain. The worktree is clean and both remote tips are parents of the signed-off merge commit.
  • All 13 imported commits have sign-off trailers and retain their original upstream metadata.
  • Tests, builds, lint, and benchmarks were not run locally, following repository guidance.

danking and others added 14 commits November 10, 2026 10:04
Pcodec 1.1.0 exposes a MaybeUninit-based API for safe writing into
uninitialized memory. Let's use it!

Signed-off-by: Daniel King <dan@spiraldb.com>
…host (vortex-data#10916)

## Summary

- Tracking Issue: vortex-data#10914

`compress-bench --gpu-decompress --gpu-verify` panics with `expected
host buffer` on develop for CMSprovider and Euro2016. A run-end encoded
validity mask stays encoded on the device after `execute_cuda`.
`CanonicalCudaExt::into_host` then executes it on the host while its
buffers are still on the device (`validity_into_host` →
`child_into_host` → `RunEnd::execute` → `runend_decode_bools` →
`BufferHandle::as_host`).

A nullable dictionary's validity, a lazy `take` through device-resident
codes, fails the same way.

## Changes

- `canonical.rs`: `child_into_host` copies every device buffer of a
child that isn't canonical to the host before executing it there.
  - Children that are already canonical take the existing path.
- The copies run concurrently, and an array's buffers are replaced only
when one of them moves (Sparse rejects buffer replacement).
  - The decode path is unchanged.
- This doesn't add mixed CPU/GPU execution (vortex-data#7838): the child moves to
the host whole, so nothing decodes from device buffers on the host.
- Tests (4), each building the child on the device and comparing
`into_host` with the expected result:
  - a primitive array whose validity is a run-end encoded mask;
- one whose validity is a sparse mask (Sparse keeps its fill value in a
host buffer it doesn't allow to be replaced, so only its children move);
  - a string dictionary with nullable values and device-resident codes;
- a primitive dictionary with nullable values and device-resident codes.

Evidence (A40, CUDA 13.0):

| Check | develop | this PR |
|---|---|---|
| The 4 tests | all fail: `expected host buffer` (run-end mask),
`as_slice must be called on host buffer` (sparse mask, both
dictionaries) | 4 pass |
| `--gpu-verify` on CMSprovider and Euro2016, today's preset | both
panic | both pass (234 and 22 fields verified) |
| `vortex-cuda` lib suite | — | 711 passed, 0 failed |

Everything in the table ran on this PR alone. With all three PRs,
`--gpu-verify` passes on all 17 GPU-suite datasets.

## AI assistance

Developed with Claude Code (Claude Opus 5.5), which wrote most of the
code, tests and this description under my direction. I reviewed the diff
and the measurements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Tanish Gudise <tgudise@andrew.cmu.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>
## Summary

- Tracking Issue: vortex-data#10701

Progress towards vortex-data#11284 (part 1).

CUDA has no kernel for `vortex.widen`, and the CPU fallback only runs
when every buffer is on the host, so executing a device-resident `Widen`
fails with "CPU fallback with device-resident buffers is not supported".
After CD5, every DecimalByteParts array with a narrow MSP reads from a
file with an integer `Widen` as its MSP, so CD5 can't merge until GPU
reads can execute it.

## Changes

`Widen` is not fused into the dynamic dispatch path. Dynamic dispatch
zero-extends signed narrow BitPacked and FoR children, so fusing a
signed `Widen` there needs care to sign-extend instead. The `TODO` in
`plan_builder.rs` tracks that. Until then, `WidenExecutor` executes the
child on the device and widens it in a separate kernel.

Decimal storage that is wider than the `Widen`'s logical type is an
error instead of a narrowing copy on the device. Narrowing needs either
an overflow check or trust in the stored precision, and that choice is
the open question in part 2 of vortex-data#11284.

The decimal path reuses the kernels in `decimal_cast.cu`, and the Arrow
export now shares `widen_decimal_values` with it, so there is one copy
of the widening code. The tests compare every integer and decimal
widening pair against the CPU `Widen` (with nulls, negative values, a
FoR child, and the kernel's block boundaries). They only run in the GPU
CI.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ms (vortex-data#11368)

## Summary

- Tracking Issue: vortex-data#11365
- Fixes vortex-data#11365

Open pooled Hadoop streams with the known file length so S3A versions
that honor the hint can avoid redundant metadata HEAD requests. Ports
the Hadoop-readable changes from [QC
#106](#106) onto upstream
`develop` at `2e1b05b84`.

## Changes

- Use `fs.openFile()` with optional known-length and random-read hints;
other filesystems may ignore them.
- Preserve original asynchronous-open exception types, restore
interruption and close any stream produced after an interrupted open.
- Add focused tests for supplied/discovered lengths, every pooled stream
receiving the options, optional keys, exception identity and late-stream
cleanup.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
…data#10915)

## Summary

- Tracking Issue: vortex-data#10914

The standalone dictionary kernels in `dict.cu` gathered
`values[codes[i]]` for every row. Nullable codes can hold any value at
null positions (`DictArray::try_new` doesn't bound-check them), so these
gathers read outside the dictionary. The fused planner already rejects
nullable-code dictionaries for this reason and sends them to these
kernels. Develop's own fuzzer, extended here, crashes on this within 71k
runs.

## Changes

- `dict.cu`: every kernel (primitive, decimal, bool and the 16-byte view
gather) takes `values_len`. A code at or beyond it writes a zero value.
In a valid array only null positions can hold such a code, so the zero
is never observed.
- `dict.rs` and `benches/dynamic_dispatch_cuda.rs`: pass `values_len` at
every launch.
- Tests (12), each compared with the expected result through
`assert_arrays_eq`:
- out-of-range codes under nulls for string, decimal, bool and primitive
values, with codes up to `u32::MAX` and `u64::MAX`;
- the dictionary the fuzzer crashed on (one u8 value, null code
3,960,089,894);
  - device-resident codes with nullable values and codes;
  - all-null codes;
  - slices across 1,023/1,024/1,025 and 2,047/2,048;
  - u16 codes with 65,536 values and u32 codes with 70,000;
  - empty, 12-, 13- and 4,097-byte strings;
  - non-UTF-8 binary values.
- `fuzz/src/gpu/mod.rs`: `compress_gpu` also generates string and binary
dictionaries, dictionaries with arbitrary u8/u16/u32 codes under nulls,
and strings compressed with the CUDA preset.

Evidence (A40, CUDA 13.0):

| Check | develop's kernels | this PR |
|---|---|---|
| 5 bug tests: out-of-range codes under nulls (string, decimal, bool,
primitive) and the fuzzer's input | 5 fail with
`CUDA_ERROR_ILLEGAL_ADDRESS` | 5 pass |
| 7 edge-case tests (all-null codes, block-boundary slices, full u16/u32
ranges, inline boundary, binary, device codes) | pass | 7 pass |
| `compute-sanitizer --tool memcheck` on the 6 out-of-range tests | 1-2
errors each, `Invalid __global__ read` of 1, 8 or 16 bytes | 0 errors |
| `compress_gpu` fuzz (CI flags, ASAN, empty corpus) | crash after
70,941 runs, `CUDA_ERROR_ILLEGAL_ADDRESS` | 100,000 runs in 283 s, 0
crashes |
| `vortex-cuda` lib suite | — | 719 passed, 0 failed |

Under nsys, the bounds check costs nothing measurable on numeric
dictionaries: 1.574 vs 1.563 ms per iteration on CMSprovider and 0.825
vs 0.832 ms on taxi.

On develop's kernels the `u32::MAX`/`u64::MAX` tests leave the CUDA
context in a sticky `CUDA_ERROR_ILLEGAL_ADDRESS` state. Run in one
process, they also fail the unrelated tests that follow, about 336 of
them. A regression of this fix would show up the same way, as a wall of
failures led by these tests.

Everything in the table ran on this PR alone, except the develop fuzz
crash: that ran develop's kernels under this PR's extended fuzz target.

## AI assistance

Developed with Claude Code (Claude Opus 5.5), which wrote most of the
code, tests and this description under my direction. I reviewed the diff
and the measurements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Tanish Gudise <tgudise@andrew.cmu.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>
…eset (vortex-data#10917)

## Summary

- Tracking Issue: vortex-data#10914
- Depends on vortex-data#10915 and vortex-data#10916; this branch contains both. The diff of
its own is one line plus a test.

`CompressionMode::Cuda` excludes `StringDictScheme`, but
`execute_dict_varbinview` already decodes string dictionaries on the
GPU. Keeping the scheme makes GPU-mode files 16.0% smaller across
compress-bench's 17-dataset GPU suite (3,351.3 → 2,813.9 MB).
Per-dataset gains: CMSprovider 45.7%, Food 52.9%, HashTags 33.2%, both
TPC-H l_comment variants 12.2%, Euro2016 10.7%. The issue has the full
table.

## Changes

- `vortex-btrblocks/src/builder.rs`: drops `StringDictScheme` from the
Cuda exclusions and updates the comment. A new test checks that the
preset keeps string dictionaries but not binary ones. This is the only
change to CPU-compressor code.

Evidence with this branch (A40, CUDA 13.0):
- `--gpu-verify` passes on all 17 datasets. develop plus only this line
fails 5: taxi, CMSprovider, Euro2016, Food, HashTags.
- The CPU-fallback census is identical to develop's: 59 constant-array
fallbacks, none from dictionaries.
- The dictionary gather takes at most 6.1% of decode time and reaches
62-85% of the measured 563 GB/s.
- `vortex-cuda` lib suite on this branch: 723 passed, 0 failed.
- `compress_gpu` fuzz (CI flags, ASAN): 100,000 runs in 256 s, 0
crashes.

## Benchmarks

Setup:
- `compress-bench --gpu-decompress --iterations 5`, a fresh process per
run, `taskset -c 2-7`, 3 ABBA rounds.
- Speedup = develop time / branch time.
- The same-design A/A floor is 0.14 in |log|, so a "faster" claim needs
a 95% CI above 1 and a speedup above 1.32×.

| Dataset | File MB, develop → branch | Speedup [95% CI] | |
|---|---|---:|---|
| Food | 84.7 → 39.9 | 1.55× [1.41, 1.71] | faster |
| CMSprovider | 771.0 → 418.5 | 1.37× [0.66, 2.86] | within noise |
| Euro2016 | 163.3 → 145.9 | 1.46× [0.92, 2.32] | within noise |
| TPC-H l_comment chunked | 169.6 → 148.8 | 1.36× [0.55, 3.34] | within
noise |
| taxi | 52.0 → 51.5 | 1.28× [0.32, 5.11] | within noise |
| HashTags | 235.2 → 157.0 | 1.22× [0.89, 1.69] | within noise |
| TPC-H l_comment canonical | 169.6 → 148.8 | 1.18× [0.71, 1.96] |
within noise |
| Arade | 144.2 → 141.7 | 1.05× [0.42, 2.68] | within noise |

The nine datasets whose files don't change got one AB/BA round each.
Eight came out at 1.01-1.18×, and one (wide table cols=1000 chunks=50)
at 0.91×, inside the A/A floor. The measurements are from an A40
(GA102); the repo's GPU benchmarks run on A10G.

## API changes

`only_cuda_compatible()`, and therefore `cuda_write_strategy`, can now
emit string dictionaries.
- An `ArrowDeviceArrayStream` with the default
`DictionaryExport::Preserve` rejects a batch whose Arrow schema differs
from the first.
- With this change, 6 of CMSprovider's 14 string columns mix dictionary
and constant chunks.
- Today's Cuda files already do this in numeric columns: 7 of taxi's 18
numeric columns and 5 of CMSprovider's 12.
- Streams with `DictionaryExport::Decode`, which the FFI scan uses, are
unaffected.

Overlap: vortex-data#10425 adds a line next to the removed one, so whichever merges
second needs a one-line rebase. vortex-data#9762 changes dictionary selection, so
the sizes are as of a59294e.

## AI assistance

Developed with Claude Code (Claude Opus 5.5), which wrote most of the
code, tests and this description under my direction. I reviewed the diff
and the measurements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Tanish Gudise <tgudise@andrew.cmu.edu>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alexander Droste <alexander.droste@protonmail.com>
vortex-data#11372)

## Summary

`ST_Contains` filters currently scan even when stored geometry bounds
prove that a file or zone cannot match. Register one
`SpatialContainsPrune` rule that consumes the existing `GeometryAabb`
statistic in both file and zone pruning.

Containment is directional:

- `ST_Contains(column, constant)` can skip a scope whose aggregate box
cannot cover the constant's box.
- `ST_Contains(constant, column)` can skip a scope whose aggregate box
is disjoint from the constant's box. Partial overlap must remain
eligible because individual rows can still match.

DuckDB already lowers `ST_Within(a, b)` to `ST_Contains(b, a)`, so it
uses the same rule. Missing/null bounds, null/empty constants, touching
boxes in the constant-container case, and unsupported predicate shapes
remain conservative. There is no containment proof for negated filters.

The shared column matcher also accepts direct and nested field paths so
file-level proofs can use footer statistics. Unit cases cover
bounding-box relations and operand order; write/read tests exercise file
pruning without zones and zone pruning without file statistics.

---------

Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…ex-data#11371)

Fixes vortex-data#11370.

Writing sliced `ListViewArray` chunks through `ListLayoutStrategy` can
fail with `Array vortex.slice is not registered for serialization`.
Rebuilding the view can leave a lazy elements slice, which is passed
directly to the child writer.

The ListArray path already normalizes its children with `reset_offsets`.
This applies the same normalization after rebuilding a ListView, so its
elements can be serialized too.

---------

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
- Tracking Issue: vortex-data#11366
- Fixes vortex-data#11366

Java-backed readers always use `CoalesceConfig::object_storage()` (a 1
MiB gap, a 16 MiB maximum merged span). Callers can't change it.

The right values depend on the storage underneath. Local SSD, spinning
disk and object stores each trade request count against extra bytes read
differently. This PR lets Java and Spark callers set them, keeping the
current 1 MiB / 16 MiB as the defaults.

---------

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
## Summary

Run-end decoding currently fails for logical types without a specialized
decoder. Arrow export also narrows absolute ends before applying a slice
offset, so a small slice can fail even when its relative ends fit the
requested integer width. A retained run ending at the offset produces a
zero end that Arrow rejects.

## Changes

Use a dictionary as the generic decoding fallback so existing take
implementations preserve the logical dtype. Piecewise codes avoid
building an expanded index buffer during conversion, and empty arrays
retain no source children. Specialized decoders remain in place.

Normalize Arrow run ends before narrowing, trim the values child to the
same runs, and retain the values field's extension metadata so the
exported array agrees with its schema. Unadjusted arrays retain the
existing buffer-sharing path.

Regression coverage includes clipped and boundary slices, large absolute
ends exported as Int16, empty slices, repeated slicing, logical dtype
preservation, and UUID values-field metadata. DuckDB's batch exporter
remains separate.

## API Changes

Adds `RunEndArray::to_dict` and renames the public
`runend::trimmed_ends_iter` helper to `runend::slice_relative_ends`.

<details>
<summary>Validation</summary>

- Formatting with the pinned `nightly-2026-09-10`.
- Scoped Clippy for vortex-arrow and vortex-uuid with all targets, all
features, and warnings denied.
- Commit hook: workspace Rust and TOML formatting.
- Pre-push hook: DCO checks and workspace Clippy with all targets, all
features, and warnings denied.
- Runtime tests and doctests were not rerun.

</details>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…data#11360)

## Summary

Builds on vortex-data#11375.

Exporting a versioned Vortex UUID to Arrow and importing it back
currently drops the declared version, even when every value is
unchanged. Nested extensions can also lose metadata or fail to import
when Arrow fields disagree with their storage. The full dtype, including
nested metadata and nullability, must survive this round trip.

The raw import and export traits leave metadata transport to each
plugin. `ArrowExtVTable` puts that handling behind a typed registration
adapter, so extension authors implement their native conversions while
shared code preserves Vortex metadata.

## Changes

The session checks that inferred fields reconstruct the complete source
dtype and that successful plugin imports return the dtype inferred from
their field. These checks apply to raw and typed plugins. Explicit
export targets can still request conversions.

UUID needs a separate Vortex payload because canonical `arrow.uuid`
metadata must be empty. Keeping that representation lets other Arrow
tools recognize UUID columns. A field without the Vortex payload imports
with an unspecified version, so ordinary Arrow producers remain
supported.

Nested conversions need the full Arrow field because a `DataType` alone
cannot carry extension identity. GeoArrow's required child names also
belong to its layout contract. Metadata restoration uses inferred Vortex
storage because it can differ from Arrow's physical storage.

Regression coverage retains complete dtype and array comparisons while
removing redundant parameter products and assertions. Variant remains
excluded pending vortex-data#11359, and equality derivation remains separate in
vortex-data#11115.

## API Changes

Extension plugins implement `ArrowExtVTable`, declare
`metadata_policy()`, and register through
`ArrowSession::register_extension(vtable)`. The typed import callbacks
are associated functions. Raw plugins remain available, but inferred
fields and imported arrays must satisfy the same dtype checks.

<details>
<summary>Validation</summary>

- Formatting with the pinned `nightly-2026-09-10`.
- Commit hook: workspace Rust and TOML formatting.
- Pre-push hook: DCO checks and workspace Clippy with all targets, all
features, and warnings denied.
- Runtime tests, doctests, and fuzz runs were not repeated for these
refinements.

</details>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
String Dictionary selection now probes a distributed sample before
building an exact candidate. The previous estimator missed profitable
dictionaries when columns contained thousands of repeated strings.
Samples above 75% distinct values reject Dictionary; an incomplete probe
uses the existing estimator. Admitted candidates have explicit limits
for dictionary entries, value bytes, code bytes, and input bytes. The
compressor compares the candidate's actual size and reuses it if it
wins.

The probe reads at most 8,192 rows and 1 MiB of values. A row that
exceeds the byte budget is excluded from the completed-row count,
including the final sampled row. String samples compact their payload
before size estimates so unrelated retained bytes do not distort the
score.

---------

Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
## Summary

Follow-up to vortex-data#11375 and [the DuckDB exporter
review](vortex-data#11360 (comment)).

Chunked run-end inputs currently lose their cached dictionary when an
output batch crosses a chunk boundary. Exposing the existing batch codes
through `DictionaryParts` lets those ranges reuse the values dictionary,
including ranges shared with ordinary dictionary chunks.

## Changes

Keep the run-end specialization because routing through `to_dict()`
would make the generic exporter materialize the entire codes array. The
existing fill loop also remains: using `slice_relative_ends` increased
standalone single-row-run export time in all five trial pairs (median
paired ratio 1.064).

Empty inputs now avoid run lookup and values conversion, and zero-length
requests avoid scalar access. Constant batches include requests ending
exactly at a run boundary. Compact inline regressions cover empty
requests, slice offsets, exact run boundaries, nulls, successive
batches, and shared or different dictionaries across chunks. The
existing null and chunked-values tests remain.

Measurements on `702fd58dea`, before the test and module cleanup, showed
about 15% lower batch-export time for single-row runs and 16% for long
runs. Rust allocations per export loop fall from 146 to 112 and from 212
to 188, respectively. Standalone export and construction timings show no
clear material change. Native DuckDB allocations were not measured.

<details>
<summary>Validation</summary>

- After cleanup, `cargo nextest run -p vortex-duckdb -E
'test(exporter::run_end::)'`: 7 passed, 265 outside the filter.
- Before cleanup, the full exporter filter passed all 80 tests.
- `cargo clippy -p vortex-duckdb --all-targets --all-features -- -D
warnings`: passed.
- `cargo +nightly-2026-09-10 fmt -p vortex-duckdb`: passed.
- Commit hook: workspace Rust and TOML formatting passed.
- Pre-push hook: DCO checks and workspace Clippy with all targets, all
features, and warnings denied passed.
- Broader runtime suites and doctests were not run.

</details>

<details>
<summary>Benchmark methodology and full results</summary>

Baseline: `b367941e9d490b4a19ef9bdd6653f63d712c581f` with the same
benchmark harness. Candidate:
`702fd58dea3c0eeb214d590b3cc6d6b5b3c89c9b`. This comparison isolates the
follow-up from the Arrow extension/property stack.

The [measurement
harness](https://github.com/vortex-data/vortex/blob/702fd58dea3c0eeb214d590b3cc6d6b5b3c89c9b/vortex-duckdb/benches/run_end_export.rs)
and its Cargo target remain available in that measured commit. The
permanent benchmark target was removed during review, along with
duplicate test setup and the module split. The production algorithm is
unchanged, but these measurements were not rerun after the file move.

macOS 27.0.1, aarch64, rustc 1.99.1, LLVM 23.1.1, Cargo bench profile,
and DuckDB 2.0.0. Build both revisions with `cargo bench -p
vortex-duckdb --bench run_end_export --no-run`. Run their saved
executables with `--bench --sample-count 300 --sample-size 20 --color
never` in five pairs, alternating baseline/candidate order. The table
shows the median of each executable's five run medians, plus the median
of the five paired candidate/baseline ratios. These two aggregations
need not agree exactly.

Inputs are non-null i64 run values with run lengths 1 or 8192. Short-run
sources have 16,383 runs, and long-run sources have eight. Chunked
inputs repeat the same source four times. Every input is sliced from row
17 to 13 rows before its end, so exports span multiple batches and chunk
boundaries fall inside output batches. Fixtures execute the slice to its
encoded representation before exporter construction. Construction uses a
fresh conversion cache per iteration. Batch export uses a warm cache and
excludes exporter construction and data-chunk creation.

`AllocProfiler<MiMalloc>` counts Rust allocations in the measured
region, including its timing overhead. It does not count native DuckDB
allocations. Neither `xctrace` nor `heaptrack` is available on this
machine. Allocation figures below are allocation counts and requested
bytes, not peak resident memory.

| Phase | Input | Run length | Baseline (us) | Candidate (us) | Paired
ratio | All five paired ratios |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| batches | standalone | 1 | 14.02 | 13.51 | 0.961 | 0.961, 0.954,
0.968, 0.949, 1.028 |
| batches | standalone | 8192 | 3.349 | 3.293 | 0.980 | 0.866, 1.007,
1.015, 0.926, 0.980 |
| batches | shared chunks | 1 | 48.3 | 41.15 | 0.851 | 0.849, 0.851,
0.858, 0.852, 0.850 |
| batches | shared chunks | 8192 | 19.74 | 16.57 | 0.839 | 0.839, 0.839,
0.833, 0.864, 0.832 |
| construct | standalone | 1 | 0.8631 | 0.8464 | 0.981 | 0.951, 0.979,
0.984, 0.981, 1.016 |
| construct | standalone | 8192 | 0.7922 | 0.7714 | 0.981 | 0.934,
0.982, 0.981, 0.966, 1.004 |
| construct | shared chunks | 1 | 51.69 | 51.68 | 1.000 | 1.012, 0.999,
0.997, 1.000, 1.000 |
| construct | shared chunks | 8192 | 12.9 | 12.88 | 0.998 | 1.002,
0.997, 0.998, 0.998, 0.998 |

| Phase | Input | Run length | Baseline Rust allocations | Candidate
Rust allocations | Baseline bytes | Candidate bytes |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| Batches | Standalone | 1 | 0 | 0 | 0 | 0 |
| Batches | Standalone | 8192 | 0 | 0 | 0 | 0 |
| Batches | Shared chunks | 1 | 146 | 112 | 193.1 KB | 130.5 KB |
| Batches | Shared chunks | 8192 | 212 | 188 | 200.2 KB | 139.3 KB |
| Construct | Standalone | 1 | 5 | 5 | 308 B | 308 B |
| Construct | Standalone | 8192 | 5 | 5 | 308 B | 308 B |
| Construct | Shared chunks | 1 | 32 | 32 | 265.8 KB | 265.8 KB |
| Construct | Shared chunks | 8192 | 22 | 22 | 1.05 MB | 1.05 MB |

The shared-chunk batch timing improved in every pair. Standalone
controls and construction moved much less, and the standalone long-run
ratios varied in direction. No compiler-output analysis was performed,
so those small movements are not attributed to a particular
code-generation change. These microbenchmarks do not establish an
end-to-end query speedup.

</details>

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Merge vortex-data/vortex develop at cb613ac. Preserve QC read metrics, Kryo-safe Hadoop broadcasts, lifecycle fixes, and session defaults while reconciling the upstream Java I/O ports.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt
ChristopherWirt merged commit 97b3669 into develop Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants