Repository navigation
Merge upstream develop into QC develop (2026-11-10) - #113
Merged
ChristopherWirt merged 7 commits intoNov 10, 2026
Merged
Conversation
…x-data#11351) ## Summary - Tracking Issue: vortex-data#11114 Follow-up to vortex-data#11301, which checks that the Arrow type of a field that names a Vortex ID imports as the storage of the array. That import can lose information about nested extension types: the UUID plugin imports every `arrow.uuid` field without a version, and a session that does not know a nested Vortex ID imports it as plain storage. So since vortex-data#11301, some valid exports fail. For example, an extension type whose storage is a struct with a `Uuid(v4)` field can no longer export to its own `to_arrow_storage_field`, which worked before. ## Changes The check now compares nested extension types by their storage types, and keeps the exact comparison at the top level. Nested fields keep their checks, because the exporter checks each nested field when it reaches it, and the temporal executor still rejects a time unit that does not match. The alternative was to require every import plugin to give back exactly the type that it exported. `ArrowImportVTable` does not require that today, and a plugin for a canonical Arrow type such as `arrow.uuid` has no standard place for Vortex-only metadata such as the UUID version. Exact round trips for the plugins that Vortex owns are still worth having, but as a separate goal with its own property tests, not as an assumption of this check. This PR also does the comment pass from the review of vortex-data#11301 on the export dispatch, the map import, and the map test. The map import still runs import plugins for the extension name of the entries field, which `from_arrow_datatype` ignores. The comment now says so instead of claiming that the two match, and a test relies on those plugins running, so the behavior is unchanged. The metadata check that the import and the export both copied is now one helper. New tests cover the nested case (it fails without the fix), an alias of a Vortex ID, and a plain array that exports to a field that names a Vortex ID. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…ortex-data#11127) - Tracking Issue: vortex-data#11124 A scan decodes `4 × available_parallelism()` splits at once. In Spark, every task in an executor runs its own scan sized to the whole machine, and each split holds the decoded chunks of every column it overlaps. On wide, nested files that is GiBs of native memory per executor (vortex-data#11123). This adds a per-scan limit on how many splits are decoded at once. | Layer | API | |---|---| | Rust | `ScanRequest::split_concurrency` → `ScanBuilder::with_some_max_splits_in_flight` | | JNI | `ScanOptions.splitConcurrency()` | | Spark | `vortex.splitConcurrency` read option | **Result:** a full scan of a 180 MB list-layout file, peak live native bytes: | Splits at once | Peak live | Time | |---|---|---| | 48 (default, 4 × 12 cores) | 3.65 GiB | 1.0 s | | 12 | 1.18 GiB | 1.0 s | --------- Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
The default write strategy removed `IntDictScheme` from the one compressor shared by `DictStrategy`'s codes path and its fallback path. The exclusion is only needed for dictionary codes, so they aren't dictionary-encoded twice. But it also applied to every column `DictStrategy` rejects, and those could never use integer dictionaries, even nested inside another encoding. This hits decimals hardest: they always take the fallback, so a `DecimalByteParts` most significant part could never be dict-encoded. --------- Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
…a#11304) ## Summary - Tracking Issue: vortex-data#11114 - Progress towards vortex-data#11286, because the export entry point now takes a target `Field` instead of a `DataType`. An Arrow export could use two different `ArrowSession`s at once, and the API made that easy to do by accident. `ArrowSession::execute_arrow` took the session as its receiver and also an `ExecutionCtx`, which carries its own `VortexSession` and so its own `ArrowSession`. The top-level dispatch and the target checks used the receiver, but plugins such as `Vector` and `Json` export their storage through the session of the ctx. So `ArrowSession::default().execute_arrow(array, None, &mut session.create_execution_ctx())` compiled, and which plugins ran depended on which part of the export was looking. A session that `arrow()` created on first use also could not import what it exported. It was a bare default without the dtype registry, so it wrote an extension type without a plugin under its Vortex ID but read the same field back as plain storage. Only `vortex_arrow::initialize` fixed that, and it replaced the existing session, so calling it after `vortex_uuid::initialize` silently dropped the UUID plugins. ## Changes Exports now go through `ExecuteArrowExt` on the array, which takes its session from the ctx, so there is no second session to pass. Keeping `session.arrow().execute_arrow(..)` and using the session of the ctx inside was rejected, because the receiver would then be silently ignored. A runtime check that both are the same session was rejected too, because it only fails on the paths that tests happen to cover. The old methods are removed instead of deprecated, since a deprecated method that takes a session keeps the same trap open. `arrow()` now builds the session from the dtype registry on first use, through the new `SessionExt::get_or_insert_with`, so every session imports what it exports. That leaves nothing for `initialize` to do, so it is removed instead of fixed. Note that `arrow()` now also creates the `DTypeSession`, so `.with::<DTypeSession>()` after the first `arrow()` call panics, as it already did after `initialize`. New tests cover a UUID session that imports by Vortex ID without `initialize`, and a dtype that is registered after the first `arrow()` call. ## API Changes - `ArrowSession::execute_arrow`, `ArrowSession::exporter`, and `ArrowExporter` are removed. Use `execute_arrow` or `execute_arrow_with_options` from `vortex_arrow::ExecuteArrowExt` on the array instead. - The deprecated `ArrowArrayExecutor` is removed. - `vortex_arrow::initialize` is removed. - `ArrowSession` has no `Default` or public constructor, so every `ArrowSession` comes from a `VortexSession`. - `vortex-session` gains `SessionExt::get_or_insert_with`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Add SparseScheme for FixedSizedList. Instead of compressing just the elements we attempt compressing whole lists using SparseArray Readers need no changes, because `Sparse` already canonicalizes fixed-size lists. On the data from vortex-data#10417: | | Before | After | |---|---|---| | Mostly-null file (1 row in 20 valid) | 25,296 B | 9,056 B | | Rewrite of both files | 292,808 B | 190,768 B | --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
…ta#11373) When zip's mask is uniformly true or false (including null masks, which select the false branch), executing the selected ConstantArray materializes it unnecessarily. Scalar access then logs `Scalar function vortex.zip() returned non-constant array from execution over all scalar inputs`. Return the selected branch cast to the output dtype without executing it again. Removing the zip node already makes execution progress; the caller can preserve constants with `Columnar` or continue materializing with `Canonical`. Added a parameterized regression test for true, false, and null masks, with matching branch types and with nullable/non-nullable branches. It checks constant encoding, values, and output nullability. Validation: reviewed the patch and execution paths. Tests, formatting, and lints were not run locally, following AGENTS.md's instruction to leave verification execution to the user. Implemented and opened by Codex at Daniel King's request. Daniel King (@danking) is the responsible human. Signed-off-by: Daniel King <dan@spiraldb.com>
Merge vortex-data/vortex develop at 7b36f7f. Retain QC scan prefetch bounds, read coalescing, Kryo-safe broadcasts, and session forks while adopting the upstream Arrow export API. Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QC
developis six commits behind upstream. Mergevortex-data/vortex:developat7b36f7fe4fdcd41929e03eaf7cb9da5683df69abinto QCdevelopate252d654187c3f46edcafe2b542aafa6d9c2a0bd, preserving original upstream history.Merge this PR with a merge commit so future syncs and release version discovery retain upstream ancestry.
Changes
ExecuteArrowExt.Testing
7a25d6d5e9bb75f32fc474606f73ab1cf426ed34.ce2b6d027has different author and sign-off email addresses.