Repository navigation
Report native read bytes in Spark V2 scan metrics - #108
Merged
Merged
Conversation
ChristopherWirt
added a commit
that referenced
this pull request
Nov 10, 2026
## Summary Fix the shared Linux musl builder failure on `develop` and PRs #103, #105, #106, #107, and #108 after the upstream lazy-constructor update. ## Changes Normalize lists rebuilt from list views through the same `reset_offsets` path as ordinary lists before forwarding their children to layout writers. Rebuilding a sliced list view can now leave a lazy elements slice; a flat child writer cannot serialize that expression. This preserves the existing referenced-element bounds and resolves the slice without forcing unrelated encodings into canonical arrays. Keep the existing nullable sliced-list-view regression and add the non-nullable case. ## CI Failure Analysis ### Status Pre-existing base-branch regression, independent of the individual PR changes. The same failure appears on [develop run 42649358621](https://github.com/quantcast/vortex/actions/runs/42649358621) and every completed failing PR builder inspected (#105–#108). ### Failed Jobs - `Rust tests (linux-musl)`: test failure in `layouts::list::reader::tests::sliced_list_chunks_round_trip::case_4_list_view_chunks`. ### Relevant Log Output ```text Error: Array vortex.slice is not registered for serialization ``` ### Correlation With PR Changes The upstream merge makes array slicing lazy. The list-view branch of `canonicalize_to_list_parts` forwarded a rebuilt list's lazy elements directly, while the ordinary-list branch already normalized them. The Spark PRs do not change this writer; #107 changes separate zone-pruning code. ### Recommended Next Step Merge this shared fix into `develop`. The same patch is being applied to the affected open PR branches so their new builders include the fix immediately. ## Validation - Reproduced the exact failing test on unmodified current `develop` (exit 100, same serialization error). - `cargo nextest run -p vortex-layout -E 'test(layouts::list::)'`: all 161 tests passed after the fix, including the original failing case and new non-nullable case. - `cargo clippy -p vortex-layout --all-targets --all-features -- -D warnings`: passed. - Pinned nightly Rust formatting and `git diff --check`: passed. Local validation ran on macOS; the fresh Linux musl builders will verify the runner-specific job. Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
ChristopherWirt
force-pushed
the
cwirt/fix-issue-99
branch
from
November 10, 2026 10:07
491c8f9 to
f109d2c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Spark V2 scans of Vortex files report 0 input bytes. Vortex reads on native worker threads, but Spark only counts Hadoop
FileSystemstatistics from the task thread.Fix
PooledReadablereports each successful read through a newonBytesReadhook.HadoopReadableforwards it to a caller-suppliedLongConsumer.VortexReadMetricspublishes worker-thread bytes from the task thread into a dedicatedFileSystem.Statisticsentry, so Spark's own refresh keeps them ininputMetrics.bytesRead.vortexBytesReadcustom metric.PooledReadable.close()now waits for in-flight reads, so the final task snapshot includes read-ahead. An interrupted close does not wait.Totals include footer reads and read-ahead. They accumulate across files and coalesced partitions, and stay separate per scan within one task.
Testing
Focused tests pass on Spark 3.5.9 / Scala 2.12 and Spark 4.0.4 / Scala 2.13:
COUNT(*), empty-result andLIMITqueries.Fixes #99
🤖 Generated with Claude Code