Repository navigation
[GH-3554] Use signature matching for checkAndGlobPathIfNecessary method selection - #3555
Merged
jiayuasu merged 1 commit intoNov 17, 2026
Conversation
Class.getMethods returns overloads in no defined order and some Spark distributions (e.g. Dataproc) declare extra overloads, so taking the first match by name could invoke the wrong method. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sguhathakurata-CLGX
marked this pull request as ready for review
November 17, 2026 18:08
jiayuasu
approved these changes
Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix for GH-3554: Non-deterministic Method Selection on Google Dataproc
Problem
Sedona 1.9.x uses order-dependent method lookup (
.headOption) to selectDataSource.checkAndGlobPathIfNecessary. On Google Dataproc, this method is overloaded: the standard 6-parameter OSS version coexists with Dataproc's custom 8-parameter version. Java does not guarantee method resolution order, so the wrong overload is selected randomly, causing shapefile read failures approximately 50% of the time.Solution
Replace order-dependent lookup with signature-based method selection. The fix checks parameter types at each position to identify the 6-parameter OSS version, ensuring deterministic method selection regardless of reflection order.
Implementation
Changes in
SedonaFileIndexHelper.scala:The fix is deterministic because the OSS 6-parameter method is uniquely identifiable by having an Integer at parameter position 4, which the Dataproc 8-parameter version does not have.
Validation
Tested Environments
Test Results
All 14 tests passed on Dataproc 3.0.4:
Detailed results available in TEST_RESULTS_IMPROVED_2026-11-17.md.
Not Tested
Risk Assessment
Risk level: Minimal
The change makes method selection more explicit and robust:
Potential concerns:
Backward Compatibility
The fix maintains full backward compatibility:
No changes to public APIs or user-facing behavior.
Why It Works
The 6-parameter OSS method signature is unique and identifiable:
The Integer at position 4 uniquely identifies the OSS method. Regardless of reflection order (whether methods appear as [6, 8] or [8, 6]), signature matching finds the correct overload.
References
sedona/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/SedonaFileIndexHelper.scala