Skip to content

[GH-3554] Use signature matching for checkAndGlobPathIfNecessary method selection - #3555

Merged
jiayuasu merged 1 commit into
apache:masterfrom
sguhathakurata-CLGX:GH-3554-fix-checkandglob-overload
Nov 17, 2026
Merged

jiayuasu merged 1 commit into
apache:masterfrom
sguhathakurata-CLGX:GH-3554-fix-checkandglob-overload

Conversation

@sguhathakurata-CLGX

@sguhathakurata-CLGX sguhathakurata-CLGX commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fix for GH-3554: Non-deterministic Method Selection on Google Dataproc

Problem

Sedona 1.9.x uses order-dependent method lookup (.headOption) to select DataSource.checkAndGlobPathIfNecessary. On Google Dataproc, this method is overloaded: the standard 6-parameter OSS version coexists with Dataproc's custom 8-parameter version. Java does not guarantee method resolution order, so the wrong overload is selected randomly, causing shapefile read failures approximately 50% of the time.

Solution

Replace order-dependent lookup with signature-based method selection. The fix checks parameter types at each position to identify the 6-parameter OSS version, ensuring deterministic method selection regardless of reflection order.

Implementation

Changes in SedonaFileIndexHelper.scala:

def resolveCheckAndGlobMethod(candidates: Seq[java.lang.reflect.Method]): java.lang.reflect.Method = {
  candidates
    .find(isOssSignature)      // Find by signature, not order
    .orElse(candidates.find(isDatabricksSignature))
    .getOrElse { throw NoSuchMethodException(...) }
}

private def isOssSignature(m: java.lang.reflect.Method): Boolean = {
  val t = m.getParameterTypes
  t.length == 6 && isSeq(t(0)) && classOf[Configuration].isAssignableFrom(t(1)) &&
  isBoolean(t(2)) && isBoolean(t(3)) && isInt(t(4)) && isBoolean(t(5))
                                        // Position 4 is Integer only in OSS version
}

The fix is deterministic because the OSS 6-parameter method is uniquely identifiable by having an Integer at parameter position 4, which the Dataproc 8-parameter version does not have.

Validation

Tested Environments

  • Google Dataproc 3.0.4, Spark 4.1.2: Both method versions present [6, 8], all tests pass
  • Test suite: 14 test cases across file formats, SQL functions, Python API bindings

Test Results

All 14 tests passed on Dataproc 3.0.4:

  • File format reading (5 tests): Shapefile, GeoPackage, GeoParquet
  • SQL geometry functions (5 tests): ST_GeomFromText, ST_Buffer, ST_Distance, ST_Contains, ST_Intersects
  • Python API (3 tests): Geometry creation and operations
  • Method resolution verification (1 test): Both [6] and [8] methods present, correct one selected

Detailed results available in TEST_RESULTS_IMPROVED_2026-11-17.md.

Not Tested

  • Databricks Runtime: Code includes fallback for 5-parameter Databricks signature but was not tested in this session
  • Apache Spark distributions other than Dataproc: Assumed compatible based on code review (only OSS signature needed)
  • Spark versions older than 3.5: Not tested

Risk Assessment

Risk level: Minimal

The change makes method selection more explicit and robust:

  • Previous code relied on method order (non-deterministic)
  • New code relies on parameter type signatures (deterministic)
  • Fallback handlers exist for Databricks and other distributions
  • Code includes error messages listing all discovered methods if no signature matches

Potential concerns:

  • If Dataproc changes the 8-parameter signature to include Integer at position 4: Method would become ambiguous. Mitigation: Error message would list all candidates.
  • If a Spark distribution has an unexpected method signature: Would throw NoSuchMethodException with diagnostic information.

Backward Compatibility

The fix maintains full backward compatibility:

  • Apache Spark (OSS): Uses isOssSignature() to find 6-parameter method
  • Databricks Runtime: Falls back to isDatabricksSignature() for 5-parameter method
  • Google Dataproc: Finds 6-parameter method among the [6, 8] pair

No changes to public APIs or user-facing behavior.


Why It Works

The 6-parameter OSS method signature is unique and identifiable:

Parameter Position OSS (6-param) Dataproc (8-param)
sequences 0 Seq[String] Seq[String]
config 1 Configuration Configuration
boolean1 2 Boolean Boolean
boolean2 3 Boolean Boolean
key_field 4 Integer Not Integer
boolean3 5 Boolean Not present

The Integer at position 4 uniquely identifies the OSS method. Regardless of reflection order (whether methods appear as [6, 8] or [8, 6]), signature matching finds the correct overload.

References

  • Code: sedona/spark/common/src/main/scala/org/apache/spark/sql/execution/datasources/SedonaFileIndexHelper.scala

Class.getMethods returns overloads in no defined order and some Spark
distributions (e.g. Dataproc) declare extra overloads, so taking the
first match by name could invoke the wrong method.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@sguhathakurata-CLGX
sguhathakurata-CLGX marked this pull request as ready for review November 17, 2026 18:08
@jiayuasu jiayuasu changed the title Fix GH-3554: Use signature matching for checkAndGlobPathIfNecessary method selection [GH-3554] Use signature matching for checkAndGlobPathIfNecessary method selection Nov 17, 2026
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Nov 17, 2026

@jiayuasu jiayuasu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@jiayuasu
jiayuasu merged commit 8c13ecb into apache:master Nov 17, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants