Skip to content

[GH-3546] Keep grouped join geometries separate on hash collisions - #3547

Merged
jiayuasu merged 1 commit into
apache:masterfrom
linetor-hyundai:fix/join-query-hash-collisions
Nov 16, 2026
Merged

jiayuasu merged 1 commit into
apache:masterfrom
linetor-hyundai:fix/join-query-hash-collisions

Conversation

@linetor-hyundai

Copy link
Copy Markdown
Contributor

Did you read the Contributor Guide?

Yes, I have read the Contributor Rules and Contributor Development Guide.

Is this PR related to a ticket?

Closes #3546

What changes were proposed in this PR?

collectGeometriesByKey currently treats a 32-bit hash as a unique geometry key. When different query geometries or user-data values have the same hash, one query disappears and its matches move to another query.

Keep the existing integer hash-bucket shuffle, then split each bucket using exact geometry equality and user-data equality. Preserve the first geometry and all matches for equivalent keys. The private equality key is used only inside a bucket and is never shuffled or returned to callers.

Add five deterministic, self-contained JUnit regressions using synthetic triangles and points. The tests cover colliding geometry hashes, equal user data on different geometries, colliding "Aa" / "BB" user-data strings on equal geometries, duplicate-match aggregation, and an empty join.

How was this patch tested?

Tested with Spark 3.5, Scala 2.12, and Java 17:

  • On the unmodified public master, the new suite runs five tests: the three collision cases fail with merged output groups, while the two compatibility cases pass.
  • With the fix, JoinQueryHashCollisionTest (5), JoinQueryDeduplicationTest (1), and JoinQueryTest (90) all pass. JoinQueryTest covers indexed/unindexed spatial joins, distance joins, and flat/count paths.
  • The existing KnnJoinQueryTest suite passes all 26 tests.
  • The targeted reactor build and Spotless checks pass.
mvn -pl spark/common -Dspark=3.5 -Dscala=2.12 spotless:apply
mvn -pl spark/common -am -Dspark=3.5 -Dscala=2.12 -DskipTests install
mvn -pl spark/common -Dspark=3.5 -Dscala=2.12 test-compile surefire:test \
  -Dtest=JoinQueryHashCollisionTest,JoinQueryDeduplicationTest,JoinQueryTest
mvn -pl spark/common -Dspark=3.5 -Dscala=2.12 scalatest:test \
  -Dsuites=org.apache.sedona.core.KnnJoinQueryTest

Did this PR include necessary documentation updates?

No public API signatures or result types change. This corrects the existing grouped-join behavior, so no documentation update is needed.

@linetor-hyundai
linetor-hyundai marked this pull request as ready for review November 16, 2026 00:01

@jiayuasu jiayuasu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I verified the original correctness bug against the exact base revision: the three collision regressions fail there, while all five tests pass with this fix. The targeted reactor build/Spotless checks, 99 JUnit tests (including three additional local review diagnostics), and 26 KNN tests also passed on Spark 3.5 / Scala 2.12 / Java 11.

Non-blocking performance follow-up: GeometryKey.hashCode() reuses the outer bucket hash, so distinct keys within each LinkedHashMap all collide again. Because GeometryKey is not Comparable, a bucket containing many distinct same-envelope geometries can require quadratic equality comparisons. A synthetic test invoking the compiled bucket function produced about 64 million comparisons for 8,000 valid distinct polygons sharing an envelope. This establishes the scaling behavior, not its prevalence in representative workloads. Please track this separately and evaluate an equality-compatible content hash or another grouping strategy if large collision buckets matter in practice. Preserving the equality rules across geometry types and user data adds complexity, so this should not block the correctness fix.

@linetor-hyundai

Copy link
Copy Markdown
Contributor Author

Thanks for the review and the additional validation. I've opened #3550 to track the performance follow-up. I plan to start with synthetic benchmarks of large collision buckets, then evaluate an equality-compatible content hash or alternative grouping strategy based on the results. I'll keep this follow-up separate from the correctness fix in this PR.

@jiayuasu
jiayuasu merged commit f70df8d into apache:master Nov 16, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grouped RDD joins merge query geometries when their hashes collide

2 participants