Repository navigation
[GH-3546] Keep grouped join geometries separate on hash collisions - #3547
Conversation
jiayuasu
left a comment
There was a problem hiding this comment.
Approved. I verified the original correctness bug against the exact base revision: the three collision regressions fail there, while all five tests pass with this fix. The targeted reactor build/Spotless checks, 99 JUnit tests (including three additional local review diagnostics), and 26 KNN tests also passed on Spark 3.5 / Scala 2.12 / Java 11.
Non-blocking performance follow-up: GeometryKey.hashCode() reuses the outer bucket hash, so distinct keys within each LinkedHashMap all collide again. Because GeometryKey is not Comparable, a bucket containing many distinct same-envelope geometries can require quadratic equality comparisons. A synthetic test invoking the compiled bucket function produced about 64 million comparisons for 8,000 valid distinct polygons sharing an envelope. This establishes the scaling behavior, not its prevalence in representative workloads. Please track this separately and evaluate an equality-compatible content hash or another grouping strategy if large collision buckets matter in practice. Preserving the equality rules across geometry types and user data adds complexity, so this should not block the correctness fix.
|
Thanks for the review and the additional validation. I've opened #3550 to track the performance follow-up. I plan to start with synthetic benchmarks of large collision buckets, then evaluate an equality-compatible content hash or alternative grouping strategy based on the results. I'll keep this follow-up separate from the correctness fix in this PR. |
Did you read the Contributor Guide?
Yes, I have read the Contributor Rules and Contributor Development Guide.
Is this PR related to a ticket?
Closes #3546
What changes were proposed in this PR?
collectGeometriesByKeycurrently treats a 32-bit hash as a unique geometry key. When different query geometries or user-data values have the same hash, one query disappears and its matches move to another query.Keep the existing integer hash-bucket shuffle, then split each bucket using exact geometry equality and user-data equality. Preserve the first geometry and all matches for equivalent keys. The private equality key is used only inside a bucket and is never shuffled or returned to callers.
Add five deterministic, self-contained JUnit regressions using synthetic triangles and points. The tests cover colliding geometry hashes, equal user data on different geometries, colliding
"Aa"/"BB"user-data strings on equal geometries, duplicate-match aggregation, and an empty join.How was this patch tested?
Tested with Spark 3.5, Scala 2.12, and Java 17:
master, the new suite runs five tests: the three collision cases fail with merged output groups, while the two compatibility cases pass.JoinQueryHashCollisionTest(5),JoinQueryDeduplicationTest(1), andJoinQueryTest(90) all pass.JoinQueryTestcovers indexed/unindexed spatial joins, distance joins, and flat/count paths.KnnJoinQueryTestsuite passes all 26 tests.Did this PR include necessary documentation updates?
No public API signatures or result types change. This corrects the existing grouped-join behavior, so no documentation update is needed.