Repository navigation
[GH-3178] Add a k-nearest-neighbour index for Geography - #3543
Merged
Merged
Conversation
GeographyKnnIndex ranks geographies by their minimum distance on the sphere. It stores each geography's latitude / longitude bounding rectangle in an STRtree and runs a best-first search that prunes with the distance on the sphere between rectangles, so it stays correct across the antimeridian and near the poles. The index lives in the common module with a generic item type, so that the Spark joins and other engines can share it.
The bounding rectangle of a geography came from Geography.region(), while distances are measured on the shapes behind Functions.distance. The two can read a WKB polygon differently: a ring along the equator was bounded as one hemisphere and measured as the other, and a polygon whose rings enclose no area had no bounds at all although it has a distance. The rectangle is now computed from the shapes that distance uses, edge by edge, and extended to the poles a polygon contains. The rectangle distance loses precision near 180 degrees, and near 90 degrees from the side of a rectangle, by up to about 4e-8 radians. That is more than the 1e-9 radians it was reduced by, so a neighbour could be ranked behind one a few centimeters farther away. The slack is now 1e-6 radians. The polygon tests also wrote WKT with the default locale, which fails to parse where the decimal separator is a comma.
Geographies are added to the index directly, like items to an STRtree, instead of through a builder. The search loop treats tree nodes and geographies as the same kind of queue entry, and the bounding rectangle code drops guards it does not need.
The index now takes the bounding rectangle of a geography from Geography.region(), as ST_Envelope and the other geography joins do, instead of computing it from the shapes of the distance function.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Part of Add Geography support for ST_KNN #3178What changes were proposed in this PR?
This adds
org.apache.sedona.common.geography.GeographyKnnIndex<T>, an in-memory index that answers k-nearest-neighbour queries between geographies. Neighbours are ranked byFunctions.distance(Geography, Geography), the minimum distance on the sphere in meters, which is whatST_Distancereturns for two geographies.It is the search structure behind the geography
ST_KNNjoin in #3542. It is split out and placed in thecommonmodule, with a generic item type, so that the broadcast join, the partitioned join that follows it, and other engines can share it. Nothing calls it yet in this PR.nearest(query, k, includeTies)runs a best-first search over the tree. Nodes are ordered and pruned by the distance on the sphere between the query rectangle and the node rectangle (S2LatLngRect.getDistance), which is a lower bound of the distance to anything under the node. A planar distance in degrees between the same rectangles is not a lower bound across the antimeridian or near the poles.merge(a, b, k, includeTies)combines the neighbours that two disjoint sets of geographies produced for the same query, with the same tie rule asnearest.bounds(Geography)andtoEnvelope(S2LatLngRect)expose the bounding rectangle of a geography and the planar envelope that places it in a planar index. A rectangle that crosses the antimeridian gets the full longitude range.distanceLowerBound(queryRect, envelope)exposes the pruning bound in meters, so that a nearest-neighbour search over another tree of such envelopes can use the same bound.Geography.region().getRectBound(), the same rectangle thatST_Envelopeand the other geography joins use. It follows the edges of the geography and reaches a pole when a polygon contains that pole. A polygon whose rings enclose no area has no such rectangle and is skipped like an empty geography.add, as items are to an STRtree, and all of them have to be added before the first search. After that the index can be searched from several threads. NULL and empty geographies are skipped on insert and have no neighbours as a query.How was this patch tested?
New
GeographyKnnIndexTest:klarger than the number of objects,kbelow 1, ties with and withoutincludeTies, NULL and empty objects and queries, an empty index, adding after the first searchmerge, Java serialization of the neighbours,boundsandtoEnvelopedistanceLowerBoundagainst the distance to random geographies inside the envelope, and across the antimeridianReplacing the rectangle distance with a planar one makes several of these tests fail, and removing the margin on the lower bound makes the two precision tests fail. The tests also pass with a default locale that writes a decimal comma.
Did this PR include necessary documentation updates?