Repository navigation
[GH-3146] Read Parquet GEOMETRY columns in the GeoParquet reader - #3544
Merged
Merged
Conversation
GeoParquet 2.0 stores geometries in the GEOMETRY and GEOGRAPHY logical types of the Parquet format and makes the `geo` metadata optional for readers. The geoparquet reader required that metadata and ignored the logical type. The reader now treats a GEOMETRY column as a geometry column with or without GeoParquet metadata and takes the SRID from the `crs` property of the logical type: absent, authority:code, srid:identifier, inline PROJJSON and projjson:key. A CRS without an EPSG code falls back to the GeoParquet metadata. GEOGRAPHY columns are read as WKB binary. parquet-java has these types since 1.16.0 (Spark 4.1). Sedona also compiles against the parquet-java of Spark 3.5 and 4.0, so the types are reached by reflection. Those older libraries cannot open such files at all, which this change does not address. A file written by Spark 4.2's own Parquet writer records Spark's spatial types in its Spark schema metadata. Schema inference now takes the type of such columns from the Parquet schema, so Spark 4.1 no longer rejects the file.
A GEOGRAPHY column that the GeoParquet metadata lists, as GeoParquet 2.0 does, was read as a planar geometry. The logical type of the Parquet format now decides, so such a column is read as WKB binary. Schema inference converted every column of the Parquet schema to find the geometry columns, so a file with a column the converter rejects, such as INT96 with spark.sql.parquet.int96AsTimestamp=false, could no longer be read. It now converts the geospatial columns alone. A CRS of an authority other than EPSG, such as ESRI:102100, resolves through Spark's registry of coordinate reference systems, which Spark has since 4.2. Spark 4.2 writes such identifiers for its own geometry columns, and the reader rejected them. Older Spark versions have no entry for them, so the SRID stays 0 there. The key of projjson:key is looked up as written instead of trimmed.
…gistry Spark has a registry of coordinate reference systems since 4.2. On older Spark versions nothing can look up an identifier such as ESRI:102100, so a column with that CRS was read with SRID 0. The code of an ESRI identifier is now taken as the SRID there, which is the SRID Spark's registry and PostGIS assign to ESRI codes. The code is not checked. Spark 4.2 keeps asking its registry.
On Spark versions without a registry of coordinate reference systems, the code of an ESRI identifier was taken as the SRID. Sedona reads an SRID as an EPSG code, and ESRI uses the number of some EPSG codes for a different system: ESRI:4812 was read as SRID 4812 and then treated as EPSG:4812, in transformations and when the column was written back. An identifier of another authority gives an SRID only through Spark's registry again, which gives every SRID one owner and has no entry for ESRI:4812. Without a registry the SRID is 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Part of Update GeoParquet writer and reader to support GeoParquet 2.0.0 #3146. This is the first of several PRs for GeoParquet 2.0, so it does not close the issue.What changes were proposed in this PR?
GeoParquet 2.0 stores geometries in the
GEOMETRYandGEOGRAPHYlogical types of the Parquet format, and requires readers to read such files with or without thegeokey. Thegeoparquetreader required that key and ignored the logical type.This PR covers the read side on Spark 4.1 and 4.2:
GEOMETRYcolumn is read as a geometry column, in files with GeoParquet metadata and in files without it. This includes the GeoParquet 2.0.0 example file and files written by Spark 4.2's own Parquet writer.crsproperty of the logical type, in every form the Parquet format allows: absent (OGC:CRS84),authority:code,srid:<identifier>, inline PROJJSON andprojjson:<key>. The key ofprojjson:<key>is looked up exactly as written.OGC:CRS84gives 4326, as before. An identifier of another authority, such asESRI:102100orOGC:CRS83, resolves through Spark's registry of coordinate reference systems. Spark 4.2 ships that registry (EPSG and ESRI entries from PROJ) and writes such identifiers for its own geometry columns.ESRI:4812is notEPSG:4812. Spark's registry gives every SRID one owner and has no entry forESRI:4812, so that identifier gives no SRID on any Spark version.crsof the GeoParquet metadata. A GeoParquet 1.x file with an ESRI identifier that Spark's registry knows was rejected on Spark 4.2 and now loads there with its SRID. On older Spark versions it reads with SRID 0, as before.crsof the GeoParquet metadata. With native Spark geometry types (Spark 4.2), a column whose CRS still gives no SRID is rejected, as it already is for GeoParquet 1.x files; an explicitGeometryUDTread schema stays permissive.GEOGRAPHYcolumns are read as WKB binary. The logical type decides, so this also holds when the GeoParquet metadata lists the column, as GeoParquet 2.0 does. Thegeoparquetformat has no geography support yet; that is a separate PR.GEOSPATIAL_DISABLED. Schema inference now takes the type of a geospatial column from the Parquet schema. Every other column keeps the type of the Spark schema metadata and is not converted, so a column the converter does not handle, such as INT96 withspark.sql.parquet.int96AsTimestamp=false, does not fail the read.parquet-java has the geospatial logical types since 1.16.0, which Spark 4.1 ships. Sedona also compiles against the parquet-java of Spark 3.5 (1.13.1) and 4.0 (1.15.2), so
ParquetGeospatialTypesreaches the types by reflection and the behaviour follows the library found at run time. Those older libraries fail with aNullPointerExceptioninParquetMetadataConverteron any footer that has aGEOMETRYcolumn, for both theparquetand thegeoparquetformat. This PR does not change that; reading such files on Spark 3.5 and 4.0 is a later PR.Not in this PR: writing GeoParquet 2.0, geography columns, and Spark 3.5/4.0.
How was this patch tested?
New fixtures under
spark/common/src/test/resources/geoparquet/native/:example-2.0.0.parquet, the example file of the GeoParquet 2.0.0 release.GEOMETRY(EPSG:3857),GEOMETRY(SRID:0),GEOGRAPHY), and a copy of the geography file with GeoParquet 2.0 metadata added to its footer.crsproperty, with no GeoParquet metadata. They includeESRI:102100and a CRS that nothing resolves,IAU_2015:49900.GeoParquetNativeTypesSuitereads them and checks the type, the geometries, the SRID, a spatial filter on a file without GeoParquet metadata, the geography column with and without GeoParquet metadata, and the error for a plain Parquet file.ParquetGeospatialTypesTestcovers the CRS resolution, including identifiers of other authorities with and without Spark's registry and theESRI:4812case, and the reflection wrapper. The suite registers its tests only when the Parquet library has the geospatial types, so it is empty on Spark 3.5 and 4.0.geoparquetIOTestshas a new test that reads a file with an INT96 column whilespark.sql.parquet.int96AsTimestampis false.NativeGeoParquetCrsSuitein thespark-4.2module writes geometry columns with SRID 102100 and 4269 through Spark's own Parquet writer and reads them back with thegeoparquetreader.The GeoParquet suites pass locally on Spark 4.1 / Scala 2.13, Spark 4.2 / Scala 2.13 (including the GeoParquet suites of the
spark-4.2module) and Spark 3.5 / Scala 2.12.Did this PR include necessary documentation updates?