Skip to content

[GH-3146] Read Parquet GEOMETRY columns in the GeoParquet reader - #3544

Merged
jiayuasu merged 4 commits into
apache:masterfrom
jiayuasu:feature/geoparquet-2.0
Nov 17, 2026
Merged

jiayuasu merged 4 commits into
apache:masterfrom
jiayuasu:feature/geoparquet-2.0

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

GeoParquet 2.0 stores geometries in the GEOMETRY and GEOGRAPHY logical types of the Parquet format, and requires readers to read such files with or without the geo key. The geoparquet reader required that key and ignored the logical type.

This PR covers the read side on Spark 4.1 and 4.2:

  • A GEOMETRY column is read as a geometry column, in files with GeoParquet metadata and in files without it. This includes the GeoParquet 2.0.0 example file and files written by Spark 4.2's own Parquet writer.
  • The SRID comes from the crs property of the logical type, in every form the Parquet format allows: absent (OGC:CRS84), authority:code, srid:<identifier>, inline PROJJSON and projjson:<key>. The key of projjson:<key> is looked up exactly as written.
  • An EPSG identifier gives its code and OGC:CRS84 gives 4326, as before. An identifier of another authority, such as ESRI:102100 or OGC:CRS83, resolves through Spark's registry of coordinate reference systems. Spark 4.2 ships that registry (EPSG and ESRI entries from PROJ) and writes such identifiers for its own geometry columns.
  • Spark 4.1 and older have no usable registry (Spark 4.1 has three entries in it), so an identifier of another authority gives SRID 0 there.
  • The code of such an identifier is never taken as the SRID on its own. Sedona reads an SRID as an EPSG code, and ESRI uses the number of some EPSG codes for a different system: ESRI:4812 is not EPSG:4812. Spark's registry gives every SRID one owner and has no entry for ESRI:4812, so that identifier gives no SRID on any Spark version.
  • The same rules apply to the crs of the GeoParquet metadata. A GeoParquet 1.x file with an ESRI identifier that Spark's registry knows was rejected on Spark 4.2 and now loads there with its SRID. On older Spark versions it reads with SRID 0, as before.
  • A CRS that gives no SRID falls back to the crs of the GeoParquet metadata. With native Spark geometry types (Spark 4.2), a column whose CRS still gives no SRID is rejected, as it already is for GeoParquet 1.x files; an explicit GeometryUDT read schema stays permissive.
  • A file with neither GeoParquet metadata nor a geospatial column is still rejected with the existing error.
  • GEOGRAPHY columns are read as WKB binary. The logical type decides, so this also holds when the GeoParquet metadata lists the column, as GeoParquet 2.0 does. The geoparquet format has no geography support yet; that is a separate PR.
  • A file written by Spark 4.2's own writer records Spark's spatial types in its Spark schema metadata. Spark 4.1 parses that type and then rejects the scan with GEOSPATIAL_DISABLED. Schema inference now takes the type of a geospatial column from the Parquet schema. Every other column keeps the type of the Spark schema metadata and is not converted, so a column the converter does not handle, such as INT96 with spark.sql.parquet.int96AsTimestamp=false, does not fail the read.

parquet-java has the geospatial logical types since 1.16.0, which Spark 4.1 ships. Sedona also compiles against the parquet-java of Spark 3.5 (1.13.1) and 4.0 (1.15.2), so ParquetGeospatialTypes reaches the types by reflection and the behaviour follows the library found at run time. Those older libraries fail with a NullPointerException in ParquetMetadataConverter on any footer that has a GEOMETRY column, for both the parquet and the geoparquet format. This PR does not change that; reading such files on Spark 3.5 and 4.0 is a later PR.

Not in this PR: writing GeoParquet 2.0, geography columns, and Spark 3.5/4.0.

How was this patch tested?

New fixtures under spark/common/src/test/resources/geoparquet/native/:

  • example-2.0.0.parquet, the example file of the GeoParquet 2.0.0 release.
  • Three files written by Spark 4.2's own Parquet writer (GEOMETRY(EPSG:3857), GEOMETRY(SRID:0), GEOGRAPHY), and a copy of the geography file with GeoParquet 2.0 metadata added to its footer.
  • Nine files written with parquet-java 1.17.0, one per form of the crs property, with no GeoParquet metadata. They include ESRI:102100 and a CRS that nothing resolves, IAU_2015:49900.

GeoParquetNativeTypesSuite reads them and checks the type, the geometries, the SRID, a spatial filter on a file without GeoParquet metadata, the geography column with and without GeoParquet metadata, and the error for a plain Parquet file. ParquetGeospatialTypesTest covers the CRS resolution, including identifiers of other authorities with and without Spark's registry and the ESRI:4812 case, and the reflection wrapper. The suite registers its tests only when the Parquet library has the geospatial types, so it is empty on Spark 3.5 and 4.0.

geoparquetIOTests has a new test that reads a file with an INT96 column while spark.sql.parquet.int96AsTimestamp is false. NativeGeoParquetCrsSuite in the spark-4.2 module writes geometry columns with SRID 102100 and 4269 through Spark's own Parquet writer and reads them back with the geoparquet reader.

The GeoParquet suites pass locally on Spark 4.1 / Scala 2.13, Spark 4.2 / Scala 2.13 (including the GeoParquet suites of the spark-4.2 module) and Spark 3.5 / Scala 2.12.

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation.

GeoParquet 2.0 stores geometries in the GEOMETRY and GEOGRAPHY logical
types of the Parquet format and makes the `geo` metadata optional for
readers. The geoparquet reader required that metadata and ignored the
logical type.

The reader now treats a GEOMETRY column as a geometry column with or
without GeoParquet metadata and takes the SRID from the `crs` property
of the logical type: absent, authority:code, srid:identifier, inline
PROJJSON and projjson:key. A CRS without an EPSG code falls back to the
GeoParquet metadata. GEOGRAPHY columns are read as WKB binary.

parquet-java has these types since 1.16.0 (Spark 4.1). Sedona also
compiles against the parquet-java of Spark 3.5 and 4.0, so the types
are reached by reflection. Those older libraries cannot open such files
at all, which this change does not address.

A file written by Spark 4.2's own Parquet writer records Spark's
spatial types in its Spark schema metadata. Schema inference now takes
the type of such columns from the Parquet schema, so Spark 4.1 no
longer rejects the file.
A GEOGRAPHY column that the GeoParquet metadata lists, as GeoParquet 2.0
does, was read as a planar geometry. The logical type of the Parquet
format now decides, so such a column is read as WKB binary.

Schema inference converted every column of the Parquet schema to find
the geometry columns, so a file with a column the converter rejects,
such as INT96 with spark.sql.parquet.int96AsTimestamp=false, could no
longer be read. It now converts the geospatial columns alone.

A CRS of an authority other than EPSG, such as ESRI:102100, resolves
through Spark's registry of coordinate reference systems, which Spark
has since 4.2. Spark 4.2 writes such identifiers for its own geometry
columns, and the reader rejected them. Older Spark versions have no
entry for them, so the SRID stays 0 there.

The key of projjson:key is looked up as written instead of trimmed.
…gistry

Spark has a registry of coordinate reference systems since 4.2. On older
Spark versions nothing can look up an identifier such as ESRI:102100,
so a column with that CRS was read with SRID 0. The code of an ESRI
identifier is now taken as the SRID there, which is the SRID Spark's
registry and PostGIS assign to ESRI codes. The code is not checked.
Spark 4.2 keeps asking its registry.
On Spark versions without a registry of coordinate reference systems,
the code of an ESRI identifier was taken as the SRID. Sedona reads an
SRID as an EPSG code, and ESRI uses the number of some EPSG codes for
a different system: ESRI:4812 was read as SRID 4812 and then treated as
EPSG:4812, in transformations and when the column was written back.

An identifier of another authority gives an SRID only through Spark's
registry again, which gives every SRID one owner and has no entry for
ESRI:4812. Without a registry the SRID is 0.
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Nov 17, 2026
@jiayuasu
jiayuasu merged commit 4d414f7 into apache:master Nov 17, 2026
36 of 41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant