ckanext-spatialdata

A CKAN extension that provides geospatial awareness of datastore data.

A product of the Western Pennsylvania Regional Data Center.

Built upon the work fo the Natural History Museum in London.

Overview

This extension provides geospatial awareness of datastore data. This includes:

• Geospatial searches within datasets;
• Pushing GeoJSON data (properties and geometries) in to the datastore;
• Support for PostGIS;
• Support for tile servers that injest PostGIS data. (e.g. martin).
• todo: Spatial extent of datastore searches

Installation

Path variables used below:

• $INSTALL_FOLDER (i.e. where CKAN is installed), e.g. /usr/lib/ckan/default
• $CONFIG_FILE, e.g. /etc/ckan/default/development.ini

1. Clone the repository into the src folder:

cd $INSTALL_FOLDER/src
git clone https://github.com/wprdc/ckanext-spatialdata.git

2. Activate the virtual env:

. $INSTALL_FOLDER/bin/activate

3. Install the requirements from requirements.txt:

cd $INSTALL_FOLDER/src/ckanext-spatialdata
pip install -r requirements.txt

4. Run setup.py:

cd $INSTALL_FOLDER/src/ckanext-spatialdata
python setup.py develop

5. Add 'spatialdata' to the list of plugins in your $CONFIG_FILE:

ckan.plugins = ... spatialdata

Configuration

There are a number of options that can be specified in your .ini config file. They all have defaults set, so none are required.

Name	Description	Default
spatialdata.postgis.field	WGS data field in the PostGIS database	_geom
spatialdata.postgis.mercator_field	Mercator field in the PostGIS database	_geom_webmercator
spatialdata.tileserver_url	Public base URL of the Martin tile server, used for TileJSON links in the dataset page's "Spatially enabled" modal. Empty hides the links.	(empty)
spatialdata.tile_source_id_format	Tile source id pattern; must match source_id_format for the datastore connection in Martin's config. Placeholders: {table}, {column}, {schema}.	table.{table}.{column}
spatialdata.basemap_style_url	MapLibre style JSON used as the basemap in the Map view.	https://tiles.openfreemap.org/styles/positron
spatialdata.tile_minzoom	Lowest zoom the tile server serves for datastore tables. Below it the Map view asks the user to zoom in.	0

Further Setup

PostGIS

To use this extension, your PostgreSQL database must have PostGIS support.

1. Install the correct version of PostGIS for your version of PostgreSQL: https://postgis.net/documentation/getting_started/

2. You will then need to create PostGIS columns on your resources. Invoking the command below will create the two columns named above (spatialdata.postgis.field and spatialdata.postgis.mercator_field) on table $RESOURCE_ID. One represents the WGS (World Geodetic System) data, and one uses the web mercator projection, which is useful for generating maps.

   ckan spatialdata create-columns $RESOURCE_ID -c $CONFIG_FILE

Map view

The plugin provides a spatialdata_view resource view ("Map") that draws a georeferenced resource with MapLibre GL JS (vendored in assets/vendor/maplibre-gl), using the resource's vector tiles from the tile server. Points, lines and polygons are all drawn, and clicking a feature shows its attributes.

• It is offered for resources with spatialdata_active set, and only when spatialdata.tileserver_url is configured.
• A Map view is added automatically when a georeferencing job completes. For resources georeferenced before this existed, add it from the resource's "Views" tab, or run ckan views create spatialdata_view.
• The browser loads tiles straight from the tile server, so the tile server must send CORS headers that allow the CKAN site's origin.

Usage

Geospatial metadata

GeoJSON files can be parsed without any extra metadata.

To parse tabular files, you must update the resources extra fields.

The file can't be parsed unless either spatialdata_longitude_field AND spatialdata_latitude_field are provided OR spatialdata_wkt_field is provided.

Writable Fields

Field	Description
spatialdata_longitude_field	Name of field that contains longitude data
spatialdata_latitude_field	Name of field that contains latitude data
spatialdata_wkt_field	Name of field that contains Well-Known Text data
spatialdata_fields_definition	Optional, Only used with GeoJSON resources. Must be a valid Fields json object. Used to provide field types when loading a GeoJSON into the datastore.'

Read-only fields

Field	Description
spatialdata_active	is true if resource has been georeferenced
spatialdata_status	status of georeferencing job
spatialdata_last_geom_updated	timestamp of last time georeferencing was conducted

Actions

spatialdata_submit

Submit resource qeoreferencing job to be processed by CKAN worker.

curl -X POST https://data.wprdc.org/api/action/spatialdata_submit \
   -H 'Content-Type: application/json' \
   -H 'Authorization: <API_KEY>' \
   -d '{"resource_id": "<RESOURCE_ID>"}'

from ckan.plugins import toolkit

toolkit.get_action('spatialdata_submit')(
    context,
    {
        'resource_id': '<RESOURCE_ID>',
    }
)

spatialdata_submit_package

Submit every eligible resource in a dataset for georeferencing. A resource is eligible if it is a GeoJSON file, or if it is in the datastore and has latitude/longitude or WKT fields configured. Sysadmins only.

curl -X POST https://data.wprdc.org/api/3/action/spatialdata_submit_package \
   -H 'Content-Type: application/json' \
   -H 'Authorization: <API_TOKEN>' \
   -d '{"id": "<DATASET_ID_OR_NAME>"}'

Returns the dataset id, the ids of the resources that were submitted, and the resources that were skipped, each with a reason:

{
  "package_id": "<DATASET_ID>",
  "submitted": ["<RESOURCE_ID>"],
  "skipped": [{"resource_id": "<RESOURCE_ID>", "reason": "No latitude/longitude or WKT field configured on the resource."}]
}

spatialdata_status

Get georeferencing status of a resource with the following data:

Field	Description
job_id	ID of CKAN worker job
status	Status of the worker job
last_updated	Timestamp of last time the status was updated
rows_completed	Number of rows parsed
notes	Description of current status

curl -X POST https://data.wprdc.org/api/3/action/spatialdata_status \
   -H 'Content-Type: application/json' \
   -d '{"resource_id": "<RESOURCE_ID>"}'

from ckan.plugins import toolkit

status = toolkit.get_action('spatialdata_status')(
    context,
    {
        'resource_id': '<RESOURCE_ID>',
    }
)

datastore_search

Searching by geospatial fields involves passing a custom filter to datastore_search. The filter _tmgeom contains a WKT (Well-Known Text) string representing the area to be searched ( currently, only the types POLYGON or MULTIPOLYGON will work). e.g.:

from ckan.plugins import toolkit

search_params = {
    'resource_id': '<RESOURCE_ID>',
    'filters': '_tmgeom:POLYGON(36 114, 36 115, 37 115, 37 114, 36 114)'
}
search = toolkit.get_action(u'datastore_search')(context, search_params)

CLI

spatialdata

1. create-columns: create the PostGIS columns on the $RESOURCE_ID table.

   ckan spatialdata create-columns $RESOURCE_ID --geom-type=$GEOM_TYPE -c $CONFIG_FILE

2. create-index: create index for PostGIS columns on the $RESOURCE_ID table.

   ckan spatialdata create-index $RESOURCE_ID -c $CONFIG_FILE

3. populate-columns: populate the PostGIS columns from the given lat & long fields. Equivalent to the update_geom_columns() action.

   ckan spatialdata populate-columns $RESOURCE_ID -l $LATITUDE_COLUMN -g $LONGITUDE_COLUMN -c $CONFIG_FILE

Testing

tests coming soon

Acknowledgements

Based on ckanext-spatialdata created by the Natural History Museum in London, UK.