Adding Spatial Indices & Metadata¶

The add commands enhance GeoParquet files with spatial indices, geometry metrics, and administrative metadata.

Bounding Boxes¶

Add precomputed bounding boxes for faster spatial queries:

gpio add bbox input.parquet output.parquet

# Works with remote files
gpio --aws-profile prod add bbox s3://bucket/input.parquet s3://bucket/output.parquet

import geoparquet_io as gpio

gpio.read('input.parquet').add_bbox().write('output.parquet')

# Custom column name
gpio.read('input.parquet').add_bbox(column_name='bounds').write('output.parquet')

Creates a struct column with {xmin, ymin, xmax, ymax} for each feature, plus the covering metadata that points at it. Because gpio computes the column from the geometry in the same statement, it can declare the covering. For a column gpio did not compute, the only name it will vouch for is a struct column called exactly bbox — the universal convention every 1.0-era writer emitted before covering existed, which any write carries forward with a covering. Any other name (bounds, extent, tile_bbox) is left undeclared, since a covering asserts a relationship gpio cannot verify from a column name; gpio add bbox-metadata is where you assert it deliberately. Inventing one is the only thing the name rule governs: a covering that already exists — one gpio just computed under --bbox-name bounds, or one GDAL wrote over geometry_bbox — is somebody's assertion already, and every write carries it forward whatever the column is called. The covering key is not part of the GeoParquet 2.0 specification text — it was introduced in 1.1 and removed in 2.0 in favour of the native statistics. 2.0 readers must tolerate unknown fields, so a covering stays legal to carry, and geoparquet#302 proposes reinstating it as an option (still open at time of writing). The motivation is real either way: native statistics prune whole row groups, while a bbox column's page index also prunes pages within one.

Existing Bbox Detection¶

An "existing bbox column" means the same two things everywhere in gpio: the column the primary geometry column's own covering.bbox names, or a struct column called exactly bbox that no other column's covering claims. A bounds, tile_extent or boundary_bbox struct nothing declared is ordinary data — gpio add bbox computes its own column beside it, gpio extract --bbox filters on the geometry rather than pre-filtering on values nothing vouched for, and gpio add bbox-metadata reports that there is no bbox column to declare unless you name one with --bbox-name.

The command automatically checks for existing bbox columns:

• If bbox exists with metadata: Nothing is recomputed, and the input is copied to the output file unchanged
• If bbox exists without metadata: Same copy, plus a suggestion to use gpio add bbox-metadata
• If --bbox-name asks for a different name: That column is computed and added alongside the existing one, with a warning that the file will have two bbox columns
• If a write option is asked for explicitly (--geoparquet-version, --compression, --compression-level, --row-group-size): the column is recomputed instead of copied, because a verbatim copy reproduces the input's settings and cannot honour the requested ones
• Use --force: Replace existing bbox column with a freshly computed one
• If the name is taken by something that is not a bbox struct (a string tile id called bbox, or the same name spelled BBOX): that column is the input's own data, so it is kept as it is and the computed bbox is written under the next free name (bbox_1, bbox_2, …) with the covering declared over that column; a warning names both. gpio convert and the Python API's Table.add_bbox() take the same decision, so one collision means one thing whichever path writes the column (#1176)

The copy matters for pipelines: gpio add bbox in.parquet out.parquet always leaves a file at out.parquet with a bbox column, whether or not one had to be computed, so the next step in a script has something to read (#728). Every copy is reported on the console, so a recomputed bbox is never confused with a copied one. --dry-run says which of the two it would be. When no output file is given there is nothing to write, and the command only reports.

The same holds for a stream: ... | gpio add bbox - out.parquet on an input that already has a bbox column passes the data through untouched and declares no covering for a column gpio did not compute.

That copy is a remote write like any other, so it goes through the object store gpio is configured to use: --s3-endpoint, --s3-region, --s3-no-ssl and --aws-profile apply to it exactly as they do to a recomputed output, and a copy to MinIO no longer ends up pointed at AWS (#810). Inputs may be local paths or s3://, gs:// and https:// URLs — an https:// input is fetched with a plain streaming GET of the URL exactly as given, so a presigned URL keeps its signature. Outputs may be local paths or s3://, gs:// and az:// URLs. az:// is a write destination only: reading from Azure is not supported, and an az:// input is refused up front with a message saying so, rather than failing deep in the metadata probe. An Azure output URL is written account-first, az://<account>/<container>/<path>, with the credential taken from AZURE_STORAGE_ACCOUNT_KEY or AZURE_STORAGE_SAS_TOKEN (#864); the abfs://, abfss:// and azure:// spellings order those parts differently and are refused rather than mis-read. Any scheme the copy cannot serve is refused by name before anything is read, rather than failing partway through.

The copy resolves S3 credentials the same way gpio publish upload does: through the full AWS chain — environment keys, --aws-profile, ~/.aws/credentials, and the SSO, assume-role and credential_process configurations botocore supports (#865) — so a session that can read the input can also write the copy. Credentials are resolved once at the start of the write; a session token that expires mid-upload is not refreshed.

# Already has a bbox: copies input to output, computing nothing
gpio add bbox input.parquet output.parquet

# Force replace existing bbox
gpio add bbox input.parquet output.parquet --force

Options:

# Custom column name
gpio add bbox input.parquet output.parquet --bbox-name bounds

# Force replace existing bbox
gpio add bbox input.parquet output.parquet --force

# With compression settings
gpio add bbox input.parquet output.parquet --compression ZSTD --compression-level 15

# Dry run (preview SQL)
gpio add bbox input.parquet output.parquet --dry-run

Add Bbox Metadata Only¶

If your file already has a bbox column but lacks covering metadata (e.g., from external tools):

gpio add bbox-metadata myfile.parquet

import geoparquet_io as gpio

# Add bbox column, then add covering metadata
table = gpio.read('input.parquet')
table_with_bbox = table.add_bbox().add_bbox_metadata()
table_with_bbox.write('output.parquet')

# Or just add metadata if bbox column already exists
table = gpio.read('file_with_bbox.parquet')
table.add_bbox_metadata().write('output.parquet')

Declaring a column gpio will not vouch for¶

Auto-detection names two columns and no others: the one the primary geometry column's own covering.bbox already points at, and a struct called exactly bbox that no other column's covering claims. A file GDAL wrote carries its struct as geometry_bbox — exactly the right four numbers, under a name that is not evidence of what they bound. --bbox-name is where you supply that evidence:

gpio add bbox-metadata gdal_written.parquet --bbox-name geometry_bbox

import geoparquet_io as gpio

table = gpio.read('gdal_written.parquet')
table.add_bbox_metadata(bbox_column='geometry_bbox').write('output.parquet')

Naming a column is an assertion about its values, so three things stay refused. A column the file does not have. A struct the spec forbids a covering to point at (Overture's xmin, xmax, ymin, ymax) — see the field-order note above. And a struct some other geometry column already declares as its covering: those values bound that column's geometry, not the primary's, so declaring them on the primary would make readers prune rows that genuinely match. Compute the primary's own column instead with gpio add bbox.

STAC Collection Metadata¶

A stac-geoparquet file holds STAC Items as rows. The specification lets it carry the Collections those Items belong to in the file's own footer, under the stac-geoparquet key, so a reader that has the file has the Collection too and does not need a catalog to interpret it. gpio add stac-metadata writes that key:

The setup each example below runs first — copying a file to items.parquet and writing a small collection.json — lives in a doctest: comment above each fence, so the examples execute in the docs lane rather than being skipped.

# In place
gpio add stac-metadata items.parquet --collection collection.json

# Or to a copy
gpio add stac-metadata items.parquet out.parquet --collection collection.json

from geoparquet_io.api import ops

# The file form patches the footer of a file already on disk
ops.add_stac_metadata('items.parquet', 'collection.json')

import geoparquet_io as gpio

# Or set it before the file is written
table = gpio.read('items.parquet')
table.add_stac_metadata('collection.json').write('out.parquet')

The value written is:

{"version": "1.1.0", "collections": {"PSScene": {"type": "Collection", "id": "PSScene"}}}

H3 Hexagonal Cells¶

Add H3 hexagonal cell IDs based on geometry centroids:

gpio add h3 input.parquet output.parquet --resolution 9

# From HTTPS to S3
gpio add h3 https://example.com/data.parquet s3://bucket/indexed.parquet --resolution 9

import geoparquet_io as gpio

gpio.read('input.parquet').add_h3(resolution=9).write('output.parquet')

# Custom column name
gpio.read('input.parquet').add_h3(column_name='h3_index', resolution=13).write('output.parquet')

Resolution guide:

• Resolution 7: ~5 km² cells
• Resolution 9: ~105 m² cells (default)
• Resolution 11: ~2 m² cells
• Resolution 13: ~0.04 m² cells

Options:

# Custom column name
gpio add h3 input.parquet output.parquet --h3-name h3_index

# Different resolution
gpio add h3 input.parquet output.parquet --resolution 13

# With row group sizing
gpio add h3 input.parquet output.parquet --row-group-size-mb 256MB

S2 Spherical Cells¶

Add S2 spherical cell IDs based on geometry centroids. S2 cells are computed by the geography DuckDB community extension, which gpio installs on first use:

gpio add s2 input.parquet output.parquet --level 13

# From HTTPS to S3
gpio add s2 https://example.com/data.parquet s3://bucket/indexed.parquet --level 13

import geoparquet_io as gpio

gpio.read('input.parquet').add_s2(level=13).write('output.parquet')

# Custom column name
gpio.read('input.parquet').add_s2(column_name='s2_index', level=18).write('output.parquet')

S2 uses Google's Spherical Geometry library which divides the Earth's surface into a hierarchy of cells using quadtree subdivision. Unlike H3's hexagonal grid, S2 cells are variable quads that provide hierarchical spatial indexing.

Level guide:

Options:

# Custom column name
gpio add s2 input.parquet output.parquet --s2-name s2_index

# Different level
gpio add s2 input.parquet output.parquet --level 18

# With row group sizing
gpio add s2 input.parquet output.parquet --row-group-size-mb 256MB

Technical Details¶

S2 cell IDs are computed using DuckDB's geography extension:

s2_cell_token(
    s2_cell_parent(
        s2_cellfromlonlat(
            ST_X(ST_Centroid(geometry)),
            ST_Y(ST_Centroid(geometry))
        ),
        level
    )
)

• s2_cellfromlonlat: Converts lon/lat to S2 cell at maximum precision (level 30)
• s2_cell_parent: Gets parent cell at desired level
• s2_cell_token: Converts to hex token string for portability

Cell IDs are stored as hex strings (e.g., "89c25901") rather than integers for maximum portability across systems.

A5 Cells¶

Add A5 spatial cell IDs based on geometry centroids.

gpio add a5 input.parquet output.parquet --resolution 15

# From HTTPS to S3
gpio add a5 https://example.com/data.parquet s3://bucket/indexed.parquet --resolution 15

import geoparquet_io as gpio

gpio.read('input.parquet').add_a5(resolution=15).write('output.parquet')

# Custom column name
gpio.read('input.parquet').add_a5(column_name='a5_index', resolution=12).write('output.parquet')

Options:

# Custom column name
gpio add a5 input.parquet output.parquet --a5-name a5_index

# Different resolution
gpio add a5 input.parquet output.parquet --resolution 12

# With row group sizing
gpio add a5 input.parquet output.parquet --row-group-size-mb 256MB

KD-Tree Partitions¶

Add balanced spatial partition IDs using KD-tree:

# Auto-select partitions (default: ~120k rows each)
gpio add kdtree input.parquet output.parquet

# Explicit partition count (must be power of 2)
gpio add kdtree input.parquet output.parquet --partitions 32

# Exact mode (deterministic but slower)
gpio add kdtree input.parquet output.parquet --partitions 16 --exact

import geoparquet_io as gpio

# Auto mode: size the tree from the row count, like the bare CLI call
gpio.read('input.parquet').add_kdtree(auto=True).write('output.parquet')

# A different target, still auto
gpio.read('input.parquet').add_kdtree(
    auto=True, target_rows=50000
).write('output.parquet')

# Custom column name and an explicit iteration count
gpio.read('input.parquet').add_kdtree(
    column_name='partition_id',
    iterations=5  # 2^5 = 32 partitions
).write('output.parquet')

Auto mode (default): - Targets ~120k rows per partition - Uses approximate computation (O(n)) - Fast on large datasets

Explicit mode: - Specify partition count (2, 4, 8, 16, 32, ...) - Control granularity

Exact vs Approximate: - Approximate: O(n), samples 100k points - Exact: O(n × log₂(partitions)), deterministic

Options:

# Custom target rows per partition
gpio add kdtree input.parquet output.parquet --auto 200000

# Custom sample size for approximate mode
gpio add kdtree input.parquet output.parquet --approx 200000

# Track progress
gpio add kdtree input.parquet output.parquet --verbose

Geometry Metrics¶

Add geodesic area and perimeter measurements to each feature:

gpio add geometry-metrics input.parquet output.parquet

# Preview SQL without executing
gpio add geometry-metrics input.parquet output.parquet --dry-run

# Without Vecorel metadata
gpio add geometry-metrics input.parquet output.parquet --no-vecorel

import geoparquet_io as gpio
from geoparquet_io.core.add.geometry_metrics import add_geometry_metrics

# Add area and perimeter columns
add_geometry_metrics('input.parquet', 'output.parquet')

# Without Vecorel schema metadata
add_geometry_metrics('input.parquet', 'output.parquet', vecorel=False)

Calculates two columns using WGS84 spheroid-based calculations:

• metrics:area — geodesic area in square meters (m²)
• metrics:perimeter — geodesic perimeter in meters (m)

By default, the output follows the Vecorel geometry-metrics extension specification. This adds collection metadata to the Parquet file referencing the schema URL, and ensures required columns (id, geometry) are present and non-nullable. Use --no-vecorel to skip the metadata and just add the raw metric columns.

Options:

# With compression settings
gpio add geometry-metrics input.parquet output.parquet --compression ZSTD --compression-level 15

# With row group sizing
gpio add geometry-metrics input.parquet output.parquet --row-group-size-mb 256MB

# Show generated SQL
gpio add geometry-metrics input.parquet output.parquet --show-sql

Administrative Divisions¶

Add administrative division columns via spatial join with remote boundaries datasets:

How It Works¶

Performs spatial intersection between your data and remote admin boundaries to add admin division columns. Uses efficient spatial extent filtering to query only relevant boundaries from remote datasets.

Input in a non-CRS84 CRS is reprojected to the boundaries' CRS (OGC:CRS84) before the intersection, so projected data joins correctly instead of erroring on a CRS mismatch. (For a non-CRS84 input the bbox pre-filter is skipped, since the stored bbox is in the source CRS — the extent filter still bounds the query.)

For native-geometry inputs (GeoParquet 2.0 and GeoParquet 1.1 with geoarrow encoding), the spatial join relies on native Parquet column statistics to skip irrelevant data, but this is not quite working yet, so expect those to be a bit slower until #462 is implemented. For non-native inputs (GeoParquet 1.x that carry a bbox column), a cheap bbox-overlap test is applied before ST_Intersects to prune candidates, and using it will be 5-10x faster.

The join is a streaming LEFT JOIN so it scales to very large inputs (hundreds of millions of features) with bounded memory. A feature is attributed to the admin polygon(s) it intersects; because the per-level caches are non-overlapping, this is normally exactly one polygon per level. A feature straddling overlapping polygons at a border is emitted once per match. De-duplicating such border overlaps is intentionally left to a future dedicated operation rather than folded into this join, as including it greatly degraded performance.

Quick Start¶

# Add all GAUL levels (continent, country, department)
gpio add admin-divisions input.parquet output.parquet --dataset gaul

# Preview SQL before execution
gpio add admin-divisions input.parquet output.parquet --dataset gaul --dry-run

import geoparquet_io as gpio

# Add all GAUL levels (continent, country, department) -- the default,
# exactly like the CLI with no --dataset/--levels
table = gpio.read('input.parquet')
enriched = table.add_admin_divisions()
enriched.write('output.parquet')

# Or name the levels explicitly
enriched = table.add_admin_divisions(dataset='gaul', levels=['country'])
enriched.write('output.parquet')

Multi-Level Admin Divisions¶

Add multiple hierarchical administrative levels:

# Add all GAUL levels (adds admin:continent, admin:country, admin:department)
gpio add admin-divisions buildings.parquet output.parquet --dataset gaul

# Add specific levels only
gpio add admin-divisions buildings.parquet output.parquet --dataset gaul \
  --levels continent,country

# Use Overture Maps dataset
gpio add admin-divisions buildings.parquet output.parquet --dataset overture \
  --levels country,region

import geoparquet_io as gpio

# Add multiple levels
table = gpio.read('buildings.parquet')
enriched = table.add_admin_divisions(
    dataset='gaul',
    levels=['continent', 'country', 'department']
)
enriched.write('output.parquet')

# Overture dataset, pinning the pre-1.4 column prefix
enriched = table.add_admin_divisions(
    dataset='overture',
    levels=['country', 'region'],
    prefix='overture'
)

Vecorel-Compliant Output¶

Use --vecorel for output that follows the Vecorel administrative division extension:

# Vecorel-compliant admin columns (uses Overture dataset automatically)
gpio add admin-divisions input.parquet output.parquet --vecorel

# Equivalent to:
gpio add admin-divisions input.parquet output.parquet \
  --dataset overture --levels country,region --prefix admin

import geoparquet_io as gpio

table = gpio.read('input.parquet')
enriched = table.add_admin_divisions(
    dataset='overture',
    levels=['country', 'region'],
    vecorel=True
)
enriched.write('output.parquet')

The --vecorel flag:

• Forces the Overture dataset (overrides --dataset)
• Sets levels to country,region (overrides --levels)
• Adds admin:country_code (ISO 3166-1 alpha-2) and admin:subdivision_code (ISO 3166-2)
• Writes collection metadata with the Vecorel schema URL
• Ensures id and geometry columns are present and non-nullable

Datasets¶

Two remote admin boundary datasets are supported:

Vecorel Compliance (Overture Dataset Only)¶

The overture dataset follows the Vecorel administrative division extension specification with standardized ISO codes:

• admin:country_code (REQUIRED): ISO 3166-1 alpha-2 country code (e.g., "US", "AR", "DE")
• admin:subdivision_code: ISO 3166-2 subdivision code WITHOUT country prefix (e.g., "CA" not "US-CA")

The tool automatically transforms Overture's native region codes (e.g., "US-CA") to strip the country prefix for Vecorel compliance.

Note: The GAUL dataset uses FAO's standardized naming system but is NOT Vecorel compliant: - Has ISO 3166-1 alpha-3 codes (e.g., "TZA"), but Vecorel requires alpha-2 (e.g., "TZ") - Uses GAUL's standardized naming for subnational units, not ISO 3166-2 codes - Columns: admin:continent (continent name), admin:country (GAUL country name), admin:department (GAUL L2 name)

Notes¶

• Overture dataset: Vecorel compliant with ISO 3166-1 alpha-2 and ISO 3166-2 codes
• GAUL dataset: FAO standardized naming system - source.coop GAUL L2
• Performs spatial intersection to assign admin divisions based on geometry
• Requires internet connection to access remote datasets
• Uses spatial extent filtering and bbox columns for optimization

Caching¶

Admin datasets (GAUL, Overture) are automatically cached locally on first use:

• First run: Downloads and caches the full dataset (~5-50MB depending on dataset)
• Subsequent runs: Uses cached version (instant startup)
• Cache location: ~/.geoparquet-io/cache/admin/
• Warning: Shown if cache is older than 6 months

Cache management options:

# Skip cache and use remote directly
gpio add admin-divisions input.parquet output.parquet --dataset gaul --no-cache

# Clear all cached datasets (prompts for confirmation)
gpio add admin-divisions input.parquet output.parquet --dataset gaul --clear-cache

import geoparquet_io as gpio
from geoparquet_io.core.admin_datasets import get_cache_dir, clear_cache

# Default: uses cached datasets automatically
gpio.read('input.parquet').add_admin_divisions(
    dataset='overture',
    levels=['country', 'region']
).write('output.parquet')

# Check cache location (~/.geoparquet-io/cache/admin/)
cache_dir = get_cache_dir()
print(f"Cache location: {cache_dir}")

# Clear all cached admin datasets
result = clear_cache(confirm=True)
print(f"Cleared {result['files_deleted']} files, {result['bytes_freed'] / 1024 / 1024:.2f} MB freed")

Common Options¶

All add commands support:

# Compression settings
--compression [ZSTD|GZIP|BROTLI|LZ4|SNAPPY|UNCOMPRESSED]
--compression-level [1-22]

# Row group sizing
--row-group-size [exact row count]
--row-group-size-mb [target size like '256MB' or '1GB']

# Workflow options
--dry-run          # Preview SQL without executing
--verbose          # Detailed output
--preview          # Preview results (partition commands)
--hive             # Use Hive-style partitioning
--overwrite        # Overwrite existing files
--aws-profile NAME # AWS profile for S3 operations

--add-bbox         # Auto-add bbox if missing (some commands)

See Also¶

• CLI Reference - Full command reference
• partition command
• sort command