Converting Between Formats¶
The convert command transforms between GeoParquet and other vector formats with automatic format detection and optimization.
CLI vs Python Behavior
The CLI gpio convert applies Hilbert sorting by default for optimal spatial queries.
The Python gpio.convert() does NOT sort by default - chain .sort_hilbert() explicitly if needed.
Basic Usage¶
gpio convert input.shp output.parquet
Automatically applies:
- ZSTD compression (level 15)
- Bbox column with proper metadata
- Hilbert spatial ordering
- GeoParquet metadata (version auto-detected — see GeoParquet Version; 1.1 for non-GeoParquet inputs)
- 49,152-row row groups, the same default every other gpio write uses
Row groups used to be left to the Parquet writer's own default — 122,880
rows for DuckDB-backed writes — which is outside the 10,000-50,000 band
gpio check optimization scores, so convert produced files its own
spatial check marked [fail]
(#981). Pass
--row-group-size to choose your own; anything above one 2,048-row writer
vector is snapped to a whole vector the same way the sort commands snap it,
so the number you ask for is the number that lands.
A request of 2,048 rows or fewer is the exception: gpio passes it
through untouched, because pyarrow honours it exactly while DuckDB's
COPY rounds it up to 2,048 regardless — so what lands depends on which
writer runs, and gpio cannot snap it without overriding the one writer that
could have obeyed. convert writes through DuckDB, so
--row-group-size 249 really does produce 2,048-row groups; it now prints
one line naming both numbers before the footer does
(#986).
import geoparquet_io as gpio
# Convert with Hilbert sorting (recommended)
gpio.convert('input.shp').sort_hilbert().write('output.parquet')
# Or without sorting (faster but less optimal for spatial queries)
gpio.convert('input.shp').write('output.parquet')
Supported Formats¶
Input Formats (to GeoParquet)¶
Auto-detected by file extension:
- Shapefile (.shp)
- GeoJSON (.geojson, .json)
- GeoPackage (.gpkg)
- FlatGeobuf (.fgb)
- File Geodatabase (.gdb)
- CSV/TSV (.csv, .tsv, .txt) - See CSV/TSV Support below
Any format supported by DuckDB's spatial extension (50+ formats) can be read.
A .json file has to be GeoJSON. A bare array of records, or records wrapped
in an object such as {"data": [...]}, is not a vector dataset GDAL can open,
so gpio convert stops with Failed to read input file and gpio.convert()
raises a GeoParquetError. DuckDB 1.5.5 crashed on such files instead, which
is why gpio requires DuckDB 1.5.6 or later
(#1102).
The table gpio.convert() returns describes its geometry column and CRS the
same way a table from gpio.read() does, so every add_* method works on it
directly. A projected input such as EPSG:6933 now gets the same H3, S2, A5,
quadkey and KD-tree cells whether it is converted or read back from GeoParquet
(#1222).
Output Formats (from GeoParquet)¶
Auto-detected from output file extension:
- GeoParquet (.parquet) - Optimized cloud-native format
- GeoPackage (.gpkg) - SQLite-based OGC standard
- FlatGeobuf (.fgb) - Cloud-native streaming format
- CSV (.csv) - Tabular with WKT geometry
- Shapefile (.shp) - Legacy ESRI format
- GeoJSON (.geojson, .json) - Web-friendly JSON format
Converting FROM GeoParquet¶
Convert GeoParquet to other formats with automatic format detection:
# Auto-detects format from extension
gpio convert data.parquet output.gpkg # → GeoPackage
gpio convert data.parquet output.fgb # → FlatGeobuf
gpio convert data.parquet output.csv # → CSV with WKT
gpio convert data.parquet output.shp # → Shapefile
gpio convert data.parquet output.geojson # → GeoJSON
# Use explicit subcommand
gpio convert geopackage data.parquet output.gpkg
gpio convert flatgeobuf data.parquet output.fgb
gpio convert csv data.parquet output.csv
gpio convert shapefile data.parquet output.shp
gpio convert geojson data.parquet output.geojson
import geoparquet_io as gpio
# Load and convert
table = gpio.read('data.parquet')
# Auto-detects from extension
table.write('output.gpkg') # → GeoPackage
table.write('output.fgb') # → FlatGeobuf
table.write('output.csv') # → CSV with WKT
table.write('output.shp') # → Shapefile
table.write('output.geojson') # → GeoJSON
# Or use explicit format
table.write('output.dat', format='csv')
Format-Specific Options¶
GeoPackage:
# Custom layer name
gpio convert data.parquet output.gpkg --layer-name buildings
# Overwrite existing
gpio convert data.parquet output.gpkg --overwrite
Shapefile:
# Custom encoding (default: UTF-8)
gpio convert data.parquet output.shp --encoding ISO-8859-1
# Overwrite existing
gpio convert data.parquet output.shp --overwrite
Shapefile Limitations
- Column names truncated to 10 characters
- File size limit of 2GB
- Limited data type support
- Creates multiple files (.shp, .shx, .dbf, .prj)
- Consider using GeoPackage or FlatGeobuf instead
Remote Shapefile Storage
When writing shapefiles to remote storage (S3, GCS, Azure), all sidecar files (.shp, .shx, .dbf, .prj, etc.) are automatically packaged into a single .shp.zip archive before upload. This ensures atomic uploads and avoids incomplete multi-file uploads.
# Local: Creates output.shp, output.shx, output.dbf, etc.
gpio convert data.parquet output.shp
# Remote: Uploads output.shp.zip containing all files
gpio convert data.parquet s3://bucket/output.shp
# → Creates s3://bucket/output.shp.zip
CSV:
# Include WKT geometry (default)
gpio convert data.parquet output.csv
# Exclude geometry
gpio convert data.parquet output.csv --no-wkt
# Exclude bbox column
gpio convert data.parquet output.csv --no-bbox
GeoJSON:
# Custom precision (default: 7)
gpio convert data.parquet output.geojson --precision 5
# Pretty-print JSON
gpio convert data.parquet output.geojson --pretty
# Include bbox for each feature
gpio convert data.parquet output.geojson --write-bbox
# Use specific field as feature ID
gpio convert data.parquet with-ids.geojson --id-field osm_id
Cloud Output Support¶
All formats support cloud destinations via upload:
# Write local then upload
gpio convert data.parquet local.gpkg
gpio publish upload local.gpkg s3://bucket/output.gpkg
import geoparquet_io as gpio
# Write locally first
table = gpio.read('data.parquet')
table.write('local.gpkg')
# Upload to cloud
gpio.upload('local.gpkg', 's3://bucket/output.gpkg')
Multi-Layer Formats¶
GeoPackage and FileGDB files can contain multiple layers. By default, the first layer is read. Use --layer to select a specific layer.
# Read specific layer from GeoPackage
gpio convert geoparquet multilayer.gpkg buildings.parquet --layer buildings
# Read specific layer from FileGDB
gpio convert geoparquet data.gdb roads.parquet --layer roads
# Without --layer, reads the first/default layer
gpio convert geoparquet data.gpkg output.parquet
import geoparquet_io as gpio
# Read specific layer
gpio.convert('multilayer.gpkg', layer='buildings').write('buildings.parquet')
gpio.convert('multilayer.gpkg', layer='roads').write('roads.parquet')
# Read first layer (default)
gpio.convert('multilayer.gpkg').write('output.parquet')
Invalid Layer Names
Due to an upstream bug in DuckDB's spatial extension, specifying a non-existent layer name may cause a crash instead of raising an error. Ensure layer names are valid before conversion. You can inspect available layers using tools like ogrinfo:
ogrinfo multilayer.gpkg
GeoParquet-to-GeoParquet Conversion¶
When converting between GeoParquet files, special handling is applied to preserve GeoParquet-specific features.
Multiple Geometry Columns¶
GeoParquet files can have multiple geometry columns (e.g., geometry for point locations and boundary for polygon boundaries). When converting, all geometry columns are preserved with their:
- Original column names
- CRS (coordinate reference system)
- Encoding (WKB)
- Geometry types
# Both geometry columns preserved
gpio convert input_multi_geom.parquet output.parquet
import geoparquet_io as gpio
# Both geometry columns preserved automatically
gpio.convert('input_multi_geom.parquet').write('output.parquet')
# With Hilbert sorting (uses primary geometry column)
gpio.convert('input_multi_geom.parquet').sort_hilbert().write('output.parquet')
Bbox and Hilbert Ordering
Bbox computation and Hilbert spatial ordering use the primary geometry column only. Secondary geometry columns are preserved but do not influence spatial indexing.
When the covering metadata is written
A covering entry tells readers that a bbox column's values bound the
geometry, so engines prune on it. gpio convert declares one only when it
can stand behind the claim:
- it computed the bbox column itself, from the geometry, during this convert
- the primary geometry column's own metadata already declared a covering for the column being preserved
- the output carries a conventional
bboxstruct column (the shape every GeoParquet 1.0 writer emitted, beforecoveringexisted) — this is what makes a 1.0 → 1.1 upgrade declare it
The same two pieces of evidence decide which column convert treats as the
input's bbox in the first place. A struct column with any other name —
bounds, parcel_extent, tile_bounds, or the boundary_bbox that
belongs to a secondary geometry column — is carried through as ordinary
data: it is not declared, it is not dropped for a native-geometry output,
and a 1.x output gets convert's own computed bbox beside it. Its name is
not evidence that its values bound the primary geometry, and a wrong
covering makes readers skip rows that genuinely match. Declare such a column
deliberately with
gpio add bbox-metadata --bbox-name <column>, which is you
supplying the evidence gpio cannot read off a schema. Note that
gpio check bbox does not point this out: it reports on the primary's bbox
column, so an undeclared bounds reads there as no bbox column at all.
If the input already has a column named bbox that is not a bbox struct
(a string tile id, say), it is kept as it is, and the computed bbox is
written under the next free name (bbox_1, bbox_2, …) with the covering
declared over that column; a warning names both.
When the input's CRS is not valid PROJJSON
GeoParquet requires a column's crs to be a PROJJSON object, which means it
must carry a type member such as "GeographicCRS" or "ProjectedCRS".
gpio convert writes the input's CRS into the output, so it checks that CRS
before writing rather than passing a defect on:
- a CRS missing only
type, but carrying anidsuch as{"authority": "EPSG", "code": 3857}, is repaired — the identifier names the CRS unambiguously, so gpio rebuilds the full PROJJSON from it and says so - anything else — no usable identifier, an identifier no CRS database
knows, or a
typethat is not a PROJJSON CRS type — fails the conversion with an error naming the file and the CRS
The result is either a valid file or a clear error, never a file that
gpio check spec would reject. Fix such a CRS in the source data, or
re-export it from a tool that writes valid PROJJSON.
An explicit default CRS is normalized, not preserved
GeoParquet writes its default CRS (OGC:CRS84, equivalently EPSG:4326) by
omitting the crs key, so an input that spells that default out comes back
without the key. The coordinates and their meaning are unchanged, but a
byte-for-byte diff of the geo metadata will show the key gone. convert
rebuilds the geo block from the converted data, so this applies at every
--geoparquet-version. Run with --verbose to see a note when it happens.
Custom Geometry Column Names¶
GeoParquet files can use non-standard geometry column names (e.g., the_geom, my_geometry). The CLI preserves them during conversion:
# Input has primary_column: "the_geom"
# Output preserves "the_geom" (not renamed to "geometry")
gpio convert input.parquet output.parquet
gpio.convert()'s geometry_column is a request for the output name, not a
way to point at a source column — the source's geometry column is always
detected. The returned Table carries the name actually used, so read it off
table.geometry_column rather than assuming the request was granted
(#1176):
| Source | geometry_column='geom' gives |
|---|---|
| CSV, or Parquet with WKT / lat-lon columns | geom — the geometry is built here, so it can take the name. A column the source carries through that already has that name keeps it, and the built one moves aside to geom_1 with a warning. |
| GeoPackage, Shapefile, GeoJSON, FlatGeobuf, FileGDB, GeoParquet | geometry — the read aliases ST_AsWKB(...) AS geometry, so the request cannot be honoured and the actual name is reported instead. This is also why gpio.convert() on a the_geom GeoParquet hands back geometry, where the CLI above keeps the name. |
import geoparquet_io as gpio
# Honoured: the geometry is built from lat/lon, so it can take the name
table = gpio.convert('pts.csv', lat_column='lat', lon_column='lon',
geometry_column='geom')
table.geometry_column # 'geom'
# Not honoured: a GeoPackage's geometry is read, and reports back 'geometry'
table = gpio.convert('data.gpkg', geometry_column='geom')
table.geometry_column # 'geometry'
Edges Metadata Preservation¶
Non-planar edges metadata ("edges": "spherical", e.g. from BigQuery
GEOGRAPHY extracts) survives every rewrite: convert, extract, sort,
convert reproject, and partition all carry it through to the output —
including remote (S3/GCS/Azure) outputs.
The one exception is reprojecting into a projected CRS. gpio reprojects
vertices, not edges: it does not densify along great circles first, so the
output's edges are straight lines in the destination CRS and the declaration is
dropped (with a warning) rather than carried onto data it no longer describes —
planar, the spec default, is what the output actually is. Densify before
reprojecting if great-circle edges must be preserved. Reprojecting between two
geographic CRSs (a datum shift, e.g. EPSG:4326 → EPSG:4269) keeps the
declaration unchanged. gpio check spec warns about files that carry
edges: spherical on a projected CRS.
# Input with edges: "spherical" keeps it in the output
gpio convert geography.parquet output.parquet
gpio sort hilbert geography.parquet sorted.parquet
import geoparquet_io as gpio
# edges metadata is preserved through the pipeline
gpio.read('geography.parquet').sort_hilbert().write('output.parquet')
When a GeoParquet 2.0 input declares an ellipsoidal edges algorithm
(vincenty, karney, andoyer, thomas) and the output is 1.x, the
algorithm is mapped to "spherical" — the only non-planar value 1.x supports
— with a warning. 2.0 outputs keep the algorithm verbatim.
Z and M Dimensions¶
Geometries with Z (elevation) and/or M (measure) values are preserved, and the
written geometry_types metadata carries the spec's dimension suffixes
("Point Z", "LineString ZM"). gpio check spec validates these suffixes
against the actual coordinate dimensions in both directions.
--force-2d drops Z and M instead (ST_Force2D), for sources that are 3D
throughout when the consumer is not: some tile renderers draw nothing for 3D
geometry. It applies to every input format, including a WKT column in CSV and
every geometry column of a GeoParquet input, and the bounds, bbox column and
Hilbert ordering are all computed from the flattened geometry. The written CRS
follows: a compound "horizontal + height" CRS (EST97 + EVRF2007 height, say) is
reduced to its horizontal component, and a 3D geographic CRS such as EPSG:4979
to its 2D counterpart, so the file does not describe a dimension it no longer
has. Without --force-2d a compound CRS is kept as declared.
gpio convert etak_3d.shp etak.parquet --force-2d
import geoparquet_io as gpio
gpio.convert("etak_3d.shp", force_2d=True).write("etak.parquet")
Remote Files¶
Read from cloud storage or HTTPS:
# Convert remote file
gpio convert https://example.com/data.geojson local.parquet
# Convert from S3
gpio convert s3://bucket/input.parquet local-optimized.parquet
# Convert remote to local format
gpio convert s3://bucket/data.parquet local.gpkg
A URL is used exactly as you paste it — gpio takes it as already percent-encoded and never re-encodes it. See Remote Files Guide for authentication setup.
Options¶
Geometry Repair¶
Invalid geometry (self-intersections, unclosed rings — common in government and
municipal data) is repaired automatically with ST_MakeValid and a warning
reports how many features were fixed.
Repair is on by default across the pipeline (convert, extract wfs,
extract arcgis, extract bigquery, extract carto, extract geoparquet,
convert geojson, and pmtiles create). It prevents downstream failures such
as tippecanoe TopologyException crashes during PMTiles generation.
To preserve invalid geometry exactly (e.g. for round-tripping), opt out — gpio still counts and warns about the invalid features it left untouched.
# Default: repairs and warns ("Repaired 3 invalid geometries")
gpio convert cordoba.gpkg output.parquet
# Opt out: preserves invalid geometry, still warns
# ("Left unrepaired 3 invalid geometries (--no-repair-geometry)")
gpio convert cordoba.gpkg output.parquet --no-repair-geometry
import geoparquet_io as gpio
# Default: repairs invalid geometry
gpio.convert("cordoba.gpkg").write("output.parquet")
# Opt out: preserve invalid geometry exactly
gpio.convert("cordoba.gpkg", repair_geometry=False).write("output.parquet")
Only invalid geometry is touched: valid geometry is byte-identical, NULL is
preserved, and bounding boxes stay correct (ST_MakeValid never expands an
envelope).
Curved Geometries¶
GeoParquet cannot represent curved geometry types (CircularString,
CompoundCurve, CurvePolygon, MultiCurve, MultiSurface — common in GeoPackage
and FileGDB exports from CAD or ArcGIS), so by default gpio strokes arcs into
line segments on read, the same operation as ogr2ogr -nlt CONVERT_TO_LINEAR
or PostGIS ST_CurveToLine. A warning reports how many features were
linearized.
Note this alters the geometry: arcs become chains of straight segments,
sampled every 4° of arc by default (GDAL's default; ~0.08% area error on a
full circle). Earlier gpio versions failed on curved input — opt out with
--no-linearize-curves to keep that strict behavior.
# Default: linearizes and warns ("Linearized 66 curved geometries ...")
gpio convert curved.gpkg output.parquet
# Denser sampling: 1 degree per segment
gpio convert curved.gpkg output.parquet --max-angle-deg 1
# Opt out: curved input fails with an actionable error
gpio convert curved.gpkg output.parquet --no-linearize-curves
import geoparquet_io as gpio
# Default: linearizes curved geometries
gpio.convert("curved.gpkg").write("output.parquet")
# Denser sampling: 1 degree per segment
gpio.convert("curved.gpkg", max_angle_deg=1.0).write("output.parquet")
# Opt out: raise on curved input instead
gpio.convert("curved.gpkg", linearize_curves=False)
Linear geometries pass through untouched, NULL is preserved, and empty curves become their empty linear counterpart. The surface family (PolyhedralSurface, TIN, Triangle) is not linearized and still raises an error.
Curved input is spotted either by a header scan of local GeoPackages or on the
first pass that parses geometry. --skip-hilbert removes that pass, so for a
curved source which is not a local .gpkg (FileGDB, a GeoPackage on S3) the
write is where the curves first show up. gpio then linearizes the source and
converts again. That costs one extra read of the source, and only when it holds
geometry DuckDB cannot read; a file already at the output path is overwritten
by the second attempt or, if that fails too, left as it was.
--no-linearize-curves keeps the error instead.
Source Text Encoding¶
Every string gpio writes is valid UTF-8, or the conversion stops. A shapefile attribute table (DBF) is often in a Windows code page instead, such as Shift_JIS (CP932) for Japanese data, GBK (CP936) for Chinese, CP1251 for Cyrillic or CP1252 for Western European text.
For a shapefile, gpio finds the encoding the way GDAL does. It takes
--encoding first, then the .cpg sidecar, then the language driver id in the
DBF header. The GDAL that DuckDB bundles can only recode ISO-8859-1 by itself,
so for any other code page gpio reads the raw bytes and decodes them with
Python's codec of that name. Values and field names both come out as the
original text. A character cut in half at a DBF field's width becomes U+FFFD,
with a warning.
When nothing names the encoding and a string is not UTF-8, the conversion
fails. The error names the column and asks for --encoding. Nothing is
written, so a file no reader can open is never reported as a success.
# Japanese shapefile with no .cpg sidecar
gpio convert tokyo.shp tokyo.parquet --encoding CP932
# Western European shapefile with no .cpg sidecar
gpio convert ehak.shp ehak.parquet --encoding ISO-8859-1
# Latin-1 CSV with a WKT column
gpio convert points.csv points.parquet --encoding ISO-8859-1
import geoparquet_io as gpio
gpio.convert("tokyo.shp", encoding="CP932").write("tokyo.parquet")
gpio.convert("ehak.shp", encoding="ISO-8859-1").write("ehak.parquet")
gpio.convert("points.csv", encoding="ISO-8859-1").write("points.parquet")
A shapefile with a .cpg or a language driver id needs no option. Names follow
Python's codec registry, so CP932, Shift_JIS, GBK, CP936, CP1251,
CP1252 and latin1 all work. Shift_JIS and GB2312 are read as their
Windows supersets CP932 and GBK, which is what files labeled that way hold.
gpio does not fetch the .cpg of a remote shapefile, so pass --encoding for
one that is not UTF-8. For other GDAL formats the name is passed to GDAL as its
ENCODING open option.
The CSV reader decodes UTF-8, UTF-16 and Latin-1 on its own, and the common
names are translated for it (ISO-8859-1 and latin1 both reach it as
Latin-1). Every other encoding, CP1252 included, comes from DuckDB's
encodings extension, which gpio loads on demand. Offline, without a cached
copy, the error names the extension. Parquet carries UTF-8 already, so the
option is refused there.
Skip Hilbert Ordering¶
For faster conversion when spatial ordering isn't critical:
gpio convert large.gpkg output.parquet --skip-hilbert
Trade-off: Faster conversion but less optimal for spatial queries.
Memory on Very Large Inputs¶
Hilbert ordering sorts the whole input, so a conversion larger than memory
spills the sort to disk under TMPDIR. gpio caps DuckDB's write at half the
memory ceiling it detects — physical RAM, or the container or batch-job (Slurm)
cgroup limit when that is lower — leaving room for the memory DuckDB allocates
outside its limit. Set the cap yourself with --write-memory:
gpio convert large.gpkg output.parquet --geoparquet-version 2.0 --write-memory 8GB
If DuckDB reports Out of Memory Error rather than spilling, raise the limit
or point TMPDIR at a volume with room for the spill. See
Write Strategies.
Custom Compression¶
Control compression type and level:
# GZIP compression
gpio convert input.shp output.parquet --compression GZIP --compression-level 6
# Uncompressed (not recommended)
gpio convert input.geojson output.parquet --compression UNCOMPRESSED
Available compression types:
- ZSTD (default, level 15) - Best compression + speed balance
- GZIP (level 1-9) - Wide compatibility
- BROTLI (level 1-11) - High compression
- LZ4 - Fastest decompression
- SNAPPY - Fast compression
- UNCOMPRESSED - No compression
GeoParquet Version¶
Control the GeoParquet encoding version written to output.
Auto-detection (default): when --geoparquet-version is not specified, the
output version is resolved from the input:
- GeoParquet 2.0 input → stays 2.0 (no silent downgrade to 1.1)
- Bare native geo types (Parquet
GEOMETRY/GEOGRAPHYcolumns withoutgeometadata) → upgraded to 2.0 - GeoParquet 1.x input → written as 1.1
- Non-GeoParquet input (Shapefile, GeoJSON, ...) → written as 1.1
Auto-detection applies to both gpio convert and gpio convert reproject. An
explicit --geoparquet-version always wins. The Python API resolves the
version the same way, so gpio.read('native.parquet').write('out.parquet')
writes true 2.0 native output just like the CLI.
The "bare native geo types" rule holds for an in-memory Arrow table too: a table
with no geo metadata whose geometry field declares a GeoArrow extension type —
whether PyArrow resolved that type or it arrives as an ARROW:extension:name
field-metadata key — is the same shape as a native-geo file, and auto mode
writes it as 2.0. A geo block that declares a version still wins over both.
# Auto mode: a 2.0 input stays 2.0
gpio convert input_v2.parquet output.parquet
# Auto mode: reproject preserves the input version too
gpio convert reproject input_v2.parquet output.parquet --dst-crs EPSG:3857
# Explicit version always wins
gpio convert input_v2.parquet output.parquet --geoparquet-version 1.1
import geoparquet_io as gpio
# Auto mode: a 2.0 or native-geo input stays/becomes 2.0
gpio.read('input_v2.parquet').write('output.parquet')
# Explicit version always wins
gpio.read('input_v2.parquet').write(
'output.parquet', geoparquet_version='1.1'
)
Explicit versions:
# GeoParquet 1.1 with native GeoArrow nested-coordinate encoding
# (no bbox column; incompatible mixed-geometry columns fall back to WKB)
gpio convert input.geojson output.parquet --geoparquet-version 1.1-geoarrow
# GeoParquet 1.0 with WKB encoding
gpio convert input.shp output.parquet --geoparquet-version 1.0
# GeoParquet 1.1 with WKB encoding (default)
gpio convert input.shp output.parquet --geoparquet-version 1.1
import geoparquet_io as gpio
# GeoParquet 1.1 with native GeoArrow nested-coordinate encoding
# Converts geometry from any input (GeoJSON, Shapefile, GeoPackage, CSV, WKB GeoParquet)
# to native GeoArrow types. No bbox column is added.
# Columns mixing incompatible geometry types fall back to WKB.
gpio.convert('input.geojson').write(
'output.parquet', geoparquet_version='1.1-geoarrow'
)
# GeoParquet 1.0 with WKB encoding
gpio.convert('input.shp').write(
'output.parquet', geoparquet_version='1.0'
)
Available versions:
- 1.0 — GeoParquet 1.0 with WKB encoding
- 1.1 — GeoParquet 1.1 with WKB encoding (default for 1.x and non-GeoParquet inputs)
- 1.1-geoarrow — GeoParquet 1.1 with native GeoArrow (nested-coordinate) encoding; no bbox
column; compatible geometry type mixes are promoted (e.g. Polygon + MultiPolygon →
MultiPolygon); incompatible mixes fall back to WKB
- 2.0 — GeoParquet 2.0 with native Parquet geo types
- parquet-geo-only — Native Parquet geo types without GeoParquet metadata
Verbose Output¶
Track progress and see detailed information:
gpio convert input.gpkg output.parquet --verbose
Shows: - Geometry column detection - Dataset bounds calculation - Bbox column creation - Hilbert ordering progress - File size and validation
Examples¶
Basic Shapefile Conversion¶
gpio convert buildings.shp buildings.parquet
Output:
Converting buildings.shp...
Done in 2.3s
Output: buildings.parquet (4.2 MB)
✓ Output passes GeoParquet validation
import geoparquet_io as gpio
gpio.convert('buildings.shp').sort_hilbert().write('buildings.parquet')
Large Dataset Without Hilbert¶
gpio convert large_dataset.gpkg output.parquet --skip-hilbert
import geoparquet_io as gpio
# Python doesn't sort by default, so just skip sort_hilbert()
gpio.convert('large_dataset.gpkg').write('output.parquet')
Skips Hilbert ordering for faster processing on large files.
Custom Compression Settings¶
gpio convert roads.geojson roads.parquet \
--compression ZSTD \
--compression-level 22 \
--verbose
Maximum ZSTD compression with progress tracking.
Convert and Inspect¶
# Convert
gpio convert input.shp output.parquet
# Verify
gpio inspect output.parquet
# Validate
gpio check all output.parquet
Tabular Sources: CSV, TSV and Parquet Without Geometry¶
A source whose coordinates sit in ordinary columns — a CSV, or a Parquet file
that carries no geometry column at all — gets its geometry built from those
columns. Scientific archives are full of the second kind: a table of
observations with LAT and LON columns and no geo metadata, which used to
fail with "No geometry column detected".
# Auto-detect WKT or lat/lon, CSV or Parquet alike
gpio convert points.csv points.parquet
# Name the columns when auto-detection cannot
gpio convert data.csv out.parquet --wkt-column geom_wkt
gpio convert data.csv out.parquet --lat-column lat --lon-column lng
# Same flags, Parquet input
gpio convert observations.parquet out.parquet --lat-column LAT --lon-column LON
# Custom delimiter (CSV/TSV only)
gpio convert data.txt out.parquet --delimiter "|"
import geoparquet_io as gpio
gpio.convert("points.csv").sort_hilbert().write("points.parquet")
gpio.convert("data.csv", wkt_column="geom_wkt").write("out.parquet")
gpio.convert("data.csv", lat_column="lat", lon_column="lng").write("out.parquet")
gpio.convert(
"observations.parquet", lat_column="LAT", lon_column="LON"
).write("out.parquet")
The lat/lon columns are replaced by a single geometry column; every other
column is carried through. A row missing a coordinate keeps its attributes and
gets NULL geometry.
What Auto-Detection Looks For¶
WKT first, then a lat/lon pair:
| Step | Matches |
|---|---|
| WKT column | wkt, geometry, geom, the_geom, shape — the value is parsed before it is believed |
| Exact lat/lon names | lat/lon, latitude/longitude, lng, long, and y/x |
| Matching numeric pair | Two numeric columns alike but for the axis word: decimalLatitude/decimalLongitude, lat_dd/lon_dd, LATITUDE_WGS84/LONGITUDE_WGS84, location.lat/location.lon |
Words are matched whole, so a column like GCLONG01 is not mistaken for a
longitude. When several matching pairs exist — pickup_latitude/
pickup_longitude beside dropoff_latitude/dropoff_longitude — the first is
used and the others are named in a warning; pass --lat-column/--lon-column
to choose. A Parquet file that already has a geometry column is never
reinterpreted: those flags are refused for it, and auto-detection does not run.
A mistyped name is answered with the nearest column names rather than a bare failure:
Error: column 'LONG' not found in input. Did you mean 'LON'?
Available columns: OBS_TIMESTAMP, MSG_TYPE, ... (+161 more)
CRS and Validation¶
Default: WGS84 (EPSG:4326). Override with --crs for WKT data — in a CSV or
in a Parquet file whose geometry gpio is building from its columns:
gpio convert projected.csv out.parquet --crs EPSG:3857
--crs is still refused for a source that states its own CRS (a GeoPackage,
a shapefile, a Parquet file with a geo block).
Validates lat/lon ranges (-90 to 90, -180 to 180). Warns on large coordinates suggesting projected CRS.
Invalid Geometries¶
Fails on invalid WKT by default. Skip the unparsable rows with
--skip-invalid:
gpio convert messy.csv out.parquet --skip-invalid
A row whose geometry column is empty is not invalid — it is a row without geometry. Those rows are kept either way, with NULL geometry, so their attributes survive the conversion; they sort after the ordered rows, as they do for every other input format.
Skips rows whose geometry does not parse; the output is still Hilbert-ordered
(add --skip-hilbert to keep the input order). Mixed geometry types supported.
Delimiters¶
Auto-detects comma and tab. Override with --delimiter for semicolon, pipe, or any single character.
gpio convert data.csv out.parquet --delimiter ";"
Columns Whose Names Differ Only by Case¶
Some sources carry two columns whose names differ only by case -- most often a
GeoServer WFS response, where the driver materialises the feature-level id
member beside the publisher's own Id. DuckDB identifiers are
case-insensitive, so such a file cannot be read as it stands.
convert keeps both columns: the first spelling it meets keeps its name, and
each later collision is suffixed (Id becomes Id_1). Parquet field names are
case-sensitive, so nothing is dropped, and every rename is logged. Sources
without a collision are unaffected.
gpio convert geoparquet counties.geojson counties.parquet
# Renamed column "Id" to "Id_1": it collides with an earlier column
# under DuckDB's case-insensitive identifiers
import geoparquet_io as gpio
# columns land as 'id' and 'Id_1'
gpio.convert('counties.geojson').write('counties.parquet')
Performance¶
The convert command uses DuckDB's spatial extension - the fastest option for GeoParquet conversion, especially for large files.
Benchmarks on representative datasets:
| Dataset | Size | Features | DuckDB | PyOGRIO | ogr2ogr | Fiona |
|---|---|---|---|---|---|---|
| GAUL L2 Shapefile | 739 MB | 45k | 4.6s | 5.9s | 4.1s | 187s |
| Argentina Roads | 1.1 GB | 3.5M | 30s | 66s | 117s | 349s |
DuckDB also uses significantly less memory than alternatives (near-zero vs 600MB-2GB for GeoPandas).
To run your own benchmarks:
gpio benchmark compare input.geojson --iterations 3
See gpio benchmark for details.
See Also¶
- CLI Reference: convert
- benchmark command - Compare conversion performance
- add command - Add indices to existing GeoParquet
- sort command - Sort existing GeoParquet spatially
- check command - Validate and fix GeoParquet files