Partitioning Files¶

The partition commands split GeoParquet files into separate files based on column values or spatial indices.

Smart Analysis: All partition commands automatically analyze your strategy before execution, providing statistics and recommendations.

Auto-Resolution Mode¶

All spatial partitioning commands (H3, S2, A5, Quadkey) support automatic resolution calculation using the --auto flag. This eliminates the need to manually specify resolution levels by calculating the optimal value based on your data.

How It Works¶

Auto-resolution analyzes your dataset and calculates the optimal spatial index resolution to achieve your target partition size:

Counts total rows in your input file
Calculates how many partitions are needed to achieve --target-rows per partition
Selects the resolution that produces approximately that many partitions
Respects --max-partitions as an upper bound

Common Options¶

Option	Default	Description
`--auto`	off	Enable auto-resolution calculation
`--target-rows`	100,000	Target rows per partition
`--max-partitions`	10,000	Maximum partitions to create

Quick Examples¶

# H3 with ~100K rows per partition (default)
gpio partition h3 input.parquet output/ --auto

# Quadkey with partition limit
gpio partition quadkey input.parquet output/ --auto --max-partitions 1000

# A5 with preview
gpio partition a5 input.parquet --auto --preview

How auto-resolution is chosen¶

Auto-resolution is extent-aware: gpio probes a bounded sample of your data and counts how many cells are actually non-empty at each candidate resolution, then picks the resolution whose non-empty cell count is closest to your target partition count (total rows ÷ --target-rows, capped by --max-partitions).

This sizes partitions to where your data actually is. A globally-uniform formula (e.g. cells = 6 × 4^resolution for A5) assumes the data is spread evenly across the whole planet, which badly under-resolves regional or national datasets — collapsing them into a handful of giant partitions. If the sample probe can't run (for example, the file has no geometry column), gpio falls back to the uniform-coverage estimate.

Note: cell assignment is CRS-aware — a projected input (e.g. EPSG:5070) is reprojected to OGC:CRS84 before the grid cell is computed, and partition admin reprojects the input to the admin boundaries' CRS before the spatial join. The auto-resolution probe still samples the input's own coordinates, so its estimate is most accurate for lon/lat data.

By String Column¶

Partition by string column values or prefixes:

CLIPython

# Preview partitions
gpio partition string input.parquet --column region --preview

# Partition by full column values
gpio partition string input.parquet output/ --column category

# Partition by first 2 characters
gpio partition string input.parquet output/ --column mgrs_code --chars 2

# Hive-style partitioning
gpio partition string input.parquet output/ --column region --hive

# To cloud storage
gpio --aws-profile prod partition string s3://bucket/input.parquet s3://bucket/output/ --column region

import geoparquet_io as gpio

# Partition by full column values
gpio.read('input.parquet').partition_by_string('output/', column='category')

# Partition by first 2 characters
gpio.read('input.parquet').partition_by_string(
    'output/',
    column='mgrs_code',
    chars=2
)

# Hive-style with options
gpio.read('input.parquet').partition_by_string(
    'output/',
    column='region',
    hive=True,
    overwrite=True
)

By H3 Cells¶

Partition by H3 hexagonal cells:

CLIPython

# Auto-calculate optimal resolution for ~100K rows per partition
gpio partition h3 input.parquet output/ --auto

# Auto with custom target partition size
gpio partition h3 input.parquet output/ --auto --target-rows 50000

# Preview at resolution 7 (~5km² cells)
gpio partition h3 input.parquet --resolution 7 --preview

# Partition at specific resolution 9
gpio partition h3 input.parquet output/ --resolution 9

# Keep H3 column in output files
gpio partition h3 input.parquet output/ --resolution 9 --keep-h3-column

# Hive-style (H3 column included by default)
gpio partition h3 input.parquet output/ --resolution 8 --hive

import geoparquet_io as gpio

# Partition by H3 (flat files by default, like the CLI; pass hive=True for key=value/)
gpio.read('input.parquet').partition_by_h3('output/')

# Custom resolution
gpio.read('input.parquet').partition_by_h3('output/', resolution=7)

# With options
gpio.read('input.parquet').partition_by_h3(
    'output/',
    resolution=8,
    compression='ZSTD',
    overwrite=True
)

Column behavior:

Non-Hive: H3 column excluded by default (redundant with path)
Hive: H3 column included by default
Use --keep-h3-column to explicitly keep

If H3 column doesn't exist, it's automatically added.

Auto-Resolution for H3¶

Use --auto to let gpio calculate the optimal H3 resolution:

CLI

# Auto-select resolution for ~100k rows per partition (default)
gpio partition h3 input.parquet output/ --auto

# Target 50k rows per partition
gpio partition h3 input.parquet output/ --auto --target-rows 50000

# Limit maximum partitions created
gpio partition h3 input.parquet output/ --auto --max-partitions 5000

# Preview auto-selected partitions
gpio partition h3 input.parquet --auto --preview

CLI-Only Feature

Auto-resolution is currently CLI-only. For Python, use partition_by_h3() with an explicit resolution parameter (see examples above).

Auto-resolution probes your data's actual extent (see How auto-resolution is chosen) to pick the H3 resolution that targets your specified rows per partition, while respecting the --max-partitions constraint.

By S2 Cells¶

S2 is unavailable in this release — use gpio partition a5

S2 partitioning needs the geography DuckDB community extension, which is published only up to DuckDB 1.5.1 while gpio requires DuckDB 1.5.2 or newer, so gpio partition s2 stops immediately with an explanation instead of partitioning. Use gpio partition a5 (or gpio partition h3) instead; A5 is the closest working substitute — another hierarchical, globally-uniform cell index over the whole sphere. Do not pin duckdb==1.5.1 to get S2 back: it is below gpio's dependency floor and carries the geometry-repair segfault the floor exists to avoid. See S2 Spherical Cells for the full story. The examples below apply unchanged once the extension is republished upstream.

Partition by S2 spherical cells:

CLIPython

# Auto-calculate optimal level for ~100K rows per partition
gpio partition s2 input.parquet output/ --auto

# Auto with custom target partition size
gpio partition s2 input.parquet output/ --auto --target-rows 500000

# Preview at level 10 (~78 km² cells)
gpio partition s2 input.parquet --level 10 --preview

# Hive-style (S2 column included by default)
gpio partition s2 input.parquet output/ --auto --hive

# Partition at specific level 13 (~1.2km² cells)
gpio partition s2 input.parquet output/ --level 13

# Keep S2 column in output files
gpio partition s2 input.parquet output/ --level 12 --keep-s2-column

import geoparquet_io as gpio

# Partition by S2 (flat files by default, like the CLI; pass hive=True for key=value/)
gpio.read('input.parquet').partition_by_s2('output/')

# Custom level
gpio.read('input.parquet').partition_by_s2('output/', level=10)

# With options
gpio.read('input.parquet').partition_by_s2(
    'output/',
    level=10,
    compression='ZSTD',
    overwrite=True
)

Column behavior:

Non-Hive: S2 column excluded by default (redundant with path)
Hive: S2 column included by default
Use --keep-s2-column to explicitly keep

If S2 column doesn't exist, it's automatically added.

Auto-Resolution for S2¶

Use --auto to let gpio calculate the optimal S2 level:

CLI

# Auto-select level for ~100k rows per partition (default)
gpio partition s2 input.parquet output/ --auto

# Target 50k rows per partition
gpio partition s2 input.parquet output/ --auto --target-rows 50000

# Limit maximum partitions created
gpio partition s2 input.parquet output/ --auto --max-partitions 5000

# Preview auto-selected partitions
gpio partition s2 input.parquet --auto --preview

CLI-Only Feature

Auto-resolution is currently CLI-only. For Python, use partition_by_s2() with an explicit level parameter (see examples above).

Auto-resolution probes your data's actual extent (see How auto-resolution is chosen) to pick the S2 level that targets your specified rows per partition, while respecting the --max-partitions constraint.

By A5 Cells¶

Partition by A5 spatial cells:

CLIPython

# Auto-calculate optimal resolution for ~100K rows per partition
gpio partition a5 input.parquet output/ --auto

# Auto with custom target partition size
gpio partition a5 input.parquet output/ --auto --target-rows 500000

# Preview at resolution 10 (~41km² cells)
gpio partition a5 input.parquet --resolution 10 --preview

# Hive-style (A5 column included by default)
gpio partition a5 input.parquet output/ --auto --hive

# Partition at specific resolution 15
gpio partition a5 input.parquet output/ --resolution 15

# Keep A5 column in output files
gpio partition a5 input.parquet output/ --resolution 12 --keep-a5-column

import geoparquet_io as gpio

# Partition by A5 (flat files by default, like the CLI; pass hive=True for key=value/)
gpio.read('input.parquet').partition_by_a5('output/', resolution=12)

CLI-Only

--auto resolution selection is CLI-only; pass an explicit resolution in Python.

Column behavior:

Non-Hive: A5 column excluded by default (redundant with path)
Hive: A5 column included by default
Use --keep-a5-column to explicitly keep

If A5 column doesn't exist, it's automatically added.

Auto-Resolution for A5¶

Use --auto to let gpio calculate the optimal A5 resolution:

CLI

# Auto-select resolution for ~100k rows per partition (default)
gpio partition a5 input.parquet output/ --auto

# Target 50k rows per partition
gpio partition a5 input.parquet output/ --auto --target-rows 50000

# Limit maximum partitions created
gpio partition a5 input.parquet output/ --auto --max-partitions 5000

# Preview auto-selected partitions
gpio partition a5 input.parquet --auto --preview

CLI-Only Feature

Auto-resolution is currently CLI-only. A5 partitioning in Python is not yet available.

Auto-resolution probes your data's actual extent (see How auto-resolution is chosen) to pick the A5 resolution that targets your specified rows per partition, while respecting the --max-partitions constraint.

By Quadkey Cells¶

Partition by Bing Maps quadkey tiles:

CLIPython

# Auto-calculate optimal resolution for ~100K rows per partition
gpio partition quadkey input.parquet output/ --auto

# Auto with custom target partition size
gpio partition quadkey input.parquet output/ --auto --target-rows 500000

# Preview with auto-resolution
gpio partition quadkey input.parquet --auto --preview

# Hive-style (quadkey column included by default)
gpio partition quadkey input.parquet output/ --auto --hive

# Partition at specific resolutions (column at 13, partition at 9)
gpio partition quadkey input.parquet output/ --resolution 13 --partition-resolution 9

# Keep quadkey column in output files
gpio partition quadkey input.parquet output/ --resolution 13 --partition-resolution 9 --keep-quadkey-column

import geoparquet_io as gpio

# Partition by quadkey
gpio.read('input.parquet').partition_by_quadkey('output/')

# Custom resolution
gpio.read('input.parquet').partition_by_quadkey('output/', partition_resolution=8)

# With options
gpio.read('input.parquet').partition_by_quadkey(
    'output/',
    partition_resolution=10,
    compression='ZSTD',
    overwrite=True
)

Column behavior:

Non-Hive: Quadkey column excluded by default (redundant with path)
Hive: Quadkey column included by default
Use --keep-quadkey-column to explicitly keep

The quadkey column is created at --resolution (for full precision) but partitions are created using the first --partition-resolution characters, allowing coarser partitioning while retaining full precision in the column.

Auto-Resolution for Quadkey¶

Use --auto to let gpio calculate the optimal quadkey zoom level:

CLIPython

# Auto-select zoom level for ~100k rows per partition (default)
gpio partition quadkey input.parquet output/ --auto

# Target 50k rows per partition
gpio partition quadkey input.parquet output/ --auto --target-rows 50000

# Limit maximum partitions created
gpio partition quadkey input.parquet output/ --auto --max-partitions 5000

# Preview auto-selected partitions
gpio partition quadkey input.parquet --auto --preview

CLI-Only Feature

Auto-resolution is currently CLI-only. For Python, use partition_by_quadkey() with an explicit partition_resolution parameter (see examples above).

Auto-resolution probes your data's actual extent (see How auto-resolution is chosen) to pick the quadkey zoom level that targets your specified rows per partition, while respecting the --max-partitions constraint.

By KD-Tree¶

Partition by balanced spatial partitions:

CLIPython

# Preview auto-selected partitions
gpio partition kdtree input.parquet --preview

# Auto-partition (default: ~120k rows each)
gpio partition kdtree input.parquet output/

# Explicit partition count (must be power of 2)
gpio partition kdtree input.parquet output/ --partitions 32

# Exact computation (deterministic)
gpio partition kdtree input.parquet output/ --partitions 16 --exact

# Hive-style with progress tracking
gpio partition kdtree input.parquet output/ --hive --verbose

import geoparquet_io as gpio

# Partition using KD-tree (creates 2^iterations partitions)
gpio.read('input.parquet').partition_by_kdtree('output/')

# 64 partitions (2^6)
gpio.read('input.parquet').partition_by_kdtree('output/', iterations=6)

# With options
gpio.read('input.parquet').partition_by_kdtree(
    'output/',
    iterations=5,  # 32 partitions
    hive=True,
    overwrite=True
)

CLI vs Python API

The Python API uses iterations which creates 2^iterations partitions (power-of-two semantics). The CLI uses --partitions N to specify an absolute count directly. For example:

Python iterations=6 → 64 partitions (2^6)
CLI --partitions 64 → 64 partitions

Column behavior: - Similar to H3: excluded by default, included for Hive - Use --keep-kdtree-column to explicitly keep

If KD-tree column doesn't exist, it's automatically added.

By Admin Boundaries¶

Split by administrative boundaries via spatial join with remote datasets:

How It Works¶

This command performs two operations:

Spatial Join: Queries remote admin boundaries using spatial extent filtering, then spatially joins them with your data
Partition: Splits the enriched data by administrative levels

Quick Start¶

CLIPython

# Preview GAUL partitions by continent
gpio partition admin input.parquet --dataset gaul --levels continent --preview

# Partition by continent
gpio partition admin input.parquet output/ --dataset gaul --levels continent

# Hive-style partitioning
gpio partition admin input.parquet output/ --dataset gaul --levels continent --hive

import geoparquet_io as gpio

# Partition by country using GAUL dataset
gpio.read('input.parquet').partition_by_admin(
    'output/',
    dataset='gaul',
    levels=['country']
)

# Hive-style partitioning
gpio.read('input.parquet').partition_by_admin(
    'output/',
    dataset='gaul',
    levels=['country'],
    hive=True
)

Multi-Level Hierarchical Partitioning¶

Partition by multiple administrative levels:

CLIPython

# Hierarchical: continent → country
gpio partition admin input.parquet output/ --dataset gaul --levels continent,country

# All GAUL levels: continent → country → department
gpio partition admin input.parquet output/ --dataset gaul --levels continent,country,department

# Hive-style multi-level (creates continent=Africa/country=Kenya/department=Accra/)
gpio partition admin input.parquet output/ --dataset gaul \
    --levels continent,country,department --hive

# Overture Maps by country and region
gpio partition admin input.parquet output/ --dataset overture --levels country,region

import geoparquet_io as gpio

# Multi-level hierarchical
gpio.read('input.parquet').partition_by_admin(
    'output/',
    dataset='gaul',
    levels=['continent', 'country', 'department'],
    hive=True
)

# Using Overture Maps dataset
gpio.read('input.parquet').partition_by_admin(
    'output/',
    dataset='overture',
    levels=['country', 'region']
)

Vecorel-compliant partitions¶

Use --vecorel to write Vecorel-compliant admin columns into each partition. This forces the Overture dataset with country,region levels, names the columns admin:country_code and admin:subdivision_code, marks admin:country_code non-nullable, and writes the Vecorel collection schema metadata. Unlike the default mode, the admin columns are kept in the output files (not just encoded in the folder names).

CLIPython

# Vecorel partitions (forces Overture country,region)
gpio partition admin input.parquet output/ --vecorel

# Combine with Hive-style layout
gpio partition admin input.parquet output/ --vecorel --hive

import geoparquet_io as gpio

gpio.read('input.parquet').partition_by_admin(
    'output/',
    vecorel=True
)

Datasets¶

Two remote admin boundary datasets are supported:

Dataset	Standard	Columns Added	Description
`gaul` (default)	GAUL naming + ISO 3166-1 alpha-3	`admin:continent`, `admin:country`, `admin:department`	FAO Global Administrative Unit Layers (GAUL) L2 - worldwide coverage with standardized naming
`overture`	Vecorel compliant (ISO 3166-1/2)	`admin:country_code`, `admin:subdivision_code`	Overture Maps Divisions with ISO 3166 codes (219 countries, 3,544 regions) - docs

Vecorel Compliance (Overture Dataset Only)¶

The overture dataset follows the Vecorel administrative division extension specification with standardized ISO codes:

admin:country_code (REQUIRED): ISO 3166-1 alpha-2 country code (e.g., "US", "AR", "DE")
admin:subdivision_code: ISO 3166-2 subdivision code WITHOUT country prefix (e.g., "CA" not "US-CA")

The tool automatically transforms Overture's native region codes (e.g., "US-CA") to strip the country prefix for Vecorel compliance.

Note: The GAUL dataset uses FAO's standardized naming system but is NOT Vecorel compliant: - Has ISO 3166-1 alpha-3 codes (e.g., "TZA"), but Vecorel requires alpha-2 (e.g., "TZ") - Uses GAUL's standardized naming for subnational units, not ISO 3166-2 codes - Columns: admin:continent (continent name), admin:country (GAUL country name), admin:department (GAUL L2 name)

Notes¶

Overture dataset: Vecorel compliant with ISO 3166-1 alpha-2 and ISO 3166-2 codes
GAUL dataset: FAO standardized naming system - source.coop GAUL L2
Performs spatial intersection to assign admin divisions based on geometry
Requires internet connection to access remote datasets
Uses spatial extent filtering and bbox columns for optimization

Common Options¶

All partition commands support:

# Compression settings
--compression [ZSTD|GZIP|BROTLI|LZ4|SNAPPY|UNCOMPRESSED]
--compression-level [1-22]

# Row group sizing
--row-group-size [exact row count]
--row-group-size-mb [target size like '256MB' or '1GB']

# Workflow options
--dry-run          # Preview SQL without executing
--verbose          # Detailed output
--preview          # Preview results (partition commands)
--hive             # Use Hive-style partitioning
--overwrite        # Overwrite existing files
--aws-profile NAME # AWS profile for S3 operations

--preview-limit 15     # Number of partitions to show (default: 15)
--force                # Override analysis warnings
--skip-analysis        # Skip analysis (performance-sensitive cases)
--prefix PREFIX        # Custom filename prefix (e.g., 'fields' → fields_USA.parquet)

Output Structures¶

Standard Partitioning¶

output/
├── partition_value_1.parquet
├── partition_value_2.parquet
└── partition_value_3.parquet

Hive-Style Partitioning¶

output/
├── column=value1/
│   └── data.parquet
├── column=value2/
│   └── data.parquet
└── column=value3/
    └── data.parquet

Custom Filename Prefix¶

Add --prefix NAME to prepend a custom prefix to partition filenames:

# Standard: fields_USA.parquet, fields_Kenya.parquet
gpio partition admin input.parquet output/ --dataset gaul --levels country --prefix fields

# Hive: country=USA/fields_USA.parquet, country=Kenya/fields_Kenya.parquet
gpio partition admin input.parquet output/ --dataset gaul --levels country --prefix fields --hive

Partition Analysis¶

Before creating files, analysis shows:

Total partition count
Rows per partition (min/max/avg/median)
Distribution statistics
Recommendations and warnings

Warnings trigger for: - Very uneven distributions - Too many small partitions - Single-row partitions

Use --force to override warnings or --skip-analysis for performance.

Preview Workflow¶

With Auto-Resolution¶

# 1. Preview with auto-resolution
gpio partition h3 large.parquet --auto --preview

# 2. Adjust target rows if needed
gpio partition h3 large.parquet --auto --target-rows 50000 --preview

# 3. Execute when satisfied
gpio partition h3 large.parquet output/ --auto --target-rows 50000

With Manual Resolution¶

# 1. Preview to understand partitioning
gpio partition h3 large.parquet --resolution 7 --preview

# 2. Adjust resolution if needed
gpio partition h3 large.parquet --resolution 8 --preview

# 3. Execute when satisfied
gpio partition h3 large.parquet output/ --resolution 8

Sub-Partitioning Large Files¶

After partitioning by admin boundaries or string columns, some files may still be too large. Use --min-size with directory input to sub-partition only the oversized files:

# Sub-partition files >100MB with H3
gpio partition h3 by_country/ --min-size 100MB --resolution 7 --in-place

See Sub-Partitioning Large Files for details.

Partitioning Files¶

Auto-Resolution Mode¶

How It Works¶

Common Options¶

Quick Examples¶

How auto-resolution is chosen¶

By String Column¶

By H3 Cells¶

Auto-Resolution for H3¶

By S2 Cells¶

Auto-Resolution for S2¶

By A5 Cells¶

Auto-Resolution for A5¶

By Quadkey Cells¶

Auto-Resolution for Quadkey¶

By KD-Tree¶

By Admin Boundaries¶

How It Works¶

Quick Start¶

Multi-Level Hierarchical Partitioning¶

Vecorel-compliant partitions¶

Datasets¶

Vecorel Compliance (Overture Dataset Only)¶

Notes¶

Common Options¶

Output Structures¶

Standard Partitioning¶

Hive-Style Partitioning¶

Custom Filename Prefix¶

Partition Analysis¶

Preview Workflow¶

With Auto-Resolution¶

With Manual Resolution¶

Sub-Partitioning Large Files¶

See Also¶