Write Strategies for Large Files¶

gpio handles larger-than-memory GeoParquet files efficiently through a pluggable write strategy system. The default strategy streams data directly to disk with constant memory usage, allowing you to process files of any size.

The Default: DuckDB Streaming¶

gpio uses the duckdb-kv strategy by default. This strategy:

• Uses O(1) constant memory regardless of file size
• Streams data through DuckDB's native COPY TO command
• Embeds GeoParquet metadata directly in the Parquet footer
• Handles files of any size without running out of memory
• Is container-aware, automatically detecting Docker/Kubernetes memory limits

For most users, the default just works. No configuration is needed:

# Process a 50GB file on a machine with 4GB RAM
gpio extract huge_dataset.parquet filtered.parquet --bbox -122.5,37.5,-122.0,38.0

# Convert a massive shapefile to GeoParquet
gpio convert large_file.shp output.parquet

import geoparquet_io as gpio

# Process large files with the fluent API
gpio.read('huge_dataset.parquet') \
    .extract(bbox=(-122.5, 37.5, -122.0, 38.0)) \
    .write('filtered.parquet')

Strategy Comparison¶

Strategy Details¶

duckdb-kv (Default)

The recommended strategy for all production workloads. Uses DuckDB's native COPY TO with KV_METADATA option for a single atomic write operation with no post-processing.

streaming

Uses PyArrow's streaming writer to process data in batches. Good alternative if you encounter issues with the DuckDB strategy. Memory usage is proportional to batch size, not file size.

disk-rewrite

Writes data with DuckDB, then rewrites row-group by row-group using PyArrow to add proper GeoParquet metadata. Uses more memory than streaming but provides maximum compatibility.

in-memory

Loads the entire dataset into memory before writing. Only use this for small files or when you need to verify that another strategy is producing correct output.

Selecting a Strategy¶

Override the default strategy when needed:

# Use streaming strategy
gpio extract input.parquet output.parquet --write-strategy streaming

# Use in-memory for verification
gpio extract input.parquet output.parquet --write-strategy in-memory

import geoparquet_io as gpio

# Use streaming strategy
gpio.read('input.parquet').write('output.parquet', write_strategy='streaming')

# Use in-memory for verification
gpio.read('input.parquet').write('output.parquet', write_strategy='in-memory')

When to Use Alternative Strategies¶

Decision Flowchart¶

1. Start with the default (duckdb-kv) - it handles any file size efficiently
2. Output seems wrong? Try in-memory to verify correct behavior
3. in-memory works but duckdb-kv doesn't? Report a bug, use streaming as workaround
4. Need maximum compatibility? Try disk-rewrite

Specific Scenarios¶

Memory Configuration¶

Where Spilled Data Goes¶

A memory limit only helps if DuckDB has somewhere to put the data that does not fit. gpio gives every DuckDB connection its own scratch directory under the system temp directory — $TMPDIR on macOS and Linux, %TEMP% on Windows — and DuckDB removes it again when the query finishes. A run that never exceeds its memory limit writes nothing there.

This matters in three situations:

• A read-only working directory. DuckDB's own default spill location is the relative path .tmp, so out of the box a large sort fails with IO Error: Failed to create directory ".tmp" even when the input and output volumes are perfectly writable. gpio never spills into the working directory.

• A small root volume. A sort of a very large file can spill more bytes than the output itself. If /tmp is too small, point TMPDIR at a volume that is not:

  TMPDIR=/mnt/scratch gpio sort hilbert huge.parquet sorted.parquet
  

  gpio pmtiles create and pyramid follow the same variable and also take it as a flag, --temporary-directory, because tippecanoe's scratch is the largest gpio ever produces — see Scratch space.

• A RAM-backed /tmp. systemd mounts /tmp as a tmpfs sized at half of RAM by default on Fedora, Arch and openSUSE, and so do Kubernetes' emptyDir: {medium: Memory} and docker run --tmpfs /tmp. Spilling onto a tmpfs spends the very memory the spill was meant to save, so on those systems TMPDIR should name real storage — df -h /tmp and findmnt /tmp say which you have.

Cleaning Up After an Interrupted Run¶

DuckDB removes its scratch directory when the query finishes — including when it fails. A run killed mid-spill (Ctrl-C, SIGTERM, the OOM killer) is the exception: the process is gone before that cleanup can run, and a directory holding gigabytes stays on disk.

gpio names those directories gpio-spill-<pid>-<random> and sweeps them at the start of every run: any leftover whose owning process has exited is removed before a new one is created, so interrupted runs do not accumulate. A directory belonging to a process that is still running is never touched. (Pids are the test, so give each container its own TMPDIR rather than bind-mounting one spill volume into several — across pid namespaces the test cannot tell a live run from a dead one.)

The one place an OS temp reaper would never help is the admin cache, because the admin-boundary commands spill onto the cache volume rather than into /tmp. --clear-cache prices and deletes those leftovers too:

gpio add admin-divisions in.parquet out.parquet --clear-cache

Cache directory: /home/you/.geoparquet-io/cache/admin
Files to delete: 1
Total size: 3.00 MB
Leftover spill directories: 1 (250.00 MB)

Automatic Detection¶

Every DuckDB connection gpio opens runs under a memory limit of 50% of the process's memory ceiling. That covers the writes — the duckdb-kv strategy (with or without a geometry column), the plain COPY that writes GeoParquet 2.0 and parquet-geo-only output when no metadata rewrite is needed, the single-pass split behind gpio partition and gpio pmtiles create --chunks, the admin dataset cache downloads, the full rewrite gpio add bbox-metadata falls back to, and the disk-rewrite strategy's first phase — and equally the DuckDB work that is not a write: the full-input spatial join gpio partition admin and gpio add admin-divisions materialize into a temp table before anything is written, the format exports behind gpio convert csv/flatgeobuf/geojson/geopackage/shapefile, and the queries behind gpio process aggregate and gpio process overview. The streaming and in-memory strategies read through a bounded connection like everything else, but the rows they have read sit in Arrow, outside DuckDB's accounting: the limit bounds the query feeding them, not the buffers they build from it.

--write-memory overrides the default on the commands that have the option, including the admin joins; gpio pmtiles create, gpio add bbox-metadata and the format exports do not have it, so they always use the default.

The precedence, highest first:

1. An explicit --write-memory (or memory_limit=, where the Python API takes one).
2. A stricter limit you set on your own DuckDB connection — gpio will not raise it, and restores it after the write.
3. gpio's ceiling-aware default, 50% of the cgroup cap or physical RAM, whichever is lower.
4. DuckDB's own default (about 80% of host RAM), which is what remains when no ceiling can be read at all.

Half, not DuckDB's own default of 80%: DuckDB's limit covers only its own buffers. The Parquet writer, compression and spatial functions allocate beside it, and a large Hilbert sort was measured running 30–70% past the limit, so 80% can reach a container or job cap before DuckDB spills, and the process is killed. For the same reason gpio lowers DuckDB's thread count while a small limit is in force (about 512 MB per thread): spread too thin, DuckDB raises Out of Memory Error instead of spilling. The cap applies around a bounded statement — a write, an export, an admin join — and again when a connection is opened under the default, so work that no such statement wraps is not left spreading a small limit across every core. It only ever lowers the thread count, and a thread count you ask for is yours.

The ceiling, not the memory currently free: a cgroup counts its page cache as used, so after a job has read a large input, "free" memory can read near zero.

A limit you set on your own DuckDB connection (Python API) is respected when it is stricter, and restored after the write — on every write path, including duckdb-kv, which previously raised it to the percentage default for the duration of the write.

The ceiling is container- and scheduler-aware. gpio reads the process's own cgroup from /proc/self/cgroup and walks up to the root, taking the tightest cap it finds:

• cgroup v2 (modern Docker, Kubernetes, Slurm): memory.max
• cgroup v1 (older Docker, Slurm on RHEL/Rocky 8): memory/…/memory.limit_in_bytes
• Bare metal: physical RAM via psutil

A batch scheduler such as Slurm caps the job's cgroup, not the root of the hierarchy, so a check of the root alone sees no limit there.

Explicit Memory Limits¶

Override auto-detection when needed:

# Limit DuckDB to 2GB for streaming writes
gpio extract input.parquet output.parquet --write-memory 2GB

# Smaller limit for restricted environments
gpio extract input.parquet output.parquet --write-memory 512MB

import geoparquet_io as gpio

# Streaming strategy for constant memory usage
gpio.read('input.parquet').write('output.parquet', write_strategy='streaming')

Memory Sizing Guidelines¶

Container Environments¶

Docker¶

gpio detects Docker memory limits automatically via cgroups. No extra configuration needed:

# Docker automatically limits memory, gpio respects it
docker run -m 2g my-gpio-image gpio extract input.parquet output.parquet

If you need explicit control:

docker run -m 2g my-gpio-image gpio extract input.parquet output.parquet --write-memory 1GB

Kubernetes¶

Memory limits from pod specifications are detected via cgroups:

resources:
  limits:
    memory: "4Gi"

gpio will automatically use approximately 2GB (50% of the limit) for DuckDB operations.

Serverless (AWS Lambda, Cloud Run)¶

For serverless environments with tight memory constraints:

# AWS Lambda with 1GB memory
gpio extract input.parquet output.parquet --write-memory 384MB

# Cloud Run with 2GB memory
gpio extract input.parquet output.parquet --write-memory 768MB

import geoparquet_io as gpio

def handler(event, context):
    # Memory limit is auto-detected from the Lambda environment
    gpio.read('s3://bucket/input.parquet') \
        .extract(bbox=event['bbox']) \
        .write('/tmp/output.parquet')

Examples¶

Process a Large Dataset¶

# 100GB dataset on a 16GB machine - just works
gpio extract large_dataset.parquet filtered.parquet \
    --bbox -122.5,37.5,-122.0,38.0 \
    --where "population > 1000"

import geoparquet_io as gpio

# Large file processing with the fluent API
gpio.read('large_dataset.parquet') \
    .extract(
        bbox=(-122.5, 37.5, -122.0, 38.0),
        where="population > 1000"
    ) \
    .write('filtered.parquet')

Troubleshoot Output Issues¶

If you suspect the default strategy is producing incorrect output:

# 1. Write with in-memory strategy (loads full dataset)
gpio extract input.parquet test_inmemory.parquet --write-strategy in-memory

# 2. Compare with default strategy output
gpio inspect stats test_inmemory.parquet
gpio inspect stats output.parquet

import geoparquet_io as gpio

# Verify with in-memory strategy
table = gpio.read('input.parquet')

# Write with default
table.write('output_default.parquet')

# Write with in-memory for comparison
table.write('output_inmemory.parquet', write_strategy='in-memory')

# Compare
default = gpio.read('output_default.parquet')
inmemory = gpio.read('output_inmemory.parquet')
print(f"Default rows: {default.num_rows}, In-memory rows: {inmemory.num_rows}")

Batch Processing in Constrained Environment¶

# Process multiple files with limited memory
for f in data/*.parquet; do
    gpio extract "$f" "output/$(basename $f)" \
        --write-memory 512MB \
        --bbox -122.5,37.5,-122.0,38.0
done

import geoparquet_io as gpio
from pathlib import Path

# Batch processing (memory limit auto-detected per environment)
for input_file in Path('data').glob('*.parquet'):
    gpio.read(input_file) \
        .extract(bbox=(-122.5, 37.5, -122.0, 38.0)) \
        .write(f'output/{input_file.name}', write_strategy='streaming')

See Also¶

• Extracting Data - Full extract command documentation
• Best Practices - GeoParquet optimization tips
• Troubleshooting - Common issues and solutions