AWS Batch Geospatial Pipelines: From 10 to 10,000 Tiles

← Back to Blog

Scaling a geospatial pipeline is rarely only a coding problem. It is an infrastructure problem involving memory, identity, concurrency, observability, and cost. When transitioning from processing local geotiffs on a workstation to handling tens of thousands of satellite tiles on AWS Batch, standard batch setups fail in subtle ways.

Here is the behind-the-scenes architecture and hard-won production lessons we use at Infryne TechWorks to turn fragile geospatial scripts into resilient, high-throughput cloud processing systems.

Treat "Essential Container Exited" as a Starting Point

Our first AWS Batch runs failed with the unhelpfully broad Essential container in task exited message. The useful signal lived one layer deeper in the exit code and container reason. AWS documents how non-zero exit codes and the /aws/batch/job log group surface failed attempts in its job-state reference.

Exit Code 137 · OOMKilled

Kernel Killed the Process

Raster operations can expand compressed imagery into massive in-memory numpy arrays. If the job reaches its container memory limit, Linux kills it instantly. Profile representative tiles before arbitrarily raising job definition memory.

Exit Code 1 · App Crash

Unbuffered Log Drops

Python standard output is buffered by default in non-interactive shells. If a crash occurs before buffers flush to CloudWatch, the task appears silently dead.

We force Python logs to stream unbuffered from the container:

# Dockerfile
ENV PYTHONUNBUFFERED=1

# Or runtime entrypoint
CMD ["python", "-u", "process_tile.py"]

⬡ Operational Lesson

Always log the S3 tile key, array index, allocated memory, and processing stage before loading raster data. A crash should immediately identify the unit of work without running expensive forensic reruns.

Separate Platform Identity from Workload Identity

A headless container running in AWS Batch does not inherit workstation credentials. Our initial NoCredentialsError was resolved once we cleanly separated the container execution role from the application job role.

Execution Role (ECS Agent)

Pulls container image from Amazon ECR
Writes container stdout/stderr to CloudWatch
Fetches SSM/Secrets Manager environment variables

Job Role (Application Code)

Reads specific S3 imagery input prefixes
Writes processed COGs to S3 output buckets
Restricted by strict least-privilege IAM policies

AWS details this exact architecture in its documentation: the execution role is used by the ECS container agent, while the job role is assumed by the Python script. Review the AWS IAM role guide.

Coordinate 10,000 Independent Tiles with Array Jobs

Rather than managing 10,000 individual job submissions, AWS Batch provides array jobs. Every child container receives an AWS_BATCH_JOB_ARRAY_INDEX environment variable that maps cleanly to a pre-validated manifest.

aws batch submit-job \
  --job-name satellite-imagery-batch \
  --job-queue geospatial-high-throughput \
  --job-definition imagery-processor:14 \
  --array-properties size=10000

AWS Batch supports array sizes from 2 up to 10,000 child jobs. The parent job acts as a single lifecycle handle while each child task executes independently with its own retries and log stream. See the official array-job behavior.

Workflow diagram showing S3 manifest fan-out into 10,000 indexed AWS Batch containers

Scale Compute Deliberately and Smooth S3 Concurrency

Submitting 10,000 tasks simultaneously can overwhelm downstream S3 prefixes with burst requests, resulting in HTTP 503 SlowDown responses. Adding bounded jitter to startup routines smooths the initial request crest.

import os, random, time

index = int(os.environ["AWS_BATCH_JOB_ARRAY_INDEX"])
# Prevent 10,000 workers from hitting S3 on the exact same second
time.sleep(random.uniform(0, 120))
process_tile(manifest[index])

Amazon S3 scales automatically to virtually any request volume, but the partition scaling curve is gradual. Read the current S3 performance guidance.

Make Geospatial Functions Explicit and Testable

Positional arguments in Rasterio functions like rasterize() can cause silent bugs across dependency updates. We enforce explicit keyword arguments and strict type validation in all worker routines.

from rasterio.features import rasterize

mask = rasterize(
    shapes=geometries,
    out_shape=(height, width),
    transform=affine_transform,
    fill=0,
    default_value=1,
    dtype="uint8",
)

Compare and verify the parameter signatures against Rasterio's official rasterize() API reference.

The Final Production Workflow

Stage	Action	Outcome
Debug	Forced unbuffered stdout logs	Isolated missing dependency and corrected Docker context.
Permissions	Assigned execution vs job IAM roles	Resolved S3 auth with zero credential leaks.
Logic	Explicit keyword arguments in Rasterio	Deterministic mask generation across tile bounds.
Scale	Controlled vCPU quotas & S3 jitter	Sustained concurrency with predictable costs.
Operate	Granular per-tile CloudWatch alarms	Full observability and automated single-tile retries.