Scaling Geospatial Pipelines: From 10 Files to 10,000 in AWS Batch

01. Treat “Essential container exited” as a starting point

Our first AWS Batch runs failed with the unhelpfully broad Essential container in task exited message. The useful signal lived one layer deeper in the exit code and container reason.

We made Python logs stream immediately from the container:

# Dockerfile
ENV PYTHONUNBUFFERED=1

# Equivalent runtime option
CMD ["python", "-u", "main.py"]

02. Give the platform and the workload separate identities

A headless container does not inherit your laptop credentials. Our NoCredentialsError disappeared once we separated the permissions required to launch a task from the permissions required by the code inside it.

AWS documents the same separation: the execution role is for the ECS/Fargate agent, while the task or job role is for application calls to services such as S3. Review the AWS IAM role guide.

03. Use one array job to coordinate 10,000 independent tiles

Once a single tile completed reliably, the workload became naturally parallel: every child job could use its own AWS_BATCH_JOB_ARRAY_INDEX to select one record from a validated manifest.

aws batch submit-job \
  --job-name imagery-production \
  --job-queue geospatial-prod \
  --job-definition imagery-worker:12 \
  --array-properties size=10000

AWS Batch supports array sizes from 2 to 10,000. The parent job becomes the management handle while each index is scheduled as a separate child job. That is ideal for tile processing because failures and retries stay isolated to individual inputs. See the official array-job behavior.

04. Scale compute deliberately—not all at once

Submitting 10,000 jobs does not mean 10,000 should start simultaneously. In our environment, the initial 32-vCPU service quota limited throughput, so we requested a case-specific increase to 1,000 vCPUs and kept the compute environment bounded by cost and downstream capacity.

We also smoothed the first S3 request from every container with bounded jitter and SDK retries. Jitter is not a replacement for proper prefix design or exponential backoff; it simply avoids an artificial burst caused by thousands of identical startup paths.

import os
import random
import time

index = int(os.environ["AWS_BATCH_JOB_ARRAY_INDEX"])
time.sleep(random.uniform(0, 180))  # smooth the startup wave
process_tile(manifest[index])

Amazon S3 scales to high request rates, but scaling is gradual and brief 503 Slow Down responses can occur during a sharp ramp. Read the current S3 performance guidance.

05. Make geospatial functions explicit and testable

Infrastructure stability exposed the next failure: a TypeError around Rasterio's rasterize() call. We removed positional ambiguity by switching to explicit keyword arguments and making the output geometry part of the log context.

from rasterio.features import rasterize

mask = rasterize(
    shapes=features,
    out_shape=(height, width),
    transform=transform,
    fill=0,
    default_value=1,
    dtype="uint8",
)

Keyword arguments do not become safer only because concurrency is high; they make the call contract clearer across library upgrades and easier to diagnose when thousands of jobs execute the same code path.

06. The final production workflow