Why your ML model is lying to you—and geography is the missing variable

The lie your model tells

Consider an illustrative production scenario. A team trains a churn model on 200,000 customer records, uses a random 80/20 split, and reports a test AUC of 0.91. Six months after deployment, performance in newly served regions is 0.67—even though the preprocessing and model code are unchanged.

The hidden difference is geography. Nearby customers from the same cities, neighbourhoods, and service areas appeared in both training and test data. The evaluation therefore measured interpolation among familiar neighbours, while deployment required extrapolation to unfamiliar places.

The model was not tested on the geographic problem it would face after deployment.

This is spatial leakage: dependence across nearby observations makes a nominally held-out test set less independent than it appears. Research on spatially structured data shows that ordinary random cross-validation can produce optimistic performance estimates when the intended prediction domain differs geographically from the training sample. Roberts et al. review the underlying validation problem, while a later spatial k-fold study demonstrates it across real datasets.

Tobler’s First Law—and why your model does not know it

Waldo Tobler’s foundational 1970 paper states: “Everything is related to everything else, but near things are more related than distant things.” The original publication captures the intuition behind spatial autocorrelation.

Standard supervised-learning workflows often treat rows as independent and identically distributed. Geographic observations may instead form correlated neighbourhoods, environmental zones, markets, or service regions. A random split preserves those local relationships across both sides of the evaluation boundary.

How leakage happens

Imagine a property model evaluated with a random split inside one city. Test properties share school catchments, transport access, flood exposure, and neighbourhood character with training properties a few streets away. The model has not seen the exact test rows, but it has seen much of their local context.

If the deployment target is a new city, that validation design answers the wrong question. The evaluation protocol must reproduce the separation between the available training geography and the intended prediction geography.

Measuring spatial autocorrelation with Moran’s I

Moran’s I, introduced in Moran’s 1950 paper, summarizes similarity between values connected by a spatial weights matrix. Its interpretation depends on the chosen weights, sample configuration, expected value, and significance procedure; there is no universal threshold that proves leakage.

The PySAL guide to global Moran’s I shows how to compute and inspect the statistic. Test the target and, importantly, the residuals from a baseline model.

from libpysal.weights import Queen
from esda.moran import Moran
import geopandas as gpd

gdf = gpd.read_file("your_data.geojson")
w = Queen.from_dataframe(gdf)
w.transform = "r"

moran = Moran(gdf["target"], w)
print(f"Moran's I: {moran.I:.4f}, p={moran.p_sim:.4f}")Python

Spatially aware train/test splits

The correct split depends on the prediction task. A model used to interpolate between sampled locations needs a different evaluation design from one intended for new cities or countries.

1. Spatial block cross-validation

Divide the study area into contiguous blocks and assign whole blocks to folds. The peer-reviewed blockCV method discusses block construction and the importance of selecting a scale appropriate to spatial dependence.

from sklearn.model_selection import GroupKFold, cross_val_score

block_size = 0.5  # example only; choose from domain scale
gdf["block_id"] = (
    (gdf.geometry.x // block_size).astype(str) + "_" +
    (gdf.geometry.y // block_size).astype(str)
)

cv = GroupKFold(n_splits=5)
scores = cross_val_score(
    model, X, y,
    cv=cv,
    groups=gdf["block_id"],
    scoring="roc_auc",
)Python

scikit-learn documents GroupKFold as a splitter in which each non-overlapping group appears once in a test fold.

2. Buffered validation

Exclude training observations within a defensible distance of test observations. The buffer should reflect the process, sampling design, and spatial dependence—not an arbitrary number copied from another dataset.

3. Leave-region-out validation

Hold out an entire operational region when the deployment question is explicitly geographic transfer: “What happens when we launch in a place the model has never seen?”

Embedding coordinates carefully

Even with correct splits, geographic information may be useful as a feature. Raw longitude has a discontinuity at ±180°, while projected coordinates introduce scale and distortion choices. Spherical encoders can preserve global geometry, but the right representation remains task-dependent. Sphere2Vec provides one research-backed approach for geospatial prediction.

import numpy as np

def unit_sphere_coords(lat_deg, lon_deg):
    lat = np.radians(lat_deg)
    lon = np.radians(lon_deg)
    return np.stack([
        np.cos(lat) * np.cos(lon),
        np.cos(lat) * np.sin(lon),
        np.sin(lat),
    ], axis=-1)Python

Your five-step action plan

1. 01
   Audit spatial dependence

   Map the target, residuals, and sampling density; compute diagnostics with a documented weights definition.

2. 02
   Map every split

   Visually inspect whether nearby observations cross the train/test boundary.

3. 03
   Rebuild evaluation around deployment

   Use blocks, buffers, or held-out regions at a scale justified by the use case.

4. 04
   Ablate location features

   Keep coordinate features only when they improve the spatially appropriate evaluation—not merely random CV.

5. 05
   Monitor geography in production

   Track performance and calibration by region, coverage, distance from training support, and time.

An honest score aligned with deployment is more valuable than an impressive score built on geographic overlap.