Why Your ML Model Is Lying to You — and Geography Is the Missing Variable

The Lie Your Model Tells

Consider an illustrative production scenario. A team trains a churn model on 200,000 customer records, uses a random 80/20 split, and reports a test AUC of 0.91. Six months after deployment, performance in newly served regions is 0.67 — even though the preprocessing and model code are unchanged.

The hidden difference is geography. Nearby customers from the same cities, neighbourhoods, and service areas appeared in both training and test data. The evaluation therefore measured interpolation among familiar neighbours, while deployment required extrapolation to unfamiliar places.

This is spatial leakage: dependence across nearby observations makes a nominally held-out test set less independent than it appears. Research on spatially structured data shows that ordinary random cross-validation can produce optimistic performance estimates when the intended prediction domain differs geographically from the training sample. Roberts et al. review the underlying validation problem, while a later spatial k-fold study demonstrates it across real datasets.

Tobler’s First Law — and Why Your Model Does Not Know It

Waldo Tobler’s foundational 1970 paper states: "Everything is related to everything else, but near things are more related than distant things." The original publication captures the intuition behind spatial autocorrelation.

Standard supervised-learning workflows often treat rows as independent and identically distributed. Geographic observations may instead form correlated neighbourhoods, environmental zones, markets, or service regions. A random split preserves those local relationships across both sides of the evaluation boundary.

How Leakage Happens

Imagine a property model evaluated with a random split inside one city. Test properties share school catchments, transport access, flood exposure, and neighbourhood character with training properties a few streets away. The model has not seen the exact test rows, but it has seen much of their local context.

If the deployment target is a new city, that validation design answers the wrong question. The evaluation protocol must reproduce the separation between the available training geography and the intended prediction geography.

Measuring Spatial Autocorrelation with Moran’s I

Moran’s I, introduced in Moran’s 1950 paper, summarizes similarity between values connected by a spatial weights matrix. Its interpretation depends on the chosen weights, sample configuration, expected value, and significance procedure.

from libpysal.weights import Queen
from esda.moran import Moran
import geopandas as gpd

gdf = gpd.read_file("observations.geojson")
w = Queen.from_dataframe(gdf)
w.transform = "r"

moran = Moran(gdf["target"], w)
print(f"Moran's I: {moran.I:.4f}, p-value: {moran.p_sim:.4f}")

Spatially Aware Train / Test Splits

The correct split depends on the prediction task. A model used to interpolate between sampled locations needs a different evaluation design from one intended for new cities or countries.

from sklearn.model_selection import GroupKFold, cross_val_score

block_size = 0.5  # degrees or projected units
gdf["block_id"] = (
    (gdf.geometry.x // block_size).astype(str) + "_" +
    (gdf.geometry.y // block_size).astype(str)
)

cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, cv=cv, groups=gdf["block_id"], scoring="roc_auc")

Embedding Coordinates Carefully

Raw longitude has a discontinuity at ±180°, while projected coordinates introduce scale and distortion choices. Spherical encoders preserve global continuity.

import numpy as np

def unit_sphere_coords(lat_deg, lon_deg):
    lat = np.radians(lat_deg)
    lon = np.radians(lon_deg)
    return np.stack([
        np.cos(lat) * np.cos(lon),
        np.cos(lat) * np.sin(lon),
        np.sin(lat),
    ], axis=-1)

Five-Step Action Plan