Spatial Dataset Versioning with DVC and LakeFS

Reproducibility in geospatial machine learning fails at the data layer long before it fails at the model layer. A land-cover classifier trained last quarter is worthless six months later if you cannot recover the exact stack of cloud-optimized GeoTIFFs, tile grids, and label rasters it consumed — byte for byte. Git handles the training code fine, but a single Sentinel-2 mosaic can exceed 40 GB, a national label raster another 10 GB, and a tiled feature store hundreds of thousands of small files. Committing any of that to git corrupts the repository and destroys clone times. The datasets and the model version must move in lockstep, yet they cannot live in the same store.

This is part of Geospatial MLOps and Model Deployment, which covers the operational lifecycle from data versioning through drift monitoring. This guide shows how to pin multi-gigabyte raster and vector datasets to model versions using two complementary tools: DVC (Data Version Control), which content-addresses files and threads dataset hashes through git history, and LakeFS, which gives an entire object-storage data lake git-like branches and immutable commits. You will initialise tracking, pin a dataset version to a model release, branch experiments on immutable snapshots, and rebuild feature extraction deterministically.

Problem Framing — why versioning geospatial data is uniquely hard

Tabular MLOps treats a dataset as a single CSV whose hash is cheap to compute and whose diffs are small. Geospatial data breaks every assumption behind that model.

Size and cardinality. A single training scene is measured in gigabytes, and a tiled feature store fans that out into hundreds of thousands of individual .tif chips. Any tool that re-hashes or re-uploads the entire dataset on every change is unusable. You need content-addressed deduplication so that changing one tile touches one object, not the whole store.

Lockstep coupling. A model is only reproducible if three artefacts are pinned together: the input rasters (imagery, DEMs, ancillary layers), the label rasters or vector ground truth, and the trained model weights. If a label raster is silently re-rasterised at a different resolution, every downstream metric shifts, and without a version pin you will never notice. Reproducible evaluation — the same discipline that underpins spatial cross-validation strategies — depends on the split, the features, and the labels all being recoverable at a fixed hash.

Metadata sensitivity. Two rasters can contain identical pixels but differ byte-for-byte because of an embedded timestamp, a changed nodata sentinel, or a rewritten CRS record. Naive checksums then report a spurious change. Versioning geospatial data means understanding when a hash change is real and when it is metadata noise.

DVC and LakeFS solve different slices of this problem, and the strongest pipelines use both. DVC lives inside the git repository: it replaces large files with small .dvc pointer files that git tracks, while the bytes sit in an S3 or MinIO remote. That makes the dataset version a first-class part of your git history, right next to the model and the training code. LakeFS sits one layer lower, wrapping the object-storage bucket itself in git-like semantics — branches, commits, and atomic merges over the entire lake — so that many teams can experiment on shared petabyte-scale data without copying it.

Choosing between DVC and LakeFS

Pick the tool that matches where your reproducibility boundary sits. If the unit you need to pin is “the exact data and transformation a single model consumed,” DVC is the natural fit because that pin lives in the same commit as the code. If the unit is “the state of a shared lake that dozens of jobs read from,” LakeFS is the natural fit because it version-controls the bucket itself.

In practice, teams reach for DVC first because it needs nothing beyond git and an S3 bucket, then adopt LakeFS once concurrent access to a shared lake becomes the bottleneck. Running both is not redundant: LakeFS pins the lake, and DVC pins the slice of it that produced a given model.

Prerequisites & Environment Setup

Pin every tool that touches the bytes. Version drift in GDAL, DVC, or the S3 client is the single most common cause of phantom checksum changes.

# Pinned requirements for spatial dataset versioning
dvc[s3]==3.51.2
lakefs-spec==0.11.0
boto3==1.34.140
rasterio==1.3.10
gdal==3.8.4
xxhash==3.4.1

Install the Python tooling:

pip install "dvc[s3]==3.51.2" "lakefs-spec==0.11.0" "boto3==1.34.140" \
            "rasterio==1.3.10" "xxhash==3.4.1"

DVC and LakeFS both assume a git repository and an S3-compatible object store. For local development, MinIO is the standard stand-in for S3:

# Local MinIO for a self-contained remote (development only)
docker run -d --name minio -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  quay.io/minio/minio server /data --console-address ":9001"

Confirm your git and DVC versions before doing anything else — DVC’s on-disk cache format changed across major versions, and a mismatched client silently rewrites pointer files:

git --version          # expect >= 2.30
dvc --version          # expect 3.x
dvc doctor             # prints platform, remotes, and cache config

All raster inputs must already be aligned to a single CRS, resolution, and grid before you version them; versioning is not a substitute for preprocessing. If your layers are misaligned, resolve that first following CRS Alignment and Projection Handling, because re-warping a raster after it has been committed changes its hash and breaks lineage.

Step-by-Step Implementation

Step 1 — Initialise DVC and track a COG and label directory

Initialise DVC inside an existing git repository, then configure the object-storage remote. Track the imagery and label directories separately so their versions can advance independently.

# Inside your git repo
dvc init
git commit -m "Initialise DVC"

# Configure an S3 / MinIO remote
dvc remote add -d storage s3://geo-ml-data/dvcstore
dvc remote modify storage endpointurl http://localhost:9000
dvc remote modify storage access_key_id minioadmin
dvc remote modify storage secret_access_key minioadmin

Now content-address the datasets. Point dvc add at directories, never at a single concatenated archive — directory tracking is what enables per-tile deduplication.

# Track imagery and labels as independent versioned units
dvc add data/imagery/          # multi-GB COG tiles
dvc add data/labels/           # rasterised ground-truth masks

git add data/imagery.dvc data/labels.dvc data/.gitignore
git commit -m "Track imagery and label rasters with DVC"

dvc push                       # upload bytes to the remote

DVC replaces each directory with a small .dvc pointer file containing the directory’s aggregate hash. Inspect one to see what git is actually storing:

# data/imagery.dvc
outs:
  - md5: a1f9c3d0b47e2f1a9c8e5d6b3f204e77.dir
    size: 42041839104
    nfiles: 1284
    path: imagery

The .dir hash summarises every file in the directory. Because DVC deduplicates by content, re-adding the directory after changing one tile uploads only that tile’s new object.

Step 2 — Pin the dataset version to a git commit and tag

The reproducibility guarantee comes from committing the .dvc pointer files alongside the training code and model, then tagging the release. The git tag now resolves to an exact dataset hash and an exact model version simultaneously.

# After training produces a model, track it too
dvc add models/landcover_xgb.json
git add models/landcover_xgb.json.dvc

# Commit code + data pointers + model pointer together, then tag
git commit -m "Release land-cover model v2.3.0"
git tag -a release-2.3.0 -m "Dataset a1f9c3 + model v2.3.0"
git push --tags
dvc push

To recover the exact state months later, checkout the tag and let DVC pull the matching bytes:

git checkout release-2.3.0
dvc pull                       # fetches imagery, labels, and model at pinned hashes

This is the core pattern: git carries the small pointers and the human-readable version tag, while DVC materialises the multi-gigabyte reality behind them. The dataset hash and the model version can never drift apart because a single commit binds them.

Step 3 — Branch experiments with LakeFS immutable snapshots

DVC pins the subset a model consumed; LakeFS versions the entire lake so multiple teams can experiment without copying terabytes. Create a branch, mutate data on it, and commit an immutable snapshot whose commit ID pins the exact bytes.

import lakefs
from lakefs.client import Client

client = Client(
    host="http://localhost:8000",
    username="AKIAIOSFOLKFSSAMPLES",
    password="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
)

repo = lakefs.Repository("geo-lake", client=client)

# Branch off main to isolate an experiment — no bytes are copied
exp = repo.branch("exp-cloudmask-v2").create(source_reference="main")

# Write a reprocessed label raster onto the branch
with open("data/labels/tile_0421.tif", "rb") as f:
    exp.object("labels/tile_0421.tif").upload(data=f.read(), mode="wb")

# Commit an immutable snapshot; the returned id pins these exact bytes
commit = exp.commit(message="Reprocess tile_0421 with cloud mask v2")
print(f"LakeFS snapshot: {commit.get_commit().id}")

Because the commit is immutable, a training run that reads lakefs://geo-lake/exp-cloudmask-v2@<commit-id>/ is guaranteed to see identical bytes on every rerun, even if main advances underneath. Merge the branch back only once the experiment is validated:

exp.merge_into("main")   # atomic; either all changes land or none do

This gives you the same branch-and-merge workflow you use for code, applied to a data lake far too large to duplicate per experiment.

Step 4 — Reproduce feature extraction with a DVC pipeline stage

Hashing raw inputs is necessary but not sufficient — the transformation from raw tiles to features must also be reproducible. A DVC pipeline stage declares its dependencies and outputs so it reruns only when inputs actually change, and so the output hash is verifiable.

# src/extract_features.py
import sys
import numpy as np
import rasterio
from pathlib import Path


def extract_features(imagery_dir: str, out_path: str) -> None:
    """Compute a deterministic NDVI feature stack from COG tiles.

    Args:
        imagery_dir: Directory of DVC-tracked COG tiles.
        out_path: Destination path for the stacked feature raster.
    """
    tiles = sorted(Path(imagery_dir).glob("*.tif"))   # sort → deterministic order
    if not tiles:
        raise FileNotFoundError(f"No .tif tiles found in {imagery_dir}")

    with rasterio.open(tiles[0]) as ref:
        profile = ref.profile.copy()

    profile.update(count=1, dtype="float32", compress="deflate", predictor=2)

    with rasterio.open(out_path, "w", **profile) as dst:
        for i, tile in enumerate(tiles):
            with rasterio.open(tile) as src:
                red = src.read(3).astype("float32")
                nir = src.read(4).astype("float32")
                denom = nir + red
                ndvi = np.where(denom == 0, 0.0, (nir - red) / denom)
            dst.write(ndvi.astype("float32"), 1, window=src.block_windows())
    print(f"Wrote features for {len(tiles)} tiles to {out_path}")


if __name__ == "__main__":
    extract_features(sys.argv[1], sys.argv[2])

Wire it into dvc.yaml so the stage is a tracked, reproducible node in the dependency graph:

# dvc.yaml
stages:
  extract_features:
    cmd: python src/extract_features.py data/imagery data/features/ndvi.tif
    deps:
      - src/extract_features.py
      - data/imagery
    outs:
      - data/features/ndvi.tif

Run and reproduce with a single command. DVC skips the stage entirely when neither the code nor the input hashes have changed:

dvc repro                      # runs only stages whose deps changed
dvc push

The pinned hash of data/features/ndvi.tif now propagates into the same release commit from Step 2, so features, labels, model, and code are all recoverable from one git tag. For the band-math logic that feeds this stage, see raster band math and index calculation.

Verification & Testing

Never trust that a dvc pull returned the bytes you expected — verify. DVC exposes the working-tree status and can validate every tracked file against its recorded hash.

dvc status                     # reports pipeline stages that are out of date
dvc status -c                  # compares local cache against the remote
dvc data status --granular     # per-file added / modified / deleted

A clean, reproducible checkout reports nothing to do:

Data and pipelines are up to date.

For a defence-in-depth check, recompute the hash of a materialised feature raster independently and assert it matches the value DVC recorded. This catches silent corruption in transit or a mismatched cache:

import hashlib
from pathlib import Path
import yaml


def verify_dvc_output(dvc_file: str, tracked_path: str) -> None:
    """Assert a file's MD5 matches the hash recorded in its .dvc pointer.

    Args:
        dvc_file: Path to the .dvc pointer file.
        tracked_path: Path to the materialised data file to verify.
    """
    meta = yaml.safe_load(Path(dvc_file).read_text())
    expected = meta["outs"][0]["md5"]

    h = hashlib.md5()
    with open(tracked_path, "rb") as f:
        for chunk in iter(lambda: f.read(8 * 1024 * 1024), b""):
            h.update(chunk)
    actual = h.hexdigest()

    assert actual == expected, (
        f"Hash mismatch for {tracked_path}: "
        f"expected {expected}, got {actual}"
    )
    print(f"Verified {tracked_path} == {expected}")

For directory outputs the recorded hash ends in .dir and summarises a manifest rather than a single file; use dvc data status for those. Wire the single-file check into CI so any drift between the pointer and the bytes fails the build before a model is promoted.

Troubleshooting & Common Errors

Large binaries accidentally committed to git

Cause: A raster was git add-ed before it was dvc add-ed, so multi-gigabyte bytes are now baked into git history and every clone drags them along.

Fix: Remove the file from git tracking, hand it to DVC, and recommit the pointer:

git rm --cached data/imagery/scene.tif
dvc add data/imagery/scene.tif
git add data/imagery/scene.tif.dvc data/imagery/.gitignore
git commit -m "Move scene.tif from git to DVC"

If the blob is already deep in history and bloating clones, rewrite history with git filter-repo to purge it, then force-push. Add a pre-commit hook that rejects any staged file over a size threshold to prevent recurrence.

.dvc pointer drift between branches

Cause: Two branches each ran dvc add on the same directory after independent edits, producing conflicting .dir hashes in their .dvc files. A git merge leaves the pointer pointing at bytes that were never pushed.

Fix: Resolve the pointer conflict in favour of one hash, then reconcile the cache:

git checkout --theirs data/imagery.dvc   # or --ours, whichever is canonical
dvc checkout                             # sync working tree to the chosen pointer
dvc pull                                 # fetch any missing objects
dvc status                               # must report "up to date"

Always dvc push immediately after dvc add so no branch ever references bytes that are absent from the remote.

Non-deterministic feature extraction breaks hashes

Cause: The feature stage iterates tiles in filesystem order, uses an unseeded random sample, or writes multi-threaded output, so pixel-identical runs still produce different bytes and a new hash on every dvc repro.

Fix: Make the transformation deterministic. Sort inputs explicitly (sorted(Path(...).glob(...))), seed every RNG, and pin thread counts. For any GDAL-backed writer set GDAL_NUM_THREADS=1 during hashing-sensitive stages so block ordering is stable. A stage is only reproducible when two clean runs yield the same output hash.

CRS or nodata metadata changes alter the checksum

Cause: Rewriting a raster with a slightly different nodata sentinel, an updated CRS WKT string, or new creation options changes the file bytes even though the pixels are identical — so DVC reports a change that is pure metadata noise.

Fix: Normalise raster metadata before versioning. Standardise the nodata value, write a canonical CRS, and fix a single set of creation options across the whole pipeline:

import rasterio

with rasterio.open("in.tif") as src:
    profile = src.profile.copy()
    profile.update(
        nodata=-9999,
        crs="EPSG:32633",
        compress="deflate",
        predictor=2,
        tiled=True,
        blockxsize=512,
        blockysize=512,
    )
    data = src.read()

with rasterio.open("normalised.tif", "w", **profile) as dst:
    dst.write(data)

dvc push fails with Access Denied or NoCredentialsError

Cause: The S3/MinIO credentials on the remote are unset, expired, or the bucket policy denies PutObject.

Fix: Verify credentials resolve (aws sts get-caller-identity for real S3, or mc admin info for MinIO) and that the IAM policy grants s3:PutObject, s3:GetObject, and s3:ListBucket on the DVC prefix. For CI, inject credentials via environment variables rather than committing them into .dvc/config.

Checkout of an old tag pulls nothing

Cause: The bytes for that historical hash were garbage-collected from the remote, or dvc push was never run for that release.

Fix: Run dvc gc only with explicit workspace and tag protection (dvc gc --workspace --all-tags --cloud) so referenced versions are never collected. Treat dvc push as mandatory in the same automation step as git push --tags; a tag with no pushed bytes is not a real release.

Performance Optimisation

Switch the hash algorithm for very large files. DVC’s default MD5 hashing dominates the cost of adding tens of gigabytes. On DVC 3.x you can adopt faster content hashing and, critically, avoid re-hashing unchanged files by preserving the cache between CI runs. Cache the .dvc/cache directory in your CI so a rebuild re-hashes only new tiles.

Parallelise transfers. Object-storage throughput, not local disk, is usually the bottleneck on push and pull. Raise the concurrency so many tiles move at once:

dvc remote modify storage jobs 16      # parallel upload/download workers
dvc push -j 16

Track directories, not archives. A single dataset.tar forces DVC to re-upload the whole file when one tile changes. Tracking the directory lets content addressing upload only the deltas — the difference between moving 40 GB and moving 30 MB on a routine label fix.

Use LakeFS zero-copy branches for experiment sprawl. When ten researchers each need a variant of a terabyte-scale lake, DVC’s per-workspace materialisation multiplies storage. A LakeFS branch is a metadata pointer — it copies no bytes until data is written — so parallel experiments cost only their deltas. This is the same efficiency mindset applied to model export in ONNX export for geospatial model inference, where the goal is minimising what moves at deploy time.

Keep the feature cache warm across pipeline runs. Because dvc repro skips stages whose input hashes are unchanged, storing the DVC cache on fast local NVMe (with the remote as backing store) turns most reruns into hash comparisons rather than recomputation — decisive when feature extraction over a national mosaic takes hours.

FAQ

Should I use DVC or LakeFS for spatial data versioning?

Use DVC when versioning is tightly coupled to code and model artefacts in git and you want reproducible pipeline stages that rebuild features on demand. Use LakeFS when many teams share one object-storage data lake and you need git-like branches, commits, and atomic merges over petabyte-scale buckets without physically copying files. The two are complementary rather than competing: LakeFS versions the lake as a whole and gives you cheap experiment branches, while DVC pins the exact subset — and the exact transformation — that a specific model consumed. Small-to-mid teams often start with DVC alone; large multi-team platforms layer LakeFS underneath and keep DVC for per-model pinning.

Does DVC store a full copy of every raster version?

No. DVC content-addresses each file by its hash and deduplicates in the remote cache, so identical bytes are stored exactly once regardless of how many versions reference them. Changing one tile in a tracked directory uploads only that tile’s new object; the unchanged tiles are shared across versions. This is why you should always dvc add a directory rather than a single concatenated archive — a monolithic archive defeats deduplication and forces a full re-upload on every edit.

Why does my COG checksum change after reprocessing identical data?

GeoTIFF writers embed metadata into the file bytes — libtiff version strings, creation timestamps, block ordering, compression predictor settings, and the CRS record. Two runs that produce pixel-identical rasters can still differ byte-for-byte, which changes the MD5 hash and makes DVC report a spurious update. Pin your GDAL and rasterio versions, fix a single canonical set of creation options, and normalise nodata and CRS metadata before versioning. When you need the transformation itself to be reproducible tile by tile, see Versioning Cloud-Optimized GeoTIFFs with DVC.

How does dataset versioning connect to drift monitoring?

A pinned dataset hash is the reference point drift detection compares against. When production inputs diverge from the versioned training distribution, you can only quantify the shift if the original feature dataset is recoverable at a fixed hash. Version pins and drift metrics are two halves of the same reproducibility loop — see model drift detection for geospatial inference for the monitoring side.

Related

• Model Drift Detection for Geospatial Inference — compare live inputs against a version-pinned training distribution
• ONNX Export for Geospatial Model Inference — package a pinned model version for portable deployment
• Containerizing Geospatial Inference Pipelines — bake pinned dataset and model versions into reproducible images
• Versioning Cloud-Optimized GeoTIFFs with DVC — tile-level deduplication and stable-checksum COG workflows
• Spatial Cross-Validation Strategies — reproducible geographic splits that depend on pinned data
• Temporal Aggregation for Time-Series Geodata — version-controlling multi-date feature stacks

Part of: Geospatial MLOps and Model Deployment