Phase 5: MLOps & Production

Dataset Versioning (DVC, LakeFS)

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Have you ever tried to bake your favorite cookies, but you changed one tiny thing, like adding more chocolate chips, and suddenly they tasted even better? Or maybe you tried a new ingredient, and they didn't turn out so great? It's fun to experiment, but sometimes you want to make that exact perfect batch again, or figure out what went wrong. How do you remember exactly what you put in, and how much, for each try?

That's where something called "Dataset Versioning" comes in, and it's like having a super-organized kitchen assistant for your ingredients. Imagine your main recipe book is where you write down the steps for baking (that's like the "code" for a computer). But the actual chocolate chips, flour, and sugar? Those are your "data." If you just wrote down "chocolate chips" in your main recipe book, you wouldn't know if you meant big chunks or mini ones, or if you used dark chocolate this time!

Instead of writing all the ingredients directly in your main recipe book, a special tool called DVC (which stands for Data Version Control) helps you keep track. Think of DVC as creating a special, detailed ingredient list for each baking attempt. So, in your main recipe book, instead of listing all the ingredients, you just write a little note that says, "For this cookie recipe, use 'Ingredient Set #3'." DVC then stores the actual ingredients for 'Ingredient Set #3' (like "1 cup mini dark chocolate chips, 2 cups all-purpose flour...") in a big, organized pantry. When you try new chocolate chips, DVC creates a brand new 'Ingredient Set #4' and updates the note in your recipe book.

So, what does this mean for you? If those cookies with 'Ingredient Set #3' were absolutely perfect, you can always tell DVC to fetch those exact ingredients from the pantry, and you'll bake the same amazing cookies every time. If your new 'Ingredient Set #4' cookies taste bad, you can easily go back to the perfect 'Ingredient Set #3' without guessing. This means you can experiment with new ideas for your baking all the time, knowing you can always reproduce your best recipes, or figure out why a new one didn't quite work. You'll always know exactly what went into your delicious creations!

As an ML Engineer, you know models are only as good as the data they're trained on. But datasets are rarely static – they evolve with new features, cleaning steps, or additional samples. Without proper tracking, it's incredibly difficult to reproduce past experiment results, debug performance regressions, or confidently deploy models to production. This is where Dataset Versioning becomes critical. It's the practice of tracking changes to your data over time, ensuring you know exactly which version of the dataset was used for a specific model training run or prediction. It forms a cornerstone of reproducible ML and robust MLOps practices.

DVC (Data Version Control) offers a Git-like experience for your data files and machine learning models. Instead of storing large datasets directly in Git (which is inefficient), DVC stores small pointer files (like .dvc files) in your Git repository. These pointers reference the actual data, which DVC then manages in remote storage like S3, GCS, Azure Blob Storage, or even a local network drive. This allows you to git commit your code and your .dvc files together, effectively versioning your code and data in sync. You can then dvc checkout a specific Git commit, and DVC will automatically retrieve the corresponding dataset version, ensuring your experiments are perfectly reproducible.

While DVC is excellent for managing datasets alongside your code, especially in local development and smaller projects, LakeFS takes a different, more powerful approach by bringing Git-like branching, committing, and merging capabilities directly to your data lake (e.g., S3-compatible object storage). Imagine being able to create an isolated "branch" of your production dataset for an experiment, perform transformations, and then merge it back without affecting production until it's ready. LakeFS achieves this with zero-copy branching, meaning you don't duplicate data. It's particularly powerful for large-scale data engineering workflows, CI/CD pipelines for data, and ensuring data consistency across multiple teams by providing atomic, isolated data operations. Together, DVC and LakeFS provide comprehensive solutions for data versioning, catering to different scales and use cases within the ML lifecycle.

Key Takeaways

  • Dataset versioning ensures reproducibility and traceability of ML experiments.
  • DVC integrates with Git, tracking data via pointer files linked to remote storage.
  • LakeFS enables Git-like branching, committing, and merging directly on data lakes.
  • Both tools allow you to retrieve the exact dataset used for any past model.
  • Essential for robust MLOps, debugging, and collaboration among ML engineers.

Code Example

bash
# Initialize DVC in your project
dvc init

# Add a dataset to be versioned (e.g., your training data)
dvc add data/train.csv

# Git add and commit the DVC metadata file
git add data/train.csv.dvc
git commit -m "Version 1 of training data"

# Push the data to remote storage (e.g., S3 configured via `dvc remote add`)
dvc push

# Later, if you modify data/train.csv and want to version it again:
dvc add data/train.csv
git add data/train.csv.dvc
git commit -m "Version 2: Added more samples"
dvc push

# To reproduce an old experiment, checkout a Git commit:
# git checkout <commit_hash_of_version_1>
# dvc checkout # Retrieves data/train.csv for version 1

How this code works

This code illustrates how to manage different versions of machine learning datasets using DVC, integrating seamlessly with Git. Its primary job is to enable reproducible experiments by linking specific dataset versions to specific code commits, without bloating your Git repository with large data files.

The process begins by initializing DVC with dvc init. Then, dvc add data/train.csv tells DVC to track the dataset. Instead of adding the large train.csv directly to Git, DVC creates a small data/train.csv.dvc metadata file, which contains a pointer to the actual data. This *.dvc file is then versioned by Git using git add data/train.csv.dvc and git commit. The actual data is uploaded to a configured remote storage by dvc push. A subtle but crucial point for beginners is understanding that dvc add creates the metadata file, which then needs to be committed to Git. If data/train.csv changes, repeating dvc add updates this metadata file, and another git commit tracks this new version. To revert to an older dataset version, one first uses git checkout to move to the desired Git commit, and then dvc checkout retrieves the corresponding data from DVC's cache or remote storage.

Related Articles

Hand-picked reads to go deeper on experiment tracking & versioning.

Back to ML Engineer Roadmap