Phase 4: Data Quality & Governance

Great Expectations & Soda for data contracts

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you love to bake, and today you're making a big chocolate cake. You have a perfect recipe that tells you exactly what to do: "Use two cups of flour, four large eggs, a cup of sugar..." This recipe is super important, right? It's like an agreement between you and the person who wrote the recipe about what ingredients you need and how much. Now, what if someone accidentally gave you salt instead of sugar, or tiny quail eggs instead of large chicken eggs? If you just started baking without checking, your cake would be a disaster! In the world of computers and information, we have something similar called "data contracts." These are like the recipes for information, making sure everyone agrees on what the data should look like and what it means. This helps us build computer systems and programs that work perfectly, just like a delicious cake from a good recipe.

Now, let's say you want to be extra careful with your cake. This is where a tool called Great Expectations comes in. Think of Great Expectations as a super smart kitchen helper. Before you even mix anything, this helper looks at each ingredient as it arrives. It says, "Okay, recipe says two cups of flour. Let me check: Is this really flour? Yes. Is it exactly two cups? Yes. Good." Or, "Recipe says large eggs. Are these eggs large? Hmm, no, these are tiny! Stop! We can't use these for this recipe, or the cake won't turn out right!" Great Expectations helps you catch mistakes with your ingredients right at the start. It acts like a gatekeeper, making sure everything is just right before you use it. This stops you from wasting time and ingredients on a cake that's already doomed.

But what about making sure your cakes are always good, day after day, if you run a bakery? That's where another tool, Soda, comes in. Soda is like a quality control manager who regularly checks on things after they've been used or made. Imagine your bakery has several batches of flour delivered every week. Soda might take a little from each new batch of flour, or even taste a slice from a finished cake, to make sure everything still meets your high standards. It's not checking every single ingredient before every single cake, but it's constantly monitoring to make sure that your suppliers are sending good stuff and your bakers are making delicious cakes. If things start to slip, Soda will notice and let you know.

So, when you hear about "Great Expectations" and "Soda" in the world of data, it just means people are using these smart tools to make sure that all the important information we use in computers is exactly what we expect it to be. They help prevent messy mistakes and ensure that everything from your favorite video game to the online store your parents use runs smoothly because the data (the ingredients!) is always correct and reliable. This means you can trust the information you see and use, knowing that someone has carefully checked it, just like you'd trust a delicious cake from a careful baker.

Data contracts are essentially agreements between data producers and consumers about the schema, quality, and semantics of data. They're critical for building reliable data pipelines and preventing downstream issues caused by unexpected data changes. Great Expectations (GX) and Soda are two powerful open-source tools that help data engineers define, enforce, and monitor these data contracts, ensuring data reliability and trust within an organization.

Great Expectations allows you to define "Expectations" – assertions about your data that act as the programmatic specification of your data contract. These expectations can cover anything from column existence and data types to value ranges, uniqueness, and consistency. You embed GX validations directly into your data pipelines (e.g., after an ETL step) to immediately flag data that doesn't meet the agreed-upon contract. This proactive validation helps catch issues before they propagate. Soda, on the other hand, focuses on "Checks" defined in YAML files, which are similar to expectations but are typically run as part of a continuous data quality monitoring process, often independent of direct pipeline execution. Soda excels at detecting anomalies, tracking metrics over time, and alerting stakeholders when data quality deviates from the contract, making it perfect for ongoing data health checks.

By combining GX and Soda, you get a robust data contract enforcement system. GX ensures that data entering or transforming within a specific pipeline stage adheres to the contract, preventing bad data from moving forward. Soda then provides continuous vigilance, monitoring the data landscape for any contract breaches or quality degradation that might occur outside of explicit validation points. Together, they automate the validation of your data contracts, fostering clearer communication between teams, building trust in your data assets, and significantly reducing the time spent debugging data quality issues.

Key Takeaways

  • Data contracts define agreed-upon data schema, quality, and semantics.
  • Great Expectations (GX) uses "Expectations" for explicit, in-pipeline data contract validation.
  • Soda uses "Checks" for continuous data quality monitoring and alerting on contract breaches.
  • Both tools help codify data contracts, automating enforcement and improving data reliability.
  • Integrating GX and Soda reduces manual data quality checks and fosters trust in data assets.

Code Example

python
import great_expectations as gx

# Assume 'validator' is an object connected to your data source 
# (e.g., a Pandas DataFrame, Spark DataFrame, or a database table)
# In a real scenario, this would be obtained from a Great Expectations DataContext.

# Example: Defining expectations (contract terms) for a 'users' dataset
validator = gx.validator.Validator()

# Contract term 1: The 'user_id' column must exist, not be null, and be unique.
validator.expect_column_to_exist("user_id")
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_be_unique("user_id")

# Contract term 2: The 'email' column must exist and match a valid email regex.
validator.expect_column_to_exist("email")
validator.expect_column_values_to_match_regex("email", r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")

# After defining, these expectations can be saved as a 'suite' and run
# against actual data batches to validate the contract.

How this code works

This code defines a "data contract" using the Great Expectations library, specifying rules that a users dataset must follow to be considered high-quality. It begins by importing the library and creating a validator object with gx.validator.Validator(). This validator acts as a container for all the rules, or "expectations," that will be defined for the data. At this stage, the validator is like an empty blueprint for checks, not yet connected to any live data.

The code then adds specific expectations to this validator. For the user_id column, it uses expect_column_to_exist, expect_column_values_to_not_be_null, and expect_column_values_to_be_unique to ensure the column is present, has no missing entries, and contains only distinct IDs. For the email column, it again checks for existence with expect_column_to_exist, then uses expect_column_values_to_match_regex with a regular expression to verify that all email entries conform to a standard format. A subtle point is that this validator object is initially empty and does not inherently know about a dataset; its purpose here is purely to define and store the contract terms. These defined expectations can later be saved as a "suite" and run against actual data to perform validation.

Related Articles

Hand-picked reads to go deeper on data testing & validation.

Back to Data Engineer Roadmap