Phase 4: Data Quality & Governance

Freshness, volume & schema change monitoring

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're baking your favorite chocolate chip cookies. To make sure they turn out yummy every time, you need to check three super important things about your ingredients and your recipe.

First, there's freshness. You wouldn't want to use old, expired milk or eggs, right? If you did, your cookies might not rise, or they could taste really bad. In the world of computers, we do something similar with information, or "data." We check if the data is fresh and new, just like we expect. If the newest information is too old, or if we didn't get the new information we were waiting for (like the latest batch of chocolate chips didn’t arrive at the store), it's like finding out your milk expired yesterday – big problem! We need the freshest stuff for our computer "recipes" to work correctly and make good decisions.

Next up is volume, which just means "how much" of something there is. When you're baking, you need the right amount of flour, sugar, and chocolate chips. What if you only had half a cup of flour when the recipe asked for two? Or suddenly found yourself with ten bags of sugar when you only needed one? Both would be weird, right? Too little flour means tiny, sad cookies, and too much sugar could make them sickly sweet. For computers, we also keep an eye on how much data we're getting. If we suddenly get way less information than we normally do, it might mean something broke upstream, like the store didn't send all the ingredients. Or if we get way more data than usual, it could mean something unexpected happened. Checking the volume helps us catch these surprises early.

Finally, there's schema change. Think of the schema as your actual recipe – the step-by-step instructions and list of ingredients for your cookies. What if, overnight, someone secretly changed your recipe? Maybe they changed "add two cups of sugar" to "add two cups of salt," or they completely removed the step to add the eggs, or they changed "bake for 10 minutes" to "bake for 10 hours"? Your cookies would be totally ruined, even if your ingredients were fresh and you had the right volume! The structure of the recipe, its blueprint, got messed up. In the computer world, data also has a structure – like a list of names, numbers, or dates. If this structure suddenly changes without anyone knowing, like a column of numbers suddenly becoming a column of text, or a whole important section of data disappearing, it can break everything else that relies on it.

So, a data engineer's job is a bit like a super careful chef or baker. We set up special alarms and checks for our computer systems. These alarms tell us: "Hey, the data isn't fresh!" or "Whoa, there's too little (or too much) data!" or "Uh oh, the recipe (schema) changed unexpectedly!" By keeping a close eye on the freshness, volume, and structure of our data, we can make sure all the amazing computer programs and reports that use this data are always working perfectly, just like your cookies always come out delicious and exactly how you wanted them. This means you can trust the information and the cool things computers do with it.

As a Data Engineer, ensuring the reliability and quality of data is paramount, and this often starts with understanding its most fundamental characteristics: freshness, volume, and schema. These three pillars form the bedrock of data observability, providing critical insights into the health of your data pipelines and assets. Freshness monitoring tracks how up-to-date your data is, alerting you when expected data hasn't arrived or when the latest records are older than anticipated. This directly impacts the timeliness of reports and machine learning models. Volume monitoring, on the other hand, keeps an eye on the quantity of data flowing through your systems, identifying sudden, unexpected drops or spikes in row counts or file sizes that could indicate upstream data issues, incomplete loads, or even malicious activity. Both are crucial for maintaining trust in your data.

Schema change monitoring focuses on the structural integrity of your data. Data schemas are the blueprints that define the organization and types of data within a table or file. Unexpected alterations—like a column being dropped, renamed, or having its data type changed—can silently break downstream applications, dashboards, or ETL jobs that rely on a specific data structure. Proactively monitoring for these changes allows you to detect them before they cause widespread failures, giving you time to adapt dependent systems or revert unintended modifications. Integrating these monitoring practices into your data lineage tools enhances your ability to trace the impact of such changes across your entire data ecosystem.

Implementing these monitoring mechanisms transforms you from reacting to data issues to proactively preventing them. Whether through custom scripts, dedicated data quality frameworks like Great Expectations or Soda Core, or features within tools like dbt, the goal is to establish baselines, define thresholds, and set up automated alerts. This proactive approach ensures data consumers can trust the information they receive, enabling more reliable decision-making and robust data products. By combining these three monitoring aspects, you gain a comprehensive view of your data's health, which is essential for any modern data engineering practice.

Key Takeaways

  • Freshness, volume, and schema monitoring are fundamental data observability practices.
  • Freshness ensures data timeliness; volume tracks quantity anomalies; schema change monitoring safeguards structural integrity.
  • Proactive monitoring with automated alerts helps prevent downstream data quality issues and silent failures.
  • Unexpected drops/spikes in volume or schema changes can silently break dependent systems.
  • Tools like custom scripts, Great Expectations, or dbt tests can be used to implement these monitoring checks.

Code Example

sql
-- SQL example for freshness and volume monitoring
SELECT
    CURRENT_TIMESTAMP() AS monitoring_time,
    MAX(transaction_timestamp) AS latest_data_timestamp,
    COUNT(*) AS total_rows,
    CURRENT_TIMESTAMP() - MAX(transaction_timestamp) AS data_freshness_lag_interval
FROM
    your_database.your_schema.transactions
WHERE
    transaction_date >= CURRENT_DATE - INTERVAL '7' DAY; -- Focus on recent data for efficiency

How this code works

This SQL code serves as a monitoring query to assess the freshness and volume of a transactions table. Its job is to quickly report how current the data is and how many records have been processed recently, offering vital insights into data quality. The query uses SELECT to gather several key metrics: monitoring_time captures when the check was performed, latest_data_timestamp identifies the most recent transaction entry, and total_rows counts the number of relevant records. Finally, data_freshness_lag_interval is calculated to show the exact time difference between the current moment and the latest transaction, quantifying data staleness.

The FROM clause targets the your_database.your_schema.transactions table. The MAX(transaction_timestamp) function is crucial for pinpointing the latest_data_timestamp within the dataset being examined. Meanwhile, COUNT(*) provides the total_rows representing the volume. A subtle but important detail is the WHERE clause: transaction_date >= CURRENT_DATE - INTERVAL '7' DAY. This clause significantly optimizes the query by limiting the scope to only the last seven days of data. This means total_rows and latest_data_timestamp will specifically reflect data within that recent week, not the entire historical table. If the entire table stopped receiving data more than 7 days ago, this query might not immediately show a problem as it only looks at the recent past, potentially giving a false sense of security about overall freshness if new data has completely stopped flowing beyond the window.

Related Articles

Hand-picked reads to go deeper on data lineage & observability.

Back to Data Engineer Roadmap