Phase 2: Data Storage

Data cataloging & schema evolution

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine your enormous collection of LEGO bricks is all mixed up in one giant box. You have thousands of pieces – red, blue, flat, tall, wheels, even special minifigure parts. It’s hard to build something specific, like a rocket ship, if you can’t quickly find the five gray sloped roof bricks you need. This is why "data cataloging" is so useful! It's like having a super-smart "LEGO Inventory Book." This book lists every brick you own: what kind it is (e.g., "2x4 red brick"), how many you have, and where it lives (e.g., "in the red bin"). It might even have notes about what you usually build with those bricks, like "these are for houses" or "these are for cars." This book helps you easily find any brick, understand what it's for, and know exactly what you have available for your next big creation, turning your messy pile into an organized treasure chest.

Now, imagine you’ve used your organized LEGOs to build a magnificent pirate ship, complete with sails and cannons. But what if you get a brilliant new idea? You want to add a secret treasure room below deck, or upgrade your cannons to fire further. This is where "schema evolution" comes in. Think of your ship's original design – its blueprint – as its "schema." It shows exactly what bricks were used and where they fit. Schema evolution is all about how you can change that blueprint, or 'evolve' your design, without having to tear down the entire ship and start over. It means you can carefully add new sections, swap out old cannons for new ones, or even change the type of bricks you used for the sails, all while keeping your ship strong and sailing smoothly. It's about being flexible and adaptable, so your creations can grow and change with your imagination without breaking.

So, how do these ideas help a "Data Engineer" – someone who builds with data? The "LEGO Inventory Book" (data catalog) helps them find all the information needed to start a new project, like building a giant data-robot. If that robot then needs a new feature, like a stronger arm or a special new sensor (that's schema evolution!), the engineer knows exactly how to add those new parts to the robot's design without breaking it. This means when building big data systems, you can easily adapt them to new ideas, ensuring everything stays organized and works perfectly, just like your best LEGO creations!

For a Data Engineer working with Data Lakes and Lakehouses, two critical concepts are data cataloging and schema evolution. Data cataloging is essentially creating a centralized inventory of all your data assets within the data lake. Imagine a massive library where books are dumped without any organization; finding a specific book (or dataset) would be impossible. A data catalog solves this by scanning your storage (e.g., S3, ADLS), extracting metadata like filenames, folders, and inferred schemas, and allowing manual tagging and descriptions. This makes data discoverable, understandable, and provides crucial context for data governance, lineage tracking, and ensuring data quality, transforming a chaotic lake into an organized resource.

Schema evolution, on the other hand, deals with the inevitable reality that data structures change over time. Source systems update, new requirements emerge, and columns are added, removed, or have their data types modified. In a traditional data warehouse, these changes could be costly and break existing processes. In a data lake or lakehouse environment, you need the flexibility to adapt without downtime or complex migrations. Schema evolution is the capability to gracefully handle these changes, ensuring that new data with an updated schema can coexist and integrate with historical data, and that downstream applications continue to function correctly.

Modern Lakehouse formats like Delta Lake, Apache Iceberg, and Apache Hudi are designed with robust schema evolution capabilities built-in. They provide transactional guarantees and allow for operations like adding new columns (often automatically), evolving data types, and even dropping columns in a controlled manner. A good data catalog integrates tightly with these Lakehouse formats, not only discovering the initial schema but also tracking and publishing the evolved schemas over time. This synergy ensures that as your data landscape evolves, it remains discoverable, usable, and governed, preventing your data lake from becoming a 'data swamp' of unfindable and misunderstood information.

Key Takeaways

  • Data cataloging creates a searchable inventory of data, essential for discovery and governance in Data Lakes.
  • Schema evolution is the process of safely changing dataset structures (adding/dropping columns, changing types) over time.
  • Lakehouse formats (Delta, Iceberg, Hudi) natively support robust schema evolution, preserving data integrity.
  • A catalog tracks and publishes evolved schemas, keeping your data understandable and usable.
  • Together, they prevent 'data swamps' by making evolving data discoverable and reliable.

Code Example

python
from pyspark.sql import SparkSession
from pyspark.sql.functions import lit

spark = SparkSession.builder.appName("SchemaEvolution").getOrCreate()
delta_path = "/tmp/delta_evolution_example"

# 1. Write initial data to a Delta table
df_initial = spark.createDataFrame([("Alice", 1), ("Bob", 2)], ["name", "id"])
df_initial.write.format("delta").mode("overwrite").save(delta_path)

# 2. Simulate new data with an additional column 'city'
df_new_data = spark.createDataFrame([("Charlie", 3), ("David", 4)], ["name", "id"])
                   .withColumn("city", lit("Unknown"))

# 3. Append new data, allowing schema evolution with mergeSchema option
df_new_data.write.format("delta").mode("append") \
    .option("mergeSchema", "true") \
    .save(delta_path)

# 4. Read the evolved table and print its schema
spark.read.format("delta").load(delta_path).printSchema()

How this code works

This code illustrates how Delta Lake tables gracefully handle schema evolution, allowing new columns to be added to existing datasets over time. It starts by setting up a SparkSession and defining a delta_path for the table. An initial DataFrame, df_initial, with "name" and "id" columns, is then created and written to a Delta table using write.format("delta").mode("overwrite").save(delta_path). This establishes the first version of the table's schema.

Next, the code simulates new incoming data. A df_new_data DataFrame is prepared, but crucially, it introduces a new column named "city" using withColumn("city", lit("Unknown")), representing a schema change. To add this data and allow the table's schema to adapt, write.format("delta").mode("append") is used along with the critical option("mergeSchema", "true"). This mergeSchema option tells Delta Lake to automatically add the new "city" column to the existing table's schema, instead of throwing an error for a mismatch, which would be the default behavior for append operations without it. Finally, spark.read.format("delta").load(delta_path).printSchema() verifies that the table's schema has indeed evolved to include the "city" column.

Related Articles

Hand-picked reads to go deeper on data lakes & lakehouses.

Back to Data Engineer Roadmap