Phase 4: Data Quality & Governance

DataHub, Amundsen & Atlan catalog tools

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine our school had the most amazing, gigantic library in the world! It wouldn’t just have books; it would have every science kit, every robot, every VR game, and every old school yearbook. It’s incredible! But with so much stuff, how would you ever find that one specific book about space your teacher recommended, or the art kit with special paints, or the robot that teaches coding? It would be like searching for a tiny pin in a huge haystack – almost impossible to find what you need quickly.

That's where a super-smart digital library catalog comes in. This catalog doesn't hold the actual books or robots. Instead, it knows everything about them. We call this "metadata." For a book, the metadata is its title, author, topic (like space!), where it's on the shelf, and even who recommended it. This super catalog helps you search and understand what’s available in the library without looking at every single item. It’s like having a magical map to all the amazing treasures, making finding things easy!

Now, think of DataHub, Amundsen, and Atlan as different versions of this super library catalog, each with special powers. DataHub is like a catalog fantastic at tracking the "journey" of items. If a project uses parts from Kit A and creates a report, DataHub shows how Kit A led to it. It helps understand how everything connects. Amundsen is like a catalog designed to be super easy and fun to search. You just type "space," and it immediately shows the best space books, popular documentaries, or VR games, all beautifully organized. It helps quickly find exactly what you need. Atlan is like the catalog that knows not just where things are, but who is responsible for them, who is allowed to use them, and if they're "official" school-approved items. It keeps everything organized and safe.

So, when grown-ups (who we call Data Engineers) are trying to keep track of literally millions of digital "items" – like customer lists, sales figures, or research projects – they use these special catalog systems. This means they can quickly find a specific report, understand where a piece of information came from, or know who is responsible for a set of numbers. It helps everyone in a big company find and trust the information they need to do their jobs, just like you’d find and trust the perfect book for your school project!

As a Data Engineer, navigating the ever-growing landscape of data sources and ensuring data quality is paramount. Data Catalogs are essential tools that centralize metadata, making data discoverable, understandable, and trustworthy. DataHub, Amundsen, and Atlan are three prominent solutions, each with distinct strengths, that help organizations achieve this. They act as a single source of truth for your data assets, providing context like schemas, descriptions, owners, and lineage, which are critical for efficient data operations and robust data governance.

DataHub (developed by LinkedIn, open-source) excels in providing a robust, extensible metadata platform with strong capabilities for data lineage tracking. It allows Data Engineers to ingest metadata from a wide array of sources (databases, data lakes, BI tools) and build a rich graph of data assets and their relationships. Its strength lies in its active community and API-first approach, making it highly customizable and suitable for integrating deeply into existing data ecosystems. Amundsen (developed by Lyft, open-source) focuses heavily on data discovery and user experience. It offers an intuitive search interface, helping users quickly find relevant datasets, understand their context, and determine their reliability. While it provides solid metadata management, its primary emphasis is on making data exploration seamless for data consumers. Lastly, Atlan (commercial SaaS) positions itself as a comprehensive "data operating system" rather than just a catalog. It integrates active governance features like data quality monitoring, access control, and collaborative workflows directly into the catalog experience. Atlan offers a more polished, enterprise-ready solution, often chosen by organizations seeking an all-in-one platform for data discovery, governance, and collaboration.

From a Data Engineer's perspective, these tools are invaluable for several practical reasons. You'll use them to programmatically ingest metadata, verify upstream and downstream dependencies through data lineage, quickly locate datasets for new projects, or understand the impact of schema changes. They empower you to attach crucial business context, such as data owners, quality metrics, and usage statistics, directly to the technical metadata. Ultimately, by providing a comprehensive view of your data landscape, DataHub, Amundsen, and Atlan reduce data silos, improve data literacy across the organization, and ensure that data is trusted and utilized effectively.

Key Takeaways

  • DataHub (open-source) is strong in metadata ingestion, customizability, and data lineage tracking.
  • Amundsen (open-source) focuses on an intuitive search and discovery experience for data users.
  • Atlan (commercial SaaS) offers a comprehensive 'data operating system' with integrated active governance, collaboration, and data quality features.
  • All three centralize metadata, enabling better data discovery, understanding, and trust within an organization.
  • Data Engineers use these tools for programmatic metadata management, impact analysis via lineage, and ensuring data quality and governance.

Code Example

python
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.metadata.schema_classes import GlobalTagsClass, TagAssociationClass

# Configure the DataHub GMS endpoint
emitter = DatahubRestEmitter(gms_server="http://localhost:8080")

# Define the URN of an existing dataset to tag
dataset_urn = "urn:li:dataset:(urn:li:dataPlatform:snowflake,my_database.my_schema.my_table,PROD)"

# Define the tag URN to add (ensure 'PII' tag exists in DataHub)
tag_to_add_urn = "urn:li:tag:PII"

# Create a MetadataChangeProposal to add the tag
tag_association = TagAssociationClass(tag=tag_to_add_urn)
global_tags = GlobalTagsClass(tags=[tag_association])

mcp = MetadataChangeProposalWrapper(
    entityUrn=dataset_urn,
    aspect=global_tags,
)

# Emit the metadata change to DataHub
emitter.emit_mcp(mcp)
print(f"Successfully added tag '{tag_to_add_urn}' to dataset '{dataset_urn}'")

How this code works

This Python code’s job is to programmatically add a predefined tag, specifically "PII," to an existing dataset within DataHub. This is a common task in data governance, allowing data engineers to mark sensitive tables for easier discovery and compliance.

The process begins by configuring a DatahubRestEmitter to connect to the DataHub metadata service, typically running locally. It then identifies the target dataset using its dataset_urn and specifies the tag_to_add_urn for "PII." To prepare the update, a TagAssociationClass is created for the tag, which is then wrapped in a GlobalTagsClass to represent the dataset’s overall tag collection. This structured information is then encapsulated in a MetadataChangeProposalWrapper, which is the standardized format DataHub expects for metadata updates. A subtle point for beginners is that the tag_to_add_urn like urn:li:tag:PII must already exist in DataHub; this script associates an existing tag, it doesn't create new ones. Finally, the emitter.emit_mcp(mcp) method sends this proposal to DataHub, applying the tag to the specified dataset.

Related Articles

Hand-picked reads to go deeper on data catalogs & discovery.

Back to Data Engineer Roadmap