Phase 1: Foundations

Branching, merging & pull request workflows

Beginner ~4 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine your favorite family recipe book. Maybe it has the perfect recipe for chocolate chip cookies that everyone loves. This recipe is really important; you don't want anyone to accidentally change it or mess it up, especially if it's used every week!

Now, what if you have a cool idea? Maybe you want to try adding sprinkles to the cookies, or use a different kind of chocolate chip, or even try to make them gluten-free for a friend. You wouldn't want to scribble all your new ideas directly into the main family recipe book, right? What if your new version doesn't turn out well? You'd have ruined the original perfect recipe for everyone!

That's exactly what a special trick called 'branching' helps us do in coding. Instead of changing the main recipe, you'd grab a fresh piece of paper. On this new paper, you'd copy the original chocolate chip cookie recipe. This new piece of paper, with your copy, is like your own 'branch' – it’s a separate, independent version. You can now experiment all you want on your paper: add those sprinkles, try new chocolate, or change ingredients for gluten-free cookies. The original recipe book stays safe and sound, just like before.

This way, you get your very own workspace, or a 'sandbox' as grown-ups call it, to play and experiment. You can test if your sprinkle cookies are delicious, or if your gluten-free ones actually work, all without anyone worrying that you're going to break the main, stable recipe. If your new recipe is fantastic, great! You might then decide to carefully add it back to the main book (that's called 'merging', but we can talk about that another time). If it's a total disaster, no problem! You just toss your experimental paper, and the main recipe is still there, perfect and untouched.

So, when grown-ups are building big data projects, they use branching just like you would with your recipe. This means someone can be working on making a new report for a client, while another person is fixing a tiny mistake in a different part, and neither of them will accidentally mess up the main project that everyone else is using. It keeps everything neat, organized, and safe!

Imagine your data project's code (like SQL scripts, Python ETL, or dbt models) lives on a "main" line. When you want to develop a new feature, fix a bug, or experiment with a different data transformation approach, you don't want to mess with that stable "main" line directly. That's where branching comes in. A branch is essentially a separate, independent line of development. You create a new branch, like making a copy of your entire project, and work on your changes there. This allows you to develop new parts of an ETL pipeline, test a new data model, or refactor existing scripts without impacting the operational version. It's your personal sandbox for changes, keeping the main project stable for everyone else.

Once you've completed your work on your branch – say, you've successfully built a new data transformation and verified its output – you'll want to integrate those changes back into the main project. This process is called merging. Merging combines the history and changes from your development branch into another branch, usually your main or develop branch. Git is usually smart enough to automatically combine these changes. Sometimes, if two people changed the same line of code independently, Git might ask for your help to resolve a "merge conflict," which just means you need to tell Git which change to keep. Merging is how your isolated work becomes part of the shared, stable codebase.

For team projects, especially in data engineering where quality and data integrity are paramount, you don't just merge your branch directly. Instead, you use a Pull Request (PR) (also known as a Merge Request in some systems). A PR is a formal proposal to merge your changes from your branch into a target branch. It's not just a technical step; it's a collaborative workflow. When you open a PR, you're asking your teammates to review your code, logic, and potential impact. Other data engineers can check your SQL queries for efficiency, your Python scripts for best practices, or your dbt models for correctness before they get integrated. This review process is crucial for catching errors early, ensuring data quality, and maintaining code standards, ultimately leading to more robust data pipelines and analytics products.

Key Takeaways

  • Branching allows isolated development without affecting the main project.
  • Merging combines changes from a feature branch back into a primary branch.
  • Pull Requests facilitate code review and collaboration before merging.
  • These workflows are essential for maintaining data quality and team efficiency in data engineering projects.

Code Example

bash
# 1. Create a new branch for your feature (e.g., adding a new ETL step)
git checkout -b feature/add-new-etl-step

# 2. Make some changes (e.g., create a new SQL file or update a Python script)
echo "SELECT * FROM new_source_table;" > dbt_models/new_etl_step.sql
# In a real scenario, you'd modify files in your editor or IDE

# 3. Add and commit your changes to your feature branch
git add .
git commit -m "feat: Add initial dbt model for new ETL step"

# 4. Switch back to the main branch (or the branch you want to merge into)
git checkout main

# 5. Merge your feature branch into main
# (In a real team scenario, you'd push your branch and open a Pull Request first)
git merge feature/add-new-etl-step

# 6. Optionally, delete the feature branch after successful merge
git branch -d feature/add-new-etl-step

How this code works

This code demonstrates a fundamental Git workflow for developing a new feature, like adding an ETL step, without directly altering the main project history. It starts by creating a dedicated development space with git checkout -b feature/add-new-etl-step, allowing isolated work. Inside this new branch, changes are simulated, such as creating a dbt SQL file using echo "SELECT * FROM new_source_table;" > dbt_models/new_etl_step.sql. These modifications are then formally recorded within the feature branch's history using git add . to stage them and git commit -m "feat: Add initial dbt model for new ETL step" to save them permanently.

After the feature is complete and committed, the workflow integrates it back into the primary codebase. First, the user switches back to the main development line using git checkout main. Then, git merge feature/add-new-etl-step incorporates all the changes from the feature branch directly into main. A subtle but crucial point for beginners is that while this example merges directly, in real team environments, a Pull Request would typically be opened on the feature/add-new-etl-step branch for review and approval before merging into main. Finally, git branch -d feature/add-new-etl-step removes the now-obsolete feature branch, keeping the repository clean.

Related Articles

Hand-picked reads to go deeper on git & version control for data.

Back to Data Engineer Roadmap