Phase 1: Programming & AI Foundations

Branching strategies for experiments & production code

Beginner ~14 min read
Prerequisites
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you and your friends are building a giant, amazing LEGO castle together. Everyone has super creative ideas! One friend wants a spooky dungeon, another a working drawbridge, and you're dreaming up a secret treasure room. If everyone just started building on the same castle, it could get messy, right? Bits pulled off, pieces misplaced. Instead, wouldn't it be great if everyone had their own little building area where they could try out their ideas without messing up the main castle or each other's work? That's what a "branching strategy" helps us do when we build computer programs. It lets everyone work on their ideas safely, all at once.

In our LEGO world, some parts of the castle are super important. The main tower, for instance – that’s like our "main" program, always complete and sturdy. Then, there are new things that need to be added, like your friend building that drawbridge. That's a clear addition, a "feature," that will join the castle. But sometimes, you just want to experiment. Maybe you want to see what the castle would look like with a rainbow roof, or if a catapult could launch a tiny LEGO figure. These are "experiments." You might build them, decide they look silly and throw them away, or they might take ages. Experiments might never make it into the final castle.

A good plan for building our LEGO castle means having different places for different types of work. You'd have one big table for the main castle – always clean, always perfect and ready to show off. Then, a separate medium-sized table where you combine finished parts, like that working drawbridge, making sure it fits perfectly before adding it to the main castle. And for all those crazy experiments – the rainbow roof, the catapult – you'd have lots of small, temporary building mats. You can try anything there, make a huge mess, or even decide to smash it and start over, without affecting the main castle or parts almost ready.

So, when you're building computer programs, especially complex "AI (Artificial Intelligence)" programs, this idea of different "building areas" is super useful. It means you can always have a version of your AI program that works perfectly and is ready to share. At the same time, you and your friends can be trying out wild new ideas – like teaching your AI a new trick or making it smarter – without risking the main program. If you're building by yourself, you might not need quite so many separate tables. But if you're working with a big team, this clear system helps everyone build amazing things together without chaos.

The core mental model for Git branching is a directed acyclic graph (DAG) of commits, where a branch is just a named pointer to a node in that graph. When you create a branch, nothing is copied -- Git just adds a pointer. When you commit, that pointer moves forward. Merging takes two pointers and creates a new commit that has both as parents. This is cheap, and that cheapness is exactly why branching-per-experiment is practical in ways it would not be in, say, a traditional filesystem-based workflow.

For an AI project, the most widely used strategy is a simplified version of Gitflow: main is production, develop is the integration branch where finished features land before they ship, and all work happens on short-lived branches off of develop. The AI-specific addition is a third branch type: experiment/. An experiment branch is not the same as a feature branch. A feature branch is expected to merge. An experiment branch might not -- it exists to isolate a hypothesis test so you can compare approaches without polluting your integration history. When you run an eval comparing two prompt templates, you want that on an experiment branch so you can share the results (push the branch, link a teammate to the eval output) without accidentally shipping half-finished work.

Here is how a pro approaches a real scenario. You are building a customer support RAG pipeline. Your current setup uses text-embedding-ada-002 and a chunk size of 512 tokens. You suspect text-embedding-3-small with 256-token chunks might give better retrieval precision on short user queries. You create experiment/embeddings-3-small-256chunk off of develop. You update the pipeline config, run your eval harness against a golden question set, and commit eval_results.json with the scores. If the results are worse, you push the branch so the results are preserved for posterity, then delete the branch and move on. If the results are better, you open a pull request from the experiment branch into develop and treat the eval results as part of the PR description. You do not squash the experiment history -- reviewers want to see the intermediate commits that show you tried 128-token chunks first and why you moved to 256.

The tradeoffs between strategies come down to team size and merge frequency. Gitflow (main + develop + feature/hotfix/release branches) is explicit and safe but adds ceremony. Trunk-based development (everyone commits directly to main with feature flags) is fast but requires mature CI and confidence in your test suite, which is hard when your "tests" are LLM eval pipelines that take 20 minutes to run. GitHub Flow (main + short-lived feature branches, no develop) is a reasonable middle ground for small AI teams: everything merges to main, releases are tagged, and experiment branches are first-class citizens alongside feature branches. For most teams building AI applications (not training models from scratch), GitHub Flow with an explicit experiment/ naming convention is the sweet spot.

At 10 users, your branching strategy barely matters. At 10k users, you need to be able to hotfix production without merging your half-finished RAG rewrite. That means main must be clean. At 10M users, you need release branches so you can patch a 6-week-old release while the new version is still in staging. The key scaling concern for AI projects specifically is that model versions and prompt versions need to be traceable through your Git history. Tag your releases with enough metadata (model name, embedding model, prompt hash) that you can answer "what exact config was serving users on November 3rd?" without guessing. A git tag like v1.4.2-gpt4o-ada002 is crude but honest.

Key Takeaways

  • Keep main always deployable; never commit directly to it.
  • Create a short-lived experiment branch per hypothesis, not per project phase.
  • Merge experiment results as a summary commit even when you discard the branch.
  • Tag every production release so you can reproduce the exact model config that shipped.

Pro tips

  • Name experiment branches with the hypothesis, not the technique. experiment/smaller-chunks-better-recall is searchable in three months; experiment/embeddings-v2 is not.
  • Even when you delete an experiment branch, push it first and record the outcome in a short commit message on main using --no-ff. That merge commit becomes a permanent, searchable record of what you tried and why you moved on.
  • Prompt templates are code. If you iterate on a system prompt during an experiment, commit the exact text to the branch -- do not leave it in a Notion doc. Six months later you will need to know exactly what was running when a recall metric dropped.
  • Use git worktree to run two experiments simultaneously on the same machine without stashing or switching branches. git worktree add ../exp-claude3 experiment/claude3-eval gives you a second working directory pointing at a different branch.

Common pitfalls

  • Mistake: Keeping experiment branches alive for months while they accumulate diverging commits. Fix: Time-box experiments to two weeks max. If unresolved, archive results and delete the branch.
  • Mistake: Committing .env files or API keys to any branch, assuming it won't get merged. Fix: Use .gitignore from day one and pre-commit hooks to block secrets on every branch, not just main.
  • Mistake: Squash-merging experiment branches so all context is lost. Fix: Use --no-ff merge for experiments so the commit graph shows what was tried; reserve squash for noisy feature branches.
  • Mistake: Branching off a stale base so experiment results are not comparable. Fix: Always git pull origin develop and rebase before starting an experiment to ensure a clean, current baseline.

Which branching strategy to use for AI projects

Option Use when Avoid when
GitHub Flow (main + short-lived branches) Small team (1-4 people), frequent deploys, CI gates are fast enough to protect main. You need to maintain multiple release versions simultaneously or your eval pipeline takes > 30 minutes.
Gitflow (main + develop + feature/release/hotfix) Larger team, scheduled releases, need hotfix capability without touching in-progress features. Solo project or team that deploys multiple times per day -- the ceremony will slow you down.
Trunk-based development Mature team with strong CI, feature flags, and very fast automated evals that give confidence before merge. Your quality gate is a human-reviewed LLM eval that takes hours -- merging to trunk that often is too risky.
Experiment branches (add-on to any strategy) Any time you are testing a hypothesis (model swap, prompt change, chunking strategy) that may not ship. The change is obviously going to merge and is small enough to review as a regular feature branch.

Code Example

bash
# git 2.39+
# Create a new experiment branch from main and push it
git checkout main
git pull origin main

# Name the branch to encode context: what you're testing and when
git checkout -b experiment/gpt4o-vs-claude3-chunking-2024-11

# Do your work, commit often with descriptive messages
git add eval_results.json prompt_v2.txt
git commit -m "eval: GPT-4o scores 0.82 ROUGE vs Claude-3 0.79 on QA set"

# Push so teammates can see the branch (and results) without merging
git push -u origin experiment/gpt4o-vs-claude3-chunking-2024-11

# When the experiment is done, record the outcome before deleting
git checkout main
git merge --no-ff experiment/gpt4o-vs-claude3-chunking-2024-11 -m "[experiment-result] gpt4o wins on QA; adopting in feature/rag-pipeline"
git branch -d experiment/gpt4o-vs-claude3-chunking-2024-11

How this code works

This code outlines a systematic workflow for conducting experiments within a Git repository. It starts by synchronizing the local main branch with git checkout main and git pull origin main, ensuring work begins from the latest stable code. An experiment-specific branch is then created using git checkout -b experiment/..., with a name that clearly encodes the experiment's context and timing. As work progresses, changes like updated evaluation results or prompt files are regularly saved using git add and git commit -m "...". The git push -u origin ... command shares this active experiment branch, allowing teammates to view progress or results without requiring a merge into the main development line.

Upon completion, the experiment's findings are integrated and recorded. The workflow returns to main and merges the experiment branch using git merge --no-ff. The --no-ff (no fast-forward) option is a subtle but important choice; it forces Git to create a dedicated merge commit, explicitly marking the integration point of the experiment's outcome into main, even if a simpler fast-forward merge was possible. This provides a clear, traceable history of the experiment's conclusion. The merge commit message summarizes the experiment's results and any next steps. Finally, git branch -d removes the local experiment branch, maintaining a clean repository history.

Production-grade example

CI script enforcing naming conventions, blocking direct main pushes, and requiring eval results on experiment branches.

bash
#!/usr/bin/env bash
# branch-guard.sh -- enforce branching rules in CI (GitHub Actions or similar)
# Requires: git 2.39+, bash 5+
# Usage: source this file or call functions individually

set -euo pipefail

BRANCH="${GITHUB_HEAD_REF:-$(git rev-parse --abbrev-ref HEAD)}"
BASE_BRANCH="${GITHUB_BASE_REF:-main}"
LOG_PREFIX="[branch-guard]"

log() { echo "${LOG_PREFIX} $*" >&2; }

validate_branch_name() {
  local branch="$1"
  # Allowed patterns: feature/, bugfix/, experiment/, hotfix/, release/
  local pattern='^(feature|bugfix|experiment|hotfix|release)/[a-z0-9._-]+$'
  if [[ ! "$branch" =~ $pattern ]]; then
    log "ERROR: Branch '${branch}' does not match naming convention."
    log "Expected: feature/*, bugfix/*, experiment/*, hotfix/*, release/*"
    exit 1
  fi
  log "OK: Branch name '${branch}' is valid."
}

guard_direct_main_push() {
  # Fail if someone tries to push directly to main or develop
  local protected=("main" "develop")
  for protected_branch in "${protected[@]}"; do
    if [[ "$BRANCH" == "$protected_branch" ]]; then
      log "ERROR: Direct push to '${protected_branch}' is not allowed."
      log "Open a pull request instead."
      exit 1
    fi
  done
}

check_experiment_has_results() {
  # Experiment branches must include an eval_results file
  if [[ "$BRANCH" == experiment/* ]]; then
    if ! git diff --name-only "origin/${BASE_BRANCH}...HEAD" | grep -q 'eval_results'; then
      log "WARNING: Experiment branch '${BRANCH}' has no eval_results file committed."
      log "Add eval_results.json or eval_results.md before merging."
      # Warn only -- do not block. Teams can escalate to exit 1.
    else
      log "OK: eval_results file found on experiment branch."
    fi
  fi
}

log "Validating branch: ${BRANCH}"
validate_branch_name "$BRANCH"
guard_direct_main_push
check_experiment_has_results
log "Branch validation passed."

How this code works

This branch-guard.sh script acts as a gatekeeper for Git branches, especially when integrated into automated systems like GitHub Actions. Its primary job is to enforce naming conventions and workflow rules, ensuring code changes integrate smoothly and follow team standards for both experimental and production work. It helps maintain code quality by preventing direct pushes to critical branches and ensuring specific branch types include necessary documentation.

The script begins with set -euo pipefail for robust, error-safe execution. It determines the current BRANCH and BASE_BRANCH, using the :- operator to provide sensible defaults if these variables aren't set, making the script flexible for different environments. The validate_branch_name function then checks if the branch name follows a predefined pattern, such as feature/ or experiment/. Next, guard_direct_main_push prevents direct commits to protected branches like main, guiding changes through pull requests instead. Finally, check_experiment_has_results specifically looks for an eval_results file within experiment/ branches, issuing a warning if it's missing. This allows teams to set expectations for experimental work without necessarily blocking progress immediately. The script executes these checks sequentially to validate the branch.

Practice & master

Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.

Exercise

Set up a local Git repo simulating an AI project branching strategy. Create a main branch and a develop branch. Then create an experiment branch that tests two different "prompt versions" (just text files). Commit eval results to the experiment branch, then merge it back to develop using --no-ff. Inspect the resulting commit graph.

bash
#!/usr/bin/env bash
# Setup: run these commands one by one in your terminal

# 1. Initialize a new repo
mkdir ai-branch-demo && cd ai-branch-demo
git init
git commit --allow-empty -m "initial commit"

# 2. TODO: Create a 'develop' branch from main

# 3. TODO: Create an experiment branch named experiment/prompt-ab-test off develop

# 4. On the experiment branch:
#    - Create prompt_v1.txt with some text
#    - Create prompt_v2.txt with different text
#    - Create eval_results.json: { "v1_score": 0.74, "v2_score": 0.81 }
#    - Commit all three files

# 5. TODO: Switch back to develop and merge the experiment branch with --no-ff

# 6. TODO: View the commit graph
# Hint: git log --oneline --graph --all

Quick check

  1. Your eval pipeline takes 45 minutes to run. A teammate wants to adopt trunk-based development. What is the main risk?

  2. An experiment branch produced worse results than the baseline. What should you do before deleting it?

  3. Which naming convention gives the most useful context when reviewing branches six months later?

Self-check: Without looking at your notes, describe the difference between a feature branch and an experiment branch in an AI project, explain what git merge --no-ff does and why it matters for experiment history, and name one scenario where trunk-based development would be a bad fit.

Resources

Related Articles

Hand-picked reads to go deeper on git & version control.

Back to AI Software Developer Roadmap