A PR is not a formality. It's a checkpoint where the team collectively decides whether the codebase is better with your change than without it. For a CRUD app, that's mostly about correctness and style. For an AI system, correctness is necessary but not sufficient. You also need to ask: does this change affect model behavior? Does it touch prompt templates, context assembly, retrieval logic, or output parsing? If yes, the PR needs evidence, not just code.
The mental model for a PR in an AI team looks like this: you have a main branch that always reflects the production-deployed system. Every change lives on a short-lived branch. When you want to merge, you create a PR that describes three things -- what you changed, why you changed it, and what evidence you have that the change is safe. That evidence is different for AI work. For a pure software change, passing tests is enough. For a prompt change, you need output samples. For a retrieval change, you need recall and precision numbers against a fixed benchmark query set. For a dependency bump, you need a smoke test that the LLM API calls still work. The PR description is where all of that lives, not in Slack, not in Notion, in the PR itself, attached permanently to the commit history.
A real-world scenario: your team is running a RAG chatbot for a SaaS product. A junior engineer opens a PR that changes the chunk size from 512 to 1024 tokens. The code diff is three lines. Without context it looks trivial. But chunk size directly affects how much context the retriever surfaces, which affects answer quality. A professional reviewer would immediately ask: was the eval suite re-run? What happened to Recall@5 on the benchmark set? Did average answer length change? Did any golden-set queries regress? The PR description should answer all of that before review even starts. A good team puts this in a PR template so nobody has to remember to ask. A .github/pull_request_template.md file with sections for "What changed," "Eval results," "Checklist" enforces this automatically every time someone opens a PR on GitHub.
On tradeoffs: some teams use trunk-based development where every commit goes directly to main behind feature flags, skipping long-lived branches entirely. This works well for pure software but is harder for AI work because AI experiments are genuinely exploratory. You often don't know if a change is an improvement until you run the eval. Gitflow with long-lived develop branches solves the isolation problem but creates merge debt. The sweet spot for most AI teams is short-lived feature branches (under three days), small PRs (under 400 lines of diff), and mandatory eval attachments for anything touching inference or retrieval. Separate your experiment branches from your production branches as covered in the branching lesson.
At scale the dynamics shift significantly. At 10 users with 2 engineers, informal review works fine -- you Slack your teammate and they look at it. At 50 engineers and 10k users, you need: branch protection rules that block merges without approvals, CI pipelines that run your eval suite on every PR, CODEOWNERS files that route LLM-related PRs to ML engineers and infrastructure PRs to platform engineers, and auto-merge policies for dependency updates that pass all checks. At 10M users with a large org, you add things like required sign-off from a safety team for any prompt changes, deployment gates that require a canary rollout before full merge, and audit logs of every PR that touched production model configuration. None of this is overkill at scale; it's how you keep a team of 100 from stepping on each other's work and accidentally shipping a prompt regression to millions of users.
Key Takeaways
- Write PR descriptions that include eval results and model version changes, not just what code changed.
- Keep AI experiment branches short-lived; stale branches diverge fast and become unmerge-able.
- Require a passing eval baseline check in CI before a PR can be approved.
- Never merge a PR that changes prompt logic without attaching before/after output samples.
Pro tips
- Put your eval numbers directly in the PR description as a Markdown table. A reviewer who has to run the eval themselves to assess impact will either skip it or block the PR indefinitely. Remove that friction at the source.
- Limit PRs that touch prompt templates to exactly that -- don't bundle a prompt change with a refactor. Mixed PRs make it impossible to attribute a quality regression to a specific change in your git history.
- Set up a GitHub PR template at
.github/pull_request_template.mdwith mandatory sections for 'Eval results' and 'Model version pins'. Blank sections in a submitted PR are a signal to the reviewer that the work isn't ready. - Use draft PRs for early feedback on AI architecture decisions. Opening a draft PR with a design sketch and no code gets async input from teammates before you've invested three days in an approach that won't survive review.
Common pitfalls
- Mistake: Merging a prompt change with no output samples attached. Fix: Add a 'Before/After outputs' section to your PR template and make it a blocking review comment if it's missing.
- Mistake: Keeping experiment branches open for two weeks until they're impossible to rebase. Fix: Time-box experiment branches to three days; if the experiment isn't ready, open a draft PR and keep rebasing daily.
- Mistake: Reviewing a large AI PR by only reading the diff without running the code. Fix: Pull the branch locally, run the eval suite, and comment on the numbers, not just the syntax.
- Mistake: Storing API keys or model weights in a PR because .gitignore wasn't set up yet. Fix: Check .gitignore before the first commit on any new branch; rotate any key that touched a PR immediately.
When to require different PR review gates
| Option | Use when | Avoid when |
|---|---|---|
| Standard code review (1 approval) | Change is purely infrastructure or tooling with no effect on model inputs or outputs. | Any prompt, retrieval, or model version pin is modified. |
| Code review + eval results (2 approvals) | Retrieval parameters, chunk size, embedding model, or re-ranking logic changes. | Team lacks a benchmark query set; you'll block PRs with no way to pass the gate. |
| Code review + safety sign-off | System prompt or output filtering logic changes for a user-facing product. | Internal tooling with no end-user exposure; overkill adds friction with no benefit. |
| Automated merge (no human review) | Dependency patch bump that passes all CI checks including the eval suite. | Minor version bumps of LLM SDKs; those often include silent behavior changes. |
Code Example
# PyGithub==2.1.1
from github import Github
import os
g = Github(os.environ["GITHUB_TOKEN"])
repo = g.get_repo("your-org/your-ai-app")
# Open a PR from a feature branch into main
pr = repo.create_pull(
title="feat: swap retrieval model to text-embedding-3-small",
body="""## What changed
- Swapped embedding model from ada-002 to text-embedding-3-small
## Eval results
| Metric | Before | After |
|--------------|--------|-------|
| Recall@5 | 0.71 | 0.74 |
| Latency p50 | 210ms | 180ms |
## Checklist
- [x] Unit tests pass
- [x] Eval suite re-run on 200-query benchmark
- [x] .env.example updated
""",
head="feat/swap-embedding-model",
base="main",
)
print(f"PR opened: {pr.html_url}")How this code works
This Python code automates the creation of a GitHub Pull Request, a key step in professional AI team workflows for proposing and reviewing code changes. Its primary job is to programmatically open a new PR on a specified repository, detailing an update to an AI model. This demonstrates how teams can integrate automated PR generation into their development pipeline, ensuring consistency and including critical information like evaluation metrics for AI model changes.
The code first establishes a connection to GitHub using a Github object, authenticating with a GITHUB_TOKEN from environment variables – crucial for security and a common point where beginners might encounter permission errors if the token lacks the necessary repo scope. It then selects the target repository using g.get_repo. The core action is repo.create_pull, which takes a descriptive title (following a feat: convention), a detailed Markdown body (including "What changed", "Eval results" comparing model performance, and a "Checklist"), and specifies the source (head) and target (base) branches for the merge. Finally, print outputs the URL of the newly created PR.
Production-grade example
Adds retries with backoff, env-var auth, timeouts, branch validation, and structured JSON logging.
# PyGithub==2.1.1 tenacity==8.2.3
import logging
import os
import time
from github import Github, GithubException
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
logging.basicConfig(
level=logging.INFO,
format='{"time": "%(asctime)s", "level": "%(levelname)s", "msg": %(message)s}',
)
log = logging.getLogger(__name__)
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"] # never hardcode
REPO_SLUG = os.environ["GITHUB_REPO"] # e.g. "your-org/your-ai-app"
@retry(
retry=retry_if_exception_type(GithubException),
wait=wait_exponential(multiplier=1, min=2, max=30),
stop=stop_after_attempt(4),
reraise=True,
)
def open_pr(title: str, body: str, head: str, base: str = "main") -> str:
g = Github(GITHUB_TOKEN, timeout=15) # 15-second socket timeout
repo = g.get_repo(REPO_SLUG)
# Graceful degradation: check the branch actually exists before trying
try:
repo.get_branch(head)
except GithubException as exc:
log.error('{"event": "branch_not_found", "branch": "%s", "status": %d}', head, exc.status)
raise
start = time.monotonic()
pr = repo.create_pull(title=title, body=body, head=head, base=base, draft=False)
latency_ms = round((time.monotonic() - start) * 1000)
log.info(
'{"event": "pr_opened", "pr_number": %d, "url": "%s", "latency_ms": %d}',
pr.number, pr.html_url, latency_ms,
)
return pr.html_url
if __name__ == "__main__":
url = open_pr(
title="feat: swap embedding model to text-embedding-3-small",
body=open("pr_body.md").read(),
head="feat/swap-embedding-model",
)
print(url)How this code works
This Python script's primary job is to programmatically open a new pull request on GitHub. For AI teams, this automates a crucial step in their Git workflow, allowing new AI model updates or features to be submitted for review without manual intervention. The code starts by setting up logging for structured output and securely retrieves GITHUB_TOKEN and REPO_SLUG from environment variables, preventing sensitive data from being hardcoded. It imports PyGithub for interacting with the GitHub API and tenacity for adding robustness to API calls. The core logic resides in the open_pr function, which accepts the PR's title, body, and the head and base branches.
The open_pr function is enhanced with a @retry decorator, which automatically retries the PR creation if a GithubException occurs, making the operation resilient to temporary network issues. A subtle but important detail is the explicit check using repo.get_branch(head) to ensure the feature branch actually exists before attempting to create the PR. This prevents errors by logging and raising an exception if the branch is missing. The base branch for the pull request defaults to "main", streamlining common merges. Once the PR is created via repo.create_pull, its details, including the URL, are logged, and the URL is returned.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Set up a GitHub PR template for an AI project repository. The template should require contributors to fill in: what changed, eval results (even if N/A), model version pins affected, and a checklist. Then open a real or practice PR using the template and verify the sections appear automatically.
# Step 1: create the template file in your repo
import os
template_dir = ".github"
os.makedirs(template_dir, exist_ok=True)
template_path = os.path.join(template_dir, "pull_request_template.md")
template_content = """
## What changed
<!-- TODO: describe the change in 2-3 sentences -->
## Eval results
<!-- TODO: paste a before/after table or write N/A with reason -->
## Model / dependency version pins affected
<!-- TODO: list any LLM SDK, model name, or embedding model changes -->
## Checklist
- [ ] TODO: add your first checklist item
- [ ] TODO: add your second checklist item
"""
# TODO: write template_content to template_path
# TODO: commit and push to a test repo, then open a PR and verify
# that the template auto-populates the PR description box
Quick check
A teammate opens a PR changing the retrieval chunk size from 512 to 1024 tokens. The diff is 3 lines. What should a reviewer ask for before approving?
Which of these belongs in a PR description for an AI project but NOT for a typical web app PR?
Your team uses trunk-based development where every commit goes to main. What is the main tradeoff when applying this to AI experiment code?