Git stores every version of every tracked file forever. That is a feature for source code and a liability for binary blobs. A 500 MB model checkpoint committed once is there in the history even after you delete it. git clone will pull it down for every new developer. Multiply by a few experiments and a few teammates and you have a repo that takes ten minutes to clone and gigabytes of storage on every machine. .gitignore solves this by telling Git not to track those files in the first place.
Under the hood, Git evaluates each file path against your .gitignore patterns before deciding whether to include it in git status output or allow it to be staged. Patterns are matched top-to-bottom; the last matching rule wins. A leading / anchors the pattern to the directory containing the .gitignore. A trailing / matches only directories. A leading ! negates a rule, which is how you can say "ignore all CSVs except the sample ones." Git also merges .gitignore files at every directory level, so you can put dataset-specific rules in data/.gitignore and keep the repo root clean. One critical mental model: .gitignore only prevents untracked files from being tracked. If a file is already in the index (already committed), adding it to .gitignore does nothing. You have to explicitly remove it from tracking with git rm --cached <file> first.
For AI projects, secrets are the highest-priority concern. Your code needs an OPENAI_API_KEY, maybe an ANTHROPIC_API_KEY, maybe a Hugging Face token. The right pattern is: create a .env file locally with the real values, commit a .env.example file with placeholder values and comments explaining what each variable does, and add .env and any variant (.env.local, .env.production) to .gitignore immediately when you create the project. The !.env.example negation rule keeps the example file tracked. Libraries like python-dotenv load this file at runtime with load_dotenv(). Never put credentials in config.py or settings.py as Python literals; those files get committed. If you use GitHub, enable secret scanning so GitHub will alert you if a known credential format appears in a push.
Model weights and checkpoints deserve their own section because the failure mode is subtle. When you run a training or fine-tuning job, frameworks like PyTorch Lightning, Hugging Face Transformers, and Keras will write checkpoints to disk automatically, often to a checkpoints/ or runs/ directory. If you do not ignore these directories, git status will show hundreds of new files after every training run, and a careless git add . will stage all of them. The fix is to ignore the directories by name, plus glob patterns for the file extensions: *.pt, *.ckpt, *.safetensors, *.h5, *.onnx. For models you actually want to version, use a purpose-built tool: Hugging Face Hub (huggingface-cli upload), DVC (Data Version Control), or cloud storage with a pointer file committed in the repo. DVC in particular is worth knowing: it commits small metadata files that reference large files stored in S3 or GCS, giving you versioned datasets and models without bloating Git.
At different scales the concerns shift. At 10 users on a private repo, the main risk is a leaked key going unnoticed for weeks. At 10k users on an open-source project, a committed secret is public the moment it's pushed, and GitHub's secret scanning may alert you, but rotation is still required. At 10M users with many contributors, you add pre-commit hooks (pre-commit library with detect-secrets or gitleaks) to block secrets before they hit the remote, and CI pipelines that fail the build if large files are staged. You also consider setting up a .gitattributes file with *.pt binary so Git does not attempt text diffs on binary files, which speeds up git status.
Two recovery scenarios you will encounter: first, you committed a secret. Rotate the key immediately, then decide whether to rewrite history. git filter-repo can scrub a file from all history, but every collaborator must re-clone. For most teams on a small repo, rotating is enough; history rewriting is worth it only if the repo is public and the key was long-lived. Second, you committed a large binary. Use git rm --cached path/to/model.pt, add the pattern to .gitignore, and commit. The file is now untracked, but it still exists in the commit history. If the repo is private and small, leave it. If the repo is public or history size is a real problem, git filter-repo --path path/to/model.pt --invert-paths removes it from all commits, but again everyone must re-clone.
Key Takeaways
- Never commit API keys or secrets; put them in .env and add .env to .gitignore immediately.
- Model weight files (.pt, .ckpt, .onnx, .h5) are binary blobs Git cannot diff; exclude them always.
- Use
git statusandgit ls-filesto verify ignored files never sneak into a commit. - If a secret was already committed, rotating the key is faster and safer than rewriting history.
Pro tips
- Commit your
.gitignoreas the very first file in a new repo, before any other files exist. Adding it later means you may have already staged something you did not intend to track, and fixing that requiresgit rm --cached. - The
!negation rule lets you exclude a whole directory and re-include only specific files, likedata/combined with!data/samples/. This is the cleanest way to keep sample fixtures tracked without accidentally pulling in multi-GB raw datasets. - Use
git check-ignore -v <file>to debug why a file is or is not being ignored. It prints the exact rule and line number that matched, which is far faster than reading through your.gitignoremanually. - GitHub's secret scanning and tools like
gitleaksordetect-secretsrun as pre-commit hooks or CI checks. Set these up on day one. The cost of rotating a leaked key is always higher than the cost of the tooling.
Common pitfalls
- Mistake: Adding a file to
.gitignoreafter it was already committed, expecting it to disappear. Fix: Rungit rm --cached <file>to remove it from the index, then commit that removal. - Mistake: Using
git add .without checkinggit statusfirst when working in an AI project. Fix: Always rungit statusbefore staging; if ignored files appear, fix.gitignorebefore proceeding. - Mistake: Storing API keys directly in a Python config file that gets committed. Fix: Use
.envwithpython-dotenv, never string literals, and verify.envis in.gitignorebefore writing any secrets to it. - Mistake: Committing a 2 GB model checkpoint once, then deleting it and assuming the repo is clean. Fix: The blob stays in history; use
git filter-repoto remove it or accept the size cost on private repos.
When to use .gitignore vs DVC vs cloud storage for large AI files
| Option | Use when | Avoid when |
|---|---|---|
| .gitignore only | Files are ephemeral outputs you can regenerate anytime (logs, caches, eval artifacts). | You need to reproduce an exact model or dataset version across team members or CI. |
| DVC (Data Version Control) | You need versioned, reproducible datasets and model checkpoints tied to code commits, stored in S3/GCS. | Your team is not comfortable with an extra tool and files change rarely or are already on Hugging Face Hub. |
| Hugging Face Hub | You are working with publicly available models or want a hosted model registry with built-in versioning. | Your model or data is proprietary and cannot leave your infrastructure, or you need fine-grained access control. |
| Cloud storage with pointer file | You need full control over storage location and access policy (e.g., AWS S3 with VPC, GCS with IAM). | You want automatic version tracking linked to Git commits without writing your own tooling. |
Code Example
# Minimal .gitignore for an AI project (works with Python + common ML frameworks)
# --- Secrets & credentials ---
.env
.env.*
!.env.example
secrets.yaml
config/credentials.ini
# --- Model weights & checkpoints ---
*.pt
*.pth
*.ckpt
*.safetensors
*.h5
*.onnx
*.pkl
*.bin
checkpoints/
runs/
# --- Datasets & large files ---
data/raw/
data/processed/
*.csv
*.parquet
*.jsonl
!data/samples/*.csv
# --- Python artifacts ---
__pycache__/
*.pyc
.venv/
*.egg-info/
dist/
# --- Jupyter ---
.ipynb_checkpoints/
How this code works
The .gitignore file defines patterns for files and directories that Git should intentionally ignore, preventing them from being accidentally committed to a repository. This is critical in AI projects for maintaining security, managing large files efficiently, and keeping the repository focused on source code. It achieves this by protecting sensitive information, such as API keys and credentials, commonly stored in files like .env, secrets.yaml, or config/credentials.ini. It also ensures that very large model files (*.pt, *.safetensors, *.h5) and entire data directories (data/raw/, data/processed/) are not tracked, which would quickly bloat the repository and hinder collaboration.
The file works by listing various patterns. For instance, *.pt tells Git to ignore all files ending with .pt, typically used for PyTorch models. Similarly, entries like __pycache__/ and .venv/ exclude Python build artifacts and virtual environments. The * acts as a wildcard, matching any sequence of characters. A subtle yet important feature is the ! (exclamation mark), as seen in !.env.example or !data/samples/*.csv. This negates a preceding ignore rule, allowing specific files to be tracked even if a broader pattern (like *.csv ignoring all CSVs) would normally exclude them. This is useful for including small sample datasets or template configuration files while keeping large, actual datasets out of version control.
Production-grade example
Pre-commit hook with structured logging, size checks, secret scanning, and graceful degradation on unreadable files.
#!/usr/bin/env python3
# Requires: gitpython>=3.1, python-dotenv>=1.0, structlog>=24.0
"""
Pre-commit validation script: blocks commits that contain secrets or large files.
Run via pre-commit hook or CI step.
"""
import os
import sys
import subprocess
import structlog
from pathlib import Path
from dotenv import load_dotenv
load_dotenv()
log = structlog.get_logger()
MAX_FILE_BYTES = int(os.getenv("MAX_COMMIT_FILE_BYTES", 5 * 1024 * 1024)) # 5 MB default
SECRET_PATTERNS = [
"OPENAI_API_KEY",
"ANTHROPIC_API_KEY",
"sk-", # OpenAI key prefix
"hf_", # Hugging Face token prefix
"Bearer ",
]
def get_staged_files() -> list[Path]:
result = subprocess.run(
["git", "diff", "--cached", "--name-only", "--diff-filter=ACM"],
capture_output=True, text=True, timeout=10
)
result.check_returncode()
return [Path(p) for p in result.stdout.splitlines() if p]
def check_file_size(path: Path) -> bool:
if not path.exists():
return True
size = path.stat().st_size
if size > MAX_FILE_BYTES:
log.error("large_file_blocked", path=str(path), size_mb=round(size / 1e6, 2))
return False
return True
def check_secrets(path: Path) -> bool:
try:
text = path.read_text(errors="replace")
except (OSError, PermissionError) as exc:
log.warning("file_read_skipped", path=str(path), reason=str(exc))
return True # graceful degradation: do not block on unreadable files
for pattern in SECRET_PATTERNS:
if pattern in text:
log.error("secret_pattern_found", path=str(path), pattern=pattern)
return False
return True
def main() -> int:
staged = get_staged_files()
if not staged:
log.info("no_staged_files")
return 0
failures: list[str] = []
for path in staged:
log.info("checking_file", path=str(path))
if not check_file_size(path):
failures.append(f"[size] {path}")
if not check_secrets(path):
failures.append(f"[secret] {path}")
if failures:
log.error("commit_blocked", violations=failures)
print("\nCommit blocked. Fix these before committing:")
for f in failures:
print(f" {f}")
return 1
log.info("commit_allowed", files_checked=len(staged))
return 0
if __name__ == "__main__":
sys.exit(main())
How this code works
This Python script acts as a crucial pre-commit validation tool, designed to prevent accidental inclusion of sensitive information or excessively large files into a Git repository. Its primary job, within the context of Git & Version Control lessons, is to enforce good practices for managing API keys, model files, and datasets. The script starts by identifying all files currently staged for the upcoming commit using a Git command executed via subprocess.run. Once the list of staged files is gathered, it proceeds to evaluate each one against predefined rules.
For every staged file, the script performs two key checks. First, check_file_size verifies that a file's size doesn't exceed a maximum limit. A subtle but important detail is the MAX_FILE_BYTES variable, which defaults to 5 MB if no custom size is specified in environment variables, preventing very large files from being committed by default. Second, check_secrets scans the file's content for SECRET_PATTERNS like OPENAI_API_KEY or common key prefixes such as sk-. If any file violates these size or secret rules, the main function records the issue, logs an error using structlog, and importantly, blocks the Git commit entirely, prompting the user to fix the identified problems before trying again.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Create a .gitignore file for an AI project that uses the OpenAI API, stores model checkpoints in a checkpoints/ directory, and keeps raw data in data/raw/. Include a .env.example file with placeholder values, and verify using git check-ignore that the right files are ignored and the example file is still tracked.
# Step 1: Initialize a temporary repo to test your .gitignore
git init test-ai-project
cd test-ai-project
# Step 2: Create your .gitignore
# TODO: add rules for .env variants, model weights, checkpoints/, data/raw/
# TODO: add a negation rule so .env.example is still tracked
touch .gitignore
# Step 3: Create test files
touch .env
touch .env.example
mkdir -p checkpoints && touch checkpoints/model_epoch5.ckpt
mkdir -p data/raw && touch data/raw/train.parquet
mkdir -p data/samples && touch data/samples/tiny.csv
# Step 4: Verify
# TODO: run git check-ignore -v on each file and confirm expected behavior
# TODO: run git status and confirm .env.example appears as untracked (not ignored)
# Step 5: Stage and commit .gitignore and .env.example
# TODO: git add and git commit
Quick check
You add
data/raw/to.gitignore, butgit statusstill shows files inside that directory. What is the most likely reason?Which
.gitignorerule combination correctly ignores all CSVs indata/but keepsdata/samples/test.csvtracked?A teammate accidentally committed an
OPENAI_API_KEYvalue inconfig.pythree commits ago and has since deleted the file. Is the key still exposed in the repo?
.gitignore snippet that ignores all PyTorch checkpoint files and the .env file, but keeps .env.example tracked. Then explain what you would do if you discovered a teammate had already committed a .env file with a real API key two weeks ago.