Phase 3: CI/CD & Automation

GitOps Repository Structure & Overlays

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you love building with LEGOs. You've got tons of sets: a cool spaceship, a bustling city, maybe even a robot! Now, imagine all the instruction booklets for everything you want to build, from the smallest spaceship part to the largest city building, are stored together in one super organized "Master Instruction Library." This library isn't just a pile of physical books; it's like a special digital notebook where every instruction is written down and saved. This is super helpful because if you ever want to build something, you don't guess – you just look it up in your Master Library. This library is your "single source of truth" – the one place that tells you exactly how every LEGO creation should look.

In this Master Instruction Library, you'd have different sections. Maybe one section is called "City Basics" for all the common parts of your LEGO city, like roads, parks, or common houses that everyone uses. Another big section might be called "Cool Vehicles" for all your specific builds like that awesome spaceship or speedy race car. Inside "Cool Vehicles," each specific LEGO model, like "Red Race Car" or "Blue Spaceship," gets its own mini-instruction book that tells you exactly how to build just that thing from start to finish. When you want to build or change anything, you only change the instructions in this central library. You don't just start building; you update the instructions first.

So, when you want to make a tiny change, like adding a new window to a building in your city, you update the instructions in the "City Basics" section of your Master Library. Then, anyone who wants to build that city building will automatically use the new instructions. This means everything you build always matches what's written down. But here's a tricky part: sometimes you want to build the "Red Race Car" for a regular race, but also a slightly different version for a "practice run" that uses cheaper tires, or a "display model" that has extra shiny pieces. It would be a huge mess if you had to copy the entire instruction book for the race car three times and then remember to change the engine instructions in all three books if you upgraded it! This is where "overlays" come in, to help you make those small, specific changes without copying everything.

A robust GitOps repository structure is crucial for managing your infrastructure and applications consistently across different environments. At its core, it means your entire desired state — from Kubernetes cluster configurations to application deployments — is defined and version-controlled in Git. Typically, you'll organize your repository with top-level directories like infrastructure/ for cluster-wide settings (e.g., CRDs, namespace definitions) and applications/ or services/ for individual application manifests. Within applications/, each application usually gets its own subdirectory, holding its core Kubernetes manifests. This establishes a "single source of truth" principle, where every change to your environment starts with a Git commit to this central repository.

While applications often share common configurations, they inevitably require environment-specific variations – for example, different replica counts, resource limits, or image tags for development versus production. Simply duplicating all manifests for each environment quickly becomes a maintenance nightmare, leading to inconsistencies and errors. This is where "overlays" come in, most commonly implemented using Kustomize. Kustomize allows you to define a base set of common manifests for an application. Then, for each specific environment (e.g., dev, staging, production), you create an overlay that applies targeted patches or additions to that base configuration.

Practically, an overlay will reference the common base and then provide small, specific modifications. For instance, a dev overlay might set a deployment's replica count to 1 and use a :latest image tag, while a production overlay sets replicas to 3 and uses a stable, versioned image tag like :v1.2.3. Kustomize merges these patches onto the base manifests at runtime, generating the final, environment-specific YAML that ArgoCD or Flux then applies to your cluster. This approach dramatically reduces configuration duplication, makes environment differences explicit and auditable, and ensures that core application definitions remain consistent while allowing necessary environment-specific adjustments.

Key Takeaways

  • Your Git repository is the single source of truth for all desired infrastructure and application states.
  • Organize your GitOps repository logically, typically separating infrastructure from applications/services.
  • Overlays (like Kustomize) are essential for managing environment-specific configurations without duplicating full manifests.
  • An overlay applies targeted patches or additions to a common base set of Kubernetes manifests.
  • This strategy improves consistency, auditability, and reduces maintenance overhead across environments.

Code Example

yaml
# base/deployment.yaml (simplified for example)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1 # Default for dev
  template:
    spec:
      containers:
        - name: my-app
          image: my-registry/my-app:dev

---

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base # Refers to the directory containing base manifests like deployment.yaml
patchesStrategicMerge:
  - patch-prod.yaml # Applies environment-specific changes from this file

---

# overlays/production/patch-prod.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3 # Overrides base replica count for production
  template:
    spec:
      containers:
        - name: my-app
          image: my-registry/my-app:v1.0.0 # Overrides base image for production

How this code works

This code demonstrates how to manage Kubernetes application configurations across different environments, like development and production, using Kustomize. It avoids duplicating entire YAML files by defining a common "base" configuration and then applying environment-specific changes on top. The base/deployment.yaml file sets up a default my-app Deployment with replicas: 1 and an image tagged dev, serving as the shared foundation for all environments.

For production, the overlays/production/kustomization.yaml acts as a build recipe. It includes the ../../base configuration using resources and then specifies patchesStrategicMerge to apply further modifications defined in patch-prod.yaml. This patch-prod.yaml file only contains the fields that need to change for production, such as overriding replicas to 3 and the image to v1.0.0. A subtle but powerful aspect here is that Kustomize intelligently merges these changes; it doesn't require restating the entire Deployment definition in the patch, only the specific values to be updated or added. This keeps environment-specific overrides concise and easy to manage.

Related Articles

Hand-picked reads to go deeper on gitops (argocd / flux).

Back to DevOps Engineer Roadmap