Phase 3: CI/CD & Automation

Retention Policies & Garbage Collection

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building amazing things with digital LEGOs, like creating a fantastic new computer game or a helpful app. Every time you try out a new idea, save a little bit of code, or finish a section, it's like creating a special "instruction manual" or "design blueprint" for that part. You store all these blueprints in a giant, super-organized digital library, much bigger than any library you've seen!

Now, if you kept every single blueprint you ever made – every early draft, every tiny experiment, every finished masterpiece – this digital library would quickly become an overflowing mess. It would be hard to find the right blueprint when you needed it, take up tons of digital space (which costs money, just like a real building!), and generally slow everything down. So, people need smart rules to manage their digital blueprints. These smart rules are what we call "retention policies." They're like the head librarian's instructions that say, "Always keep the final, finished blueprint for the game we released last year, forever!" or "Keep all the early draft blueprints from the past two weeks, just in case, but after that, they're probably not needed anymore."

While retention policies are the smart rules, "garbage collection" is the actual work of enforcing those rules. It’s like a super-efficient librarian who regularly walks through your digital library. This librarian carefully checks each blueprint against all the rules you've set. If a blueprint no longer meets any of the "keep" rules – for example, it's an old draft from a month ago, and your rule says "only keep drafts for two weeks" – then the librarian knows it's time to remove it. They then politely take those old blueprints out of the library, making space for new, important ones.

So, when people are building big computer programs and apps, they use these ideas to keep their digital workspaces tidy and efficient. They set up their own "library rules" (retention policies) and let the "librarian" (garbage collection) do the cleanup. This means they can focus on making amazing new things without worrying about their digital storage becoming a giant, cluttered mess, and they can always quickly find the exact blueprints they need for their projects.

Retention policies are the rules you define to manage the lifecycle of your stored artifacts. Think of them as intelligent housekeepers for your artifact repository. Their primary goal is to specify how long artifacts should be kept, and which ones are important enough to retain. This isn't just about tidiness; it's crucial for controlling storage costs, improving repository performance (fewer items to index), meeting compliance requirements (e.g., "keep all release binaries for 5 years"), and ensuring your developers don't drown in outdated build snapshots. Common policy examples include "keep the last N successful builds," "delete artifacts older than X days," or "retain all artifacts associated with a production release tag indefinitely."

While retention policies define what to keep or discard, garbage collection (GC) is the active process that enforces these rules. It's the execution phase where your repository scans its contents, identifies artifacts that violate your established policies (e.g., an untagged build artifact now older than its specified retention period), and then physically deletes them. Garbage collection typically runs on a scheduled basis rather than in real-time, allowing it to efficiently clean up large volumes of data without impacting immediate repository operations. It's important to differentiate: policies are the strategy, GC is the tactical operation.

Implementing effective retention policies and understanding garbage collection is a balancing act. You need enough historical data for debugging, rollbacks, and auditing, but not so much that it inflates storage costs or slows down your systems. A best practice is to tier your policies: for example, keep release-tagged artifacts much longer (or indefinitely) than development or snapshot builds. Always consider legal and compliance mandates for how long specific artifact types must be stored. Finally, actively monitor your garbage collection jobs; failures can lead to unexpected storage bloat, while overly aggressive policies might delete critical artifacts.

Key Takeaways

  • Retention policies define the rules for artifact storage duration and importance.
  • Garbage collection is the automated process that enforces these defined retention policies.
  • They are essential for managing storage costs, improving repository performance, and ensuring compliance.
  • Tailor different policies for different artifact types (e.g., indefinite for releases, short for dev builds).
  • Regularly monitor garbage collection activity to prevent storage issues or accidental data loss.

Code Example

yaml
# Example .gitlab-ci.yml snippet for job artifact retention
build_job:
  stage: build
  script:
    - echo "Building application..."
    - mkdir -p build_output
    - echo "Build artifact content" > build_output/app.zip
  artifacts:
    paths:
      - build_output/
    expire_in: 1 week # Policy: delete build job artifacts after 1 week

deploy_to_prod_job:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - echo "Deployment successful." > deploy_report.log
  artifacts:
    paths:
      - deploy_report.log
    expire_in: never # Policy: keep deploy reports indefinitely (for auditing)

How this code works

This GitLab CI/CD configuration snippet illustrates how to define specific retention policies for job artifacts, which are files generated by jobs and stored by GitLab. This is vital for managing storage space (a form of garbage collection) while ensuring critical artifacts are available for as long as needed. The code shows two distinct jobs, each producing artifacts with different lifespans according to their importance.

The build_job creates an artifact from the build_output/ directory. For these artifacts, the artifacts section includes expire_in: 1 week, meaning GitLab will automatically delete them after seven days, freeing up server space. In contrast, the deploy_to_prod_job generates a deploy_report.log artifact, and its expire_in: never policy dictates that this report should be kept indefinitely, serving as a permanent record for auditing purposes. A subtle but important detail is that if expire_in isn't explicitly set, GitLab CI will default to a project-level setting or a general default (often 30 days), which could lead to unexpected artifact deletion if not considered.

Related Articles

Hand-picked reads to go deeper on artifact management.

Back to DevOps Engineer Roadmap