Production · Delivery

GitOps with Argo CD: Git as the Source of Truth

Letting Argo CD keep each cluster matching what's in Git, so a deploy is a reviewed pull request, drift gets fixed automatically, and rolling back means reverting a commit.

Module: 16 of 20 Level: Production · Delivery Time: 10 min read Updated:
All modules in this track (20)

In this module, you'll learn to:

  • Explain how GitOps differs from a pipeline that runs kubectl apply
  • Define an Argo CD Application with automated sync, prune, and self-heal
  • Deploy and roll back through pull requests, across many environments

Before you start: this builds on Container Delivery. There, CI built and pushed an image. Here, a separate process decides what actually runs in each cluster.

Principle

The cluster should match Git, and nothing else

In a push-based pipeline, CI runs kubectl apply or helm upgrade against the cluster. That works, but the pipeline needs admin-level access to every cluster, and anything changed by hand in the cluster stays changed until someone notices.

GitOps flips it around. A controller inside the cluster (here, Argo CD) watches a Git repository and keeps the cluster matching it. If someone edits a deployment by hand, Argo CD puts it back. A deploy becomes a pull request, and the Git history is the deploy history.

Setting up an application

An Argo CD Application says which part of which repository should be running where:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-api-prod
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<org>/<manifests-repo>.git
    targetRevision: main
    path: apps/orders-api/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: orders
  syncPolicy:
    automated:
      prune: true      # remove resources that were deleted from Git
      selfHeal: true   # undo changes made directly in the cluster
    syncOptions:
      - CreateNamespace=true
Scroll sideways to see the full diagram →

selfHeal is what turns "Git is the source of truth" from a policy into something enforced. prune makes deleting a file from Git actually remove the resource from the cluster.

How a deploy flows

  1. CI builds and pushes the image, as in Container Delivery, and records its digest.
  2. CI (or a person) opens a pull request in the manifests repository that changes the image digest for one environment.
  3. The pull request is reviewed and merged. For Production, this is where approval happens.
  4. Argo CD notices the change and syncs the cluster to match.
  5. To roll back, revert the commit. Argo CD syncs the cluster back to the previous version.
A GitOps deploy: CI never touches the cluster
CI (outside the cluster)
Application repoa developer merges code
CI builds, tests, and pushes the image
Amazon ECRnew image digest
Manifests repo
Pull requestchanges one image digest for one environment
reviewed and merged; approval happens here
main branchnow describes the new desired state
Argo CD, running inside the cluster, pulls from Git every few minutes, or on a webhook
EKS cluster
Argo CDcompares Git with what's running, and applies the difference
sync
Deployment rolls outpods pull the new digest from ECR
Argo CD keeps comparing, so a manual change in the cluster is put back to match Git

Keeping application code and deployment manifests in separate repositories is common. It lets you give "who can change what's running in Production" to a smaller group than "who can change the code".

Structuring many environments and apps
  • One folder per environment, using Kustomize overlays or per-environment Helm values (both covered in Helm & Kustomize), so the difference between QA and Production is a small, readable diff.
  • The "app of apps" pattern (or an ApplicationSet) lets one Argo CD application create the rest, so adding a service or a cluster is also just a pull request.
  • Argo CD Projects limit which repositories and namespaces each team's applications can use.
Implementation notes
  • Keep secrets out of the manifests repo. Use the External Secrets Operator from Workload Identity & Secrets, so Git holds only a reference to each secret.
  • Turn on self-heal gradually. Start with automated sync and no self-heal while teams get used to it, then enable it once nobody relies on hand edits.
  • Watch for sync failures. An application stuck "OutOfSync" or "Degraded" is an alert worth sending, as covered in Observability.
  • Protect Argo CD itself. It can change everything in the cluster. Restrict who can use its UI and API, sign in through SSO, and keep its own configuration in Git as well.
Recap

Key terms

  • GitOps: a controller in the cluster keeps it matching what's in Git.
  • Argo CD Application: says which repository path should run in which cluster and namespace.
  • Sync: applying Git's desired state to the cluster.
  • Self-heal: undoing changes made directly in the cluster.
  • Prune: deleting cluster resources that were removed from Git.
  • Drift: any difference between Git and the cluster.
Check yourself

Pop quiz

Five questions: three on the ideas in this module, and two scenarios where you apply them. The order changes every time you take it, and 4 out of 5 passes.

5 questions. Decide on your answer to each, then check it.

  1. Concept

    With GitOps, what does the CI pipeline do to the cluster?

    1. Runs kubectl apply with admin credentials
    2. Nothing directly: it pushes an image and changes Git, and Argo CD applies the change from inside the cluster
    3. Restarts every pod
    4. Runs helm upgrade
    Show answer

    Nothing directly: it pushes an image and changes Git, and Argo CD applies the change from inside the cluster

    Argo CD pulls from Git and applies. CI never needs cluster credentials, which removes a powerful secret from the pipeline.

  2. Scenario

    Someone scales a Deployment by hand, and the Application has selfHeal: true. What happens?

    1. The change stays until the next commit
    2. Argo CD puts it back to what Git says
    3. Argo CD commits the change to Git
    4. The Application is deleted
    Show answer

    Argo CD puts it back to what Git says

    Self-heal treats manual changes as drift and reverts them, so Git stays the single source of truth.

  3. Concept

    How do you roll back a bad release in a GitOps setup?

    1. kubectl rollout undo in every cluster
    2. Revert the commit that changed the image, and let Argo CD sync the cluster back
    3. Delete the Argo CD Application
    4. Rebuild the previous image
    Show answer

    Revert the commit that changed the image, and let Argo CD sync the cluster back

    The Git history is the deploy history, so a revert is a reviewed, recorded rollback. A kubectl undo would be put back by self-heal anyway.

  4. Scenario

    You delete a manifest from Git, and the Application has prune: true. What happens to that object in the cluster?

    1. Argo CD deletes it
    2. It stays, unmanaged
    3. Argo CD restores the file in Git
    4. The sync fails
    Show answer

    Argo CD deletes it

    Prune removes cluster resources that no longer exist in Git. Without it, deleted manifests leave orphaned objects behind.

  5. Concept

    Why keep deployment manifests in a separate repository from application code?

    1. Argo CD can't read application repositories
    2. So the group allowed to change what runs in Production can be smaller than the group allowed to change code
    3. Git limits repository size
    4. It makes builds faster
    Show answer

    So the group allowed to change what runs in Production can be smaller than the group allowed to change code

    Separating them lets you protect the manifests repository, where a merge is a Production deploy, more tightly than the code.

See it in practice

Questions about this learning path?

I'm happy to go into more detail on any of this: how the pieces fit together, ingress costs, RBAC design, or how I triage incidents.

anupojuprudhvi@gmail.com