Skip to content

ArgoCD Operations

ArgoCD deploys every application in the cluster from this repository. This page states what exists, where it is defined, and the procedures that touch it. Commands assume --context fzymgc-house.

Property Value
Install helm_release.argocd in tf/cluster-bootstrap/argocd.tf, chart argo-cd, version var.argocd_version
Namespace argocd
Web UI https://argocd.fzymgc.house, ingress certificate from fzymgc-house-issuer
Login OIDC through Dex against the Keycloak realm fzymgc. Group argocd-admin maps to role:admin, argocd-user to role:readonly
Replicas server 2, repo-server 2, controller 1
Applications argocd/cluster-app/, a Helm chart with one Application template per app
Kustomize kustomize.buildOptions: --enable-helm, so a kustomization can inflate a Helm chart
Webhook endpoint https://argocd-wh.fzymgc.net/api/webhook, through the Cloudflare tunnel
Kubernetes Secret Source Holds
argocd-secret ExternalSecret from fzymgc-house/cluster/argocd admin.password, server.secretkey, webhook.github.secret
argocd-oidc-secret Terraform, write-only, from fzymgc-house/cluster/argocd/oidc The Keycloak client id and secret
argocd-selfhosted-repo, argocd-temporal-workers-repo Terraform, write-only, from fzymgc-house/cluster/argocd GitHub App credentials for the two repositories
argocd-repo-server-helm-registry shared-resources/ghcr-helm-registry-config.yaml The registry login the repo-server uses for private OCI charts

GitHub sends push events to argocd-wh.fzymgc.net. The Cloudflare tunnel forwards them to argocd-server.argocd.svc.cluster.local (webhook_services in tf/cloudflare/variables.tf). ArgoCD checks the signature against webhook.github.secret and refreshes the applications that use the repository, instead of waiting for its three-minute poll.

GitHub (organization webhook) -> Cloudflare Tunnel -> ArgoCD server
push event argocd-wh.fzymgc.net /api/webhook
  1. Read the secret:

    Terminal window
    vault kv get -field=webhook.github.secret secret/fzymgc-house/cluster/argocd
  2. Open https://github.com/organizations/fzymgc-house/settings/hooks and add a webhook:

    Field Value
    Payload URL https://argocd-wh.fzymgc.net/api/webhook
    Content type application/json
    Secret The value from step 1
    Events Just the push event
    Active Yes
  3. Push to any repository in the organization and read the delivery on the same page. A delivery with a 200 response means the webhook works.

Check Command Expected
DNS dig argocd-wh.fzymgc.net A CNAME to cfargotunnel.com
Endpoint curl -I https://argocd-wh.fzymgc.net/api/webhook 400 (no payload) or 200
Delivery Push to a repository and read the GitHub deliveries page 200
Refresh Read the application after a push The sync starts within seconds, not minutes
Terminal window
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server --tail=100 | rg -i webhook
Symptom Cause Fix
401 or 403 The secret in GitHub differs from Vault Paste the Vault value into the GitHub webhook
404 The tunnel route is missing Read webhook_services in tf/cloudflare/variables.tf
502 ArgoCD is unreachable Read the argocd-server pods
No sync after a push The Application repository URL differs from the webhook source Compare the two URLs

To stop the webhook, delete it in GitHub. ArgoCD falls back to its poll. No ArgoCD change is needed.

Do not combine syncOptions: RespectIgnoreDifferences=true with ignoreDifferences jqPathExpressions that address array elements, for example .spec.data[].remoteRef.conversionStrategy on an ExternalSecret. ArgoCD’s sync-time normalizer cannot merge an array when items are added or removed. The sync reports Succeeded at the right revision, map-typed fields apply, and the live array keeps its old value. On an ExternalSecret this leaves the operator in SecretSyncedError and the application OutOfSync.

ignoreDifferences alone normalizes only the diff, so schema-defaulted fields do not cause a standing OutOfSync, and the sync applies the true desired manifest. Six Applications still set the option because they carry other ignore rules (StatefulSet volumeClaimTemplates fields, CRD status, workflow template fields, webhook configurations, Secret data): clickstack, external-dns, hcp-terraform-operator, nats, system-upgrade and vault. external-dns and hcp-terraform-operator also keep .spec.data[] paths on an ExternalSecret, so a new key on that ExternalSecret does not reach the cluster.

To unstick a sync in this state, apply the desired manifest with the field manager operator. Do not use argocd-controller. ArgoCD adds the annotation argocd.argoproj.io/tracking-id at sync time, so the manifest in git does not carry it. A server-side apply removes each field that its field manager owns and the manifest leaves out. An apply as argocd-controller therefore removes the annotation, and ArgoCD stops tracking the object.

Read the diff before the apply. It must show the array change, and it must not remove the tracking annotation:

Terminal window
git show origin/main:<path-to-manifest> > /tmp/m.yaml
kubectl --context fzymgc-house diff --server-side --field-manager=operator --force-conflicts -f /tmp/m.yaml
kubectl --context fzymgc-house apply --server-side --field-manager=operator --force-conflicts -f /tmp/m.yaml

Verify a version bump at the running artifact

Section titled “Verify a version bump at the running artifact”

A merged pin is not a verified bump. Every read below is a kubectl get or an rg over the repository. Nothing writes.

Synced and Healthy answer “does the cluster match git”, not “did the upgrade take effect”. ArgoCD renders Helm with --include-crds, so a chart’s crds/ directory is applied unless the Application sets skipCrds: true. Chart version and appVersion differ, and a values tag: override shadows the appVersion. status.sync.revisions[] is what ArgoCD synced, not what runs.

Read three things for a chart pin:

Terminal window
# Clauses 1+2 (Synced, Healthy) plus the revision ArgoCD resolved per source, estate-wide
kubectl --context fzymgc-house get applications.argoproj.io -n argocd -o json \
| jq -r '.items[] | [.metadata.name, (.metadata.annotations["argocd.argoproj.io/sync-wave"] // "0"),
.status.sync.status, .status.health.status,
((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'
# one Application
kubectl --context fzymgc-house get application -n argocd <name> -o json \
| jq -r '[.status.sync.status, .status.health.status,
((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'
# (`.status.sync.revisions` is null on a single-source Application, so the fallback is required)
# Clause 3: the workloads the Application owns
kubectl --context fzymgc-house get application -n argocd <name> -o json \
| jq -r '.status.resources[] | select(.kind=="Deployment" or .kind=="StatefulSet" or .kind=="DaemonSet")
| .kind+" "+.namespace+"/"+.name'
# then chart label, appVersion label, images on each
kubectl --context fzymgc-house get <kind> -n <ns> <name> -o json \
| jq -r '[.metadata.labels["helm.sh/chart"], .metadata.labels["app.kubernetes.io/version"],
([.spec.template.spec.containers[].image]|join(" "))] | @tsv'
# and the digest the node pulled (a mutable tag can be re-published)
kubectl --context fzymgc-house get pods -n <ns> -l <selector> \
-o jsonpath='{.items[*].status.containerStatuses[*].imageID}'

Record the three reads and the imageID in the pull request.

Some applications do not carry the label that names the running version:

App What to read instead
vault The StatefulSet carries no helm.sh/chart label and its image is a tag: override. A chart bump moves the Application revision and the vault-agent-injector image only
velero The appVersion can stay put on a chart bump. The helm.sh/chart label on velero/velero and velero/node-agent is the artifact
clickstack app.kubernetes.io/version is the chart’s appVersion, not the running hyperdx image. Read the image from the values and the pod
renovate The image comes from a values tag: override. The chart label moves and the image does not
temporal-worker-controller No helm.sh/chart label. Read app.kubernetes.io/version on the manager Deployment and the workerdeployments.temporal.io CRD

For a chart that carries CRDs, locate them in the pulled chart before the bump and record each CRD’s resourceVersion:

Terminal window
helm pull <repo>/<chart> --version <v> --untar -d /tmp/chart && cd /tmp/chart/<chart>
ls crds/ 2>/dev/null; rg -l 'kind: CustomResourceDefinition' templates/; find charts -type d -name crds
kubectl --context fzymgc-house get crd <name> -o jsonpath='{.metadata.resourceVersion}'

After the bump, the resourceVersion changes only if the chart’s CRD file differs. Diff the two pulled charts’ crds/ first. An identical CRD is a no-op apply. Then make sure that the dependent objects still admit: kubectl --context fzymgc-house get <cr-kind> -A. Never kubectl apply a CRD as a remedy, because selfHeal reverts it on the next sync. Express the remedy in the repository, for example a second Helm source for a -crds sibling chart at the same or an earlier sync wave.

helm show chart <repo>/<chart> --version <v> prints the chart version and the appVersion. Record both per bump. Helm 4 fails helm show chart --repo <url> with no cached repo found against a stale repository cache. Set HELM_REPOSITORY_CACHE and HELM_REPOSITORY_CONFIG to a scratch directory, or read <repo>/index.yaml with yq.

For a kustomize helmCharts: app, the pin is helmCharts[].version in argocd/app-configs/<app>/kustomization.yaml. Before changing it, run the render diff in argocd/CLAUDE.md, section Kustomize Helm-chart inflation. kubectl kustomize --enable-helm writes a gitignored charts/ directory into the app-config. Never commit it.

For an image pin under app-configs, kubectl --context fzymgc-house get deploy,sts,ds -n <ns> -o jsonpath='{..image}' must show the committed tag. Where the pin is tag@sha256:..., re-pin the digest with the tag (crane digest <image>:<tag>) and make sure that the pod’s imageID matches.

Other surfaces:

  • Ansible: run the role through its wrapper (scripts/nas-playbook.sh --tags <role>, scripts/dns-playbook.sh --limit <node> --tags <role>, scripts/heimdall-playbook.sh, or ansible-playbook -i inventory/hosts.yml <play>.yml --tags <role> for the cluster plays), with --check --diff first, then apply, then read the installed version on the node. A merged pin that was not applied is not verified.
  • Terraform: the HCP Terraform run log’s provider line plus the .terraform.lock.hcl change in the same PR. Never a manual run.
  • Action SHAs: the merged PR’s workflow run resolves the new SHA in the job’s “Set up job” log. Dereference an annotated tag to its commit before writing uses:.

One PR per negative-wave Application and one PR per positive wave, merged in ascending wave order. Each negative wave reaches Synced, Healthy and the third read before any dependent in a positive wave moves. Majors go in their own PR after the wave’s minor and patch PR is verified. In steady state a single-dependent minor rides the automerge lane and majors are attended by packageRules. No standing gate enforces wave order.

URL Purpose
https://argocd.fzymgc.house Web UI
https://argocd-wh.fzymgc.net/api/webhook GitHub webhook endpoint