ArgoCD Operations

ArgoCD manages GitOps deployments for the cluster. This document covers operational procedures for ArgoCD.

GitHub Webhook Configuration

ArgoCD receives push notifications from GitHub via webhook to trigger immediate syncs (instead of waiting for polling).

Webhook Architecture

GitHub (org webhook) → Cloudflare Tunnel → ArgoCD Server
     push event         argocd-wh.fzymgc.net    /api/webhook

Initial Setup

Prerequisites:

Cloudflare Tunnel deployed and healthy
ArgoCD running with webhook secret configured

Step 1: Get webhook secret from Vault

vault kv get -field=webhook.github.secret secret/fzymgc-house/cluster/argocd

Step 2: Configure GitHub organization webhook

Go to: https://github.com/organizations/fzymgc-house/settings/hooks
Click “Add webhook”
Configure:

Field	Value
Payload URL	`https://argocd-wh.fzymgc.net/api/webhook`
Content type	`application/json`
Secret	(paste from Step 1)
Events	“Just the push event”
Active	✅

Click “Add webhook”

Step 3: Verify webhook delivery

Push to any fzymgc-house repository
Check webhook deliveries at: https://github.com/organizations/fzymgc-house/settings/hooks
Look for green checkmark and 200 response

Verification

Test	Command	Expected
DNS resolves	`dig argocd-wh.fzymgc.net`	CNAME to cfargotunnel.com
Endpoint reachable	`curl -I https://argocd-wh.fzymgc.net/api/webhook`	400 (missing payload) or 200
Webhook delivery	Push to repo, check GitHub	Green ✓, 200 response
ArgoCD refresh	Check app sync after push	Immediate (not 3-min delay)

Troubleshooting

Symptom	Cause	Fix
401/403 response	Webhook secret mismatch	Verify secret in Vault matches GitHub config
404 response	Tunnel route missing	Check Cloudflare Tunnel ingress config
502 response	ArgoCD unreachable	Check argocd-server pod status
No sync after push	Repo URL mismatch	Verify Application repo URL matches webhook source

Check ArgoCD logs for webhook activity:

kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server --tail=100 | rg -i webhook

Disabling Webhook

To revert to polling-only mode:

Delete the GitHub organization webhook
ArgoCD automatically falls back to polling (default: 3 minutes)

No ArgoCD configuration changes required.

Sync Pitfall: RespectIgnoreDifferences + array jqPathExpressions

Do not combine syncOptions: RespectIgnoreDifferences=true with ignoreDifferences.jqPathExpressions that address array elements (e.g. .spec.data[].remoteRef.conversionStrategy on ExternalSecrets). ArgoCD’s sync-time normalizer cannot merge arrays when items are added or removed — an upstream-acknowledged limitation (the TestNormalizeTargetResources case for new array entries is skipped as “limitation in the current implementation”). The failure mode: the sync reports Succeeded at the right revision, map-typed fields apply, but the live array silently keeps its old value. On ExternalSecrets this leaves ESO in SecretSyncedError (a synced template can end up referencing keys the stale spec.data doesn’t fetch) and the app sits OutOfSync indefinitely.

ignoreDifferences alone (without RespectIgnoreDifferences) still normalizes the diff, so schema-defaulted fields (ESO’s conversionStrategy/decodingStrategy/metadataPolicy) don’t cause perpetual OutOfSync — the sync simply applies the true desired manifest and the API server re-defaults those fields. The apps still carrying the option have additional ignore rules (StatefulSet volumeClaimTemplates server fields, CRD /status, Argo Workflow template inputs/outputs, webhook configs) pending individual analysis before removal.

To unstick a sync already in this state, apply the git-desired manifest with ArgoCD’s own field manager:

git show origin/main:<path-to-manifest> | \
  kubectl apply --server-side --field-manager=argocd-controller --force-conflicts -f -

Verifying a version bump at the running artifact

A merged pin is not a verified bump. Every read below is a kubectl --context fzymgc-house get or an rg over the repo — no writes.

Why `Synced` + `Healthy` is not enough

Sync status answers “does the cluster match Git”, not “did the upgrade take effect”. ArgoCD renders Helm with --include-crds (so a chart’s /crds is applied by ArgoCD unless the Application sets skipCrds: true; the “Helm never upgrades /crds” pitfall is a helm upgrade hazard, not an ArgoCD one). Chart version ≠ appVersion, and a values tag: override shadows appVersion entirely (vault, renovate, external-dns, clickstack). status.sync.revisions[] is what ArgoCD synced — not what runs.

The three-clause read for a chart pin

# Clauses 1+2 (Synced, Healthy) plus the revision ArgoCD resolved per source, estate-wide
kubectl --context fzymgc-house get applications.argoproj.io -n argocd -o json \
 | jq -r '.items[] | [.metadata.name, (.metadata.annotations["argocd.argoproj.io/sync-wave"] // "0"),
     .status.sync.status, .status.health.status,
     ((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'
# one Application
kubectl --context fzymgc-house get application -n argocd <name> -o json \
 | jq -r '[.status.sync.status, .status.health.status,
     ((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'
# (`.status.sync.revisions` is null on a single-source Application — the fallback is not optional)
# Clause 3: the workloads the Application owns …
kubectl --context fzymgc-house get application -n argocd <name> -o json \
 | jq -r '.status.resources[] | select(.kind=="Deployment" or .kind=="StatefulSet" or .kind=="DaemonSet")
     | .kind+" "+.namespace+"/"+.name'
# … then chart label, appVersion label, images on each
kubectl --context fzymgc-house get <kind> -n <ns> <name> -o json \
 | jq -r '[.metadata.labels["helm.sh/chart"], .metadata.labels["app.kubernetes.io/version"],
     ([.spec.template.spec.containers[].image]|join(" "))] | @tsv'
# and the digest actually pulled (a mutable tag can be re-published)
kubectl --context fzymgc-house get pods -n <ns> -l <selector> \
  -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'

Record all three clauses plus the imageID digest, dated, beside the pin.

Where the label lies

App	What to read instead
vault	The StatefulSet carries no `helm.sh/chart` label and its image is a `tag:` override; the chart bump moves the Application revision and the injector image (`vault-agent-injector`) only
velero	appVersion may not move on a chart bump (12.0.3 → 12.1.0 kept 1.18.1); the `helm.sh/chart` label on `velero/velero` and `velero/node-agent` is the artifact
clickstack	`app.kubernetes.io/version` is the chart’s appVersion, not the running `hyperdx` image — read the image from values and the pod
renovate	The image comes from a values `tag:` override; the chart label moves, the image does not
temporal-worker-controller	No `helm.sh/chart` label at all; read `app.kubernetes.io/version` on the manager Deployment plus the `temporalworkerdeployments.temporal.io` CRD

CRD-bearing charts

Before the bump, locate the CRDs in the pulled chart:

helm pull <repo>/<chart> --version <v> --untar -d /tmp/chart && cd /tmp/chart/<chart>
ls crds/ 2>/dev/null; rg -l 'kind: CustomResourceDefinition' templates/; find charts -type d -name crds

Still before the bump, record each CRD’s resourceVersion — the after-read is meaningless without it: kubectl --context fzymgc-house get crd <name> -o jsonpath='{.metadata.resourceVersion}'. After the bump, the same read has changed only if the chart’s CRD file actually differs (diff the two pulled charts’ crds/ first — identical CRDs are a no-op apply and their resourceVersion stays put), and the dependent CRs still admit (kubectl --context fzymgc-house get <cr-kind> -A). kubectl apply is forbidden for a CRD remedy — selfHeal reverts it on the next sync. Express the remedy in the repository (for example a second Helm source for a -crds sibling chart at the same or an earlier sync-wave) and let ArgoCD reconcile it.

chart ≠ appVersion

helm show chart <repo>/<chart> --version <v> prints both; record both per bump. Helm 4 quirk: helm show chart --repo <url> fails with no cached repo found against a stale repo cache — set HELM_REPOSITORY_CACHE/HELM_REPOSITORY_CONFIG to a scratch dir, or read <repo>/index.yaml directly with yq.

Kustomize `helmCharts:` apps

The pin is helmCharts[].version in argocd/app-configs/<app>/kustomization.yaml. Before changing it, run the normalized render diff in argocd/CLAUDE.md (§ “Change safety” under the kustomize-helm notes). kubectl kustomize --enable-helm writes a gitignored charts/ dir into the app-config — never commit it.

Image pins under app-configs

kubectl --context fzymgc-house get deploy,sts,ds -n <ns> -o jsonpath='{..image}' must show the committed tag; where the pin is tag@sha256:…, re-pin the digest with the tag (crane digest <image>:<tag>) and confirm the pod’s imageID matches.

Other surfaces

Ansible: run the role through its wrapper — scripts/nas-playbook.sh --tags <role>, scripts/dns-playbook.sh --limit <node> --tags <role>, scripts/heimdall-playbook.sh; router and cluster plays via ansible-playbook -i inventory/hosts.yml <play>.yml --tags <role> — with --check --diff first, then apply, then read the installed version on the node (<binary> --version, docker ps --format '{{.Image}}', or the cluster-visible image). A merged pin that was not applied is not verified.
Terraform: the PR-triggered HCP Terraform run log’s provider line (“Installing hashicorp/random v3.9.1…”) plus the .terraform.lock.hcl change in the same PR. Never a manual run.
Action SHAs: the merged PR’s workflow run resolving the new SHA in the job’s “Set up job” log (Download action repository '<owner>/<action>@<sha>'). Dereference annotated tags to the commit before writing uses:.

Sync-wave discipline for a sweep

One PR per negative-wave Application, one PR per positive wave, merged in ascending wave order. Each negative wave reaches Synced + Healthy + the third clause before any positive-wave dependent moves. Majors go in their own PR after the wave’s minor/patch PR is verified. OC-8 is a sweep-time discipline: in steady state a single-dependent minor rides the automerge lane and majors are attended by packageRules — no standing gate enforces wave order, by design.

Access

URL	Purpose
https://argocd.fzymgc.house	Web UI
https://argocd-wh.fzymgc.net/api/webhook	GitHub webhook endpoint

Cloudflare Operations - Tunnel configuration
HCP Terraform - GitOps automation
Vault Operations - Secret management
argocd/CLAUDE.md (repo file) - kustomize helmCharts: render-diff recipe for chart/values changes