ArgoCD Operations
ArgoCD deploys every application in the cluster from this repository. This page states what
exists, where it is defined, and the procedures that touch it. Commands assume
--context fzymgc-house.
What exists
Section titled “What exists”| Property | Value |
|---|---|
| Install | helm_release.argocd in tf/cluster-bootstrap/argocd.tf, chart argo-cd, version var.argocd_version |
| Namespace | argocd |
| Web UI | https://argocd.fzymgc.house, ingress certificate from fzymgc-house-issuer |
| Login | OIDC through Dex against the Keycloak realm fzymgc. Group argocd-admin maps to role:admin, argocd-user to role:readonly |
| Replicas | server 2, repo-server 2, controller 1 |
| Applications | argocd/cluster-app/, a Helm chart with one Application template per app |
| Kustomize | kustomize.buildOptions: --enable-helm, so a kustomization can inflate a Helm chart |
| Webhook endpoint | https://argocd-wh.fzymgc.net/api/webhook, through the Cloudflare tunnel |
Secrets
Section titled “Secrets”| Kubernetes Secret | Source | Holds |
|---|---|---|
argocd-secret |
ExternalSecret from fzymgc-house/cluster/argocd |
admin.password, server.secretkey, webhook.github.secret |
argocd-oidc-secret |
Terraform, write-only, from fzymgc-house/cluster/argocd/oidc |
The Keycloak client id and secret |
argocd-selfhosted-repo, argocd-temporal-workers-repo |
Terraform, write-only, from fzymgc-house/cluster/argocd |
GitHub App credentials for the two repositories |
argocd-repo-server-helm-registry |
shared-resources/ghcr-helm-registry-config.yaml |
The registry login the repo-server uses for private OCI charts |
Webhook
Section titled “Webhook”GitHub sends push events to argocd-wh.fzymgc.net. The Cloudflare tunnel forwards them to
argocd-server.argocd.svc.cluster.local (webhook_services in tf/cloudflare/variables.tf).
ArgoCD checks the signature against webhook.github.secret and refreshes the applications
that use the repository, instead of waiting for its three-minute poll.
GitHub (organization webhook) -> Cloudflare Tunnel -> ArgoCD server push event argocd-wh.fzymgc.net /api/webhookProcedures
Section titled “Procedures”Configure the GitHub webhook
Section titled “Configure the GitHub webhook”-
Read the secret:
Terminal window vault kv get -field=webhook.github.secret secret/fzymgc-house/cluster/argocd -
Open https://github.com/organizations/fzymgc-house/settings/hooks and add a webhook:
Field Value Payload URL https://argocd-wh.fzymgc.net/api/webhookContent type application/jsonSecret The value from step 1 Events Just the push event Active Yes -
Push to any repository in the organization and read the delivery on the same page. A delivery with a
200response means the webhook works.
Check the webhook
Section titled “Check the webhook”| Check | Command | Expected |
|---|---|---|
| DNS | dig argocd-wh.fzymgc.net |
A CNAME to cfargotunnel.com |
| Endpoint | curl -I https://argocd-wh.fzymgc.net/api/webhook |
400 (no payload) or 200 |
| Delivery | Push to a repository and read the GitHub deliveries page | 200 |
| Refresh | Read the application after a push | The sync starts within seconds, not minutes |
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-server --tail=100 | rg -i webhook| Symptom | Cause | Fix |
|---|---|---|
401 or 403 |
The secret in GitHub differs from Vault | Paste the Vault value into the GitHub webhook |
404 |
The tunnel route is missing | Read webhook_services in tf/cloudflare/variables.tf |
502 |
ArgoCD is unreachable | Read the argocd-server pods |
| No sync after a push | The Application repository URL differs from the webhook source | Compare the two URLs |
To stop the webhook, delete it in GitHub. ArgoCD falls back to its poll. No ArgoCD change is needed.
Unstick a sync that ignores array fields
Section titled “Unstick a sync that ignores array fields”Do not combine syncOptions: RespectIgnoreDifferences=true with ignoreDifferences
jqPathExpressions that address array elements, for example
.spec.data[].remoteRef.conversionStrategy on an ExternalSecret. ArgoCD’s sync-time normalizer
cannot merge an array when items are added or removed. The sync reports Succeeded at the right
revision, map-typed fields apply, and the live array keeps its old value. On an ExternalSecret
this leaves the operator in SecretSyncedError and the application OutOfSync.
ignoreDifferences alone normalizes only the diff, so schema-defaulted fields do not cause a
standing OutOfSync, and the sync applies the true desired manifest. Six Applications still set
the option because they carry other ignore rules (StatefulSet volumeClaimTemplates fields, CRD
status, workflow template fields, webhook configurations, Secret data): clickstack,
external-dns, hcp-terraform-operator, nats, system-upgrade and vault. external-dns and
hcp-terraform-operator also keep .spec.data[] paths on an ExternalSecret, so a new key on that
ExternalSecret does not reach the cluster.
To unstick a sync in this state, apply the desired manifest with the field manager operator.
Do not use argocd-controller. ArgoCD adds the annotation argocd.argoproj.io/tracking-id at sync
time, so the manifest in git does not carry it. A server-side apply removes each field that its
field manager owns and the manifest leaves out. An apply as argocd-controller therefore removes
the annotation, and ArgoCD stops tracking the object.
Read the diff before the apply. It must show the array change, and it must not remove the tracking annotation:
git show origin/main:<path-to-manifest> > /tmp/m.yamlkubectl --context fzymgc-house diff --server-side --field-manager=operator --force-conflicts -f /tmp/m.yamlkubectl --context fzymgc-house apply --server-side --field-manager=operator --force-conflicts -f /tmp/m.yamlVerify a version bump at the running artifact
Section titled “Verify a version bump at the running artifact”A merged pin is not a verified bump. Every read below is a kubectl get or an rg over the
repository. Nothing writes.
Synced and Healthy answer “does the cluster match git”, not “did the upgrade take effect”.
ArgoCD renders Helm with --include-crds, so a chart’s crds/ directory is applied unless the
Application sets skipCrds: true. Chart version and appVersion differ, and a values tag:
override shadows the appVersion. status.sync.revisions[] is what ArgoCD synced, not what runs.
Read three things for a chart pin:
# Clauses 1+2 (Synced, Healthy) plus the revision ArgoCD resolved per source, estate-widekubectl --context fzymgc-house get applications.argoproj.io -n argocd -o json \ | jq -r '.items[] | [.metadata.name, (.metadata.annotations["argocd.argoproj.io/sync-wave"] // "0"), .status.sync.status, .status.health.status, ((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'# one Applicationkubectl --context fzymgc-house get application -n argocd <name> -o json \ | jq -r '[.status.sync.status, .status.health.status, ((.status.sync.revisions // [.status.sync.revision]) | join(","))] | @tsv'# (`.status.sync.revisions` is null on a single-source Application, so the fallback is required)# Clause 3: the workloads the Application ownskubectl --context fzymgc-house get application -n argocd <name> -o json \ | jq -r '.status.resources[] | select(.kind=="Deployment" or .kind=="StatefulSet" or .kind=="DaemonSet") | .kind+" "+.namespace+"/"+.name'# then chart label, appVersion label, images on eachkubectl --context fzymgc-house get <kind> -n <ns> <name> -o json \ | jq -r '[.metadata.labels["helm.sh/chart"], .metadata.labels["app.kubernetes.io/version"], ([.spec.template.spec.containers[].image]|join(" "))] | @tsv'# and the digest the node pulled (a mutable tag can be re-published)kubectl --context fzymgc-house get pods -n <ns> -l <selector> \ -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'Record the three reads and the imageID in the pull request.
Some applications do not carry the label that names the running version:
| App | What to read instead |
|---|---|
vault |
The StatefulSet carries no helm.sh/chart label and its image is a tag: override. A chart bump moves the Application revision and the vault-agent-injector image only |
velero |
The appVersion can stay put on a chart bump. The helm.sh/chart label on velero/velero and velero/node-agent is the artifact |
clickstack |
app.kubernetes.io/version is the chart’s appVersion, not the running hyperdx image. Read the image from the values and the pod |
renovate |
The image comes from a values tag: override. The chart label moves and the image does not |
temporal-worker-controller |
No helm.sh/chart label. Read app.kubernetes.io/version on the manager Deployment and the workerdeployments.temporal.io CRD |
For a chart that carries CRDs, locate them in the pulled chart before the bump and record each
CRD’s resourceVersion:
helm pull <repo>/<chart> --version <v> --untar -d /tmp/chart && cd /tmp/chart/<chart>ls crds/ 2>/dev/null; rg -l 'kind: CustomResourceDefinition' templates/; find charts -type d -name crdskubectl --context fzymgc-house get crd <name> -o jsonpath='{.metadata.resourceVersion}'After the bump, the resourceVersion changes only if the chart’s CRD file differs. Diff the two
pulled charts’ crds/ first. An identical CRD is a no-op apply. Then make sure that the dependent
objects still admit: kubectl --context fzymgc-house get <cr-kind> -A. Never kubectl apply a
CRD as a remedy, because selfHeal reverts it on the next sync. Express the remedy in the
repository, for example a second Helm source for a -crds sibling chart at the same or an
earlier sync wave.
helm show chart <repo>/<chart> --version <v> prints the chart version and the appVersion.
Record both per bump. Helm 4 fails helm show chart --repo <url> with no cached repo found
against a stale repository cache. Set HELM_REPOSITORY_CACHE and HELM_REPOSITORY_CONFIG to a
scratch directory, or read <repo>/index.yaml with yq.
For a kustomize helmCharts: app, the pin is helmCharts[].version in
argocd/app-configs/<app>/kustomization.yaml. Before changing it, run the render diff in
argocd/CLAUDE.md, section Kustomize Helm-chart inflation. kubectl kustomize --enable-helm
writes a gitignored charts/ directory into the app-config. Never commit it.
For an image pin under app-configs, kubectl --context fzymgc-house get deploy,sts,ds -n <ns> -o jsonpath='{..image}'
must show the committed tag. Where the pin is tag@sha256:..., re-pin the digest with the tag
(crane digest <image>:<tag>) and make sure that the pod’s imageID matches.
Other surfaces:
- Ansible: run the role through its wrapper (
scripts/nas-playbook.sh --tags <role>,scripts/dns-playbook.sh --limit <node> --tags <role>,scripts/heimdall-playbook.sh, oransible-playbook -i inventory/hosts.yml <play>.yml --tags <role>for the cluster plays), with--check --difffirst, then apply, then read the installed version on the node. A merged pin that was not applied is not verified. - Terraform: the HCP Terraform run log’s provider line plus the
.terraform.lock.hclchange in the same PR. Never a manual run. - Action SHAs: the merged PR’s workflow run resolves the new SHA in the job’s “Set up job” log.
Dereference an annotated tag to its commit before writing
uses:.
Sweep several bumps in wave order
Section titled “Sweep several bumps in wave order”One PR per negative-wave Application and one PR per positive wave, merged in ascending wave
order. Each negative wave reaches Synced, Healthy and the third read before any dependent in a
positive wave moves. Majors go in their own PR after the wave’s minor and patch PR is verified.
In steady state a single-dependent minor rides the automerge lane and majors are attended by
packageRules. No standing gate enforces wave order.
Access
Section titled “Access”| URL | Purpose |
|---|---|
| https://argocd.fzymgc.house | Web UI |
| https://argocd-wh.fzymgc.net/api/webhook | GitHub webhook endpoint |
Related
Section titled “Related”- Cloudflare Operations, the tunnel
- HCP Terraform
- Vault Operations
argocd/CLAUDE.md, the kustomizehelmCharts:render-diff recipe