Skip to content

GitHub Credential Operations

Every GitHub credential this estate uses is a GitHub App, except the HCP Terraform VCS connection, which is an OAuth client that HCP Terraform holds. Each App’s private key lives in Vault and reaches its consumer through an ExternalSecret or an ephemeral Terraform read. No personal access token is in use. This page states which credentials exist, where each lives, and the procedures that create and rotate them.

Credential Vault path Keys Consumer
Actions runner App (org) secret/fzymgc-house/cluster/github github_org_app_id, github_org_app_private_key, github_org_app_installation_id ExternalSecret github-token in arc-runners (argocd/app-configs/arc-runners/github-token-secret.yaml)
fzymgc-terraform App secret/fzymgc-house/infrastructure/github-terraform client_id, installation_id, private_key The github provider in tf/core-services/github.tf, read ephemerally
Renovate App secret/fzymgc-house/cluster/renovate github_app_id, github_app_private_key, github_webhook_secret argocd/app-configs/renovate/external-secret.yaml. See Renovate
Octopus App secret/fzymgc-house/cluster/octopus github_app_id, github_app_private_key, github_webhook_secret, github_state_secret, github_client_id, github_client_secret argocd/app-configs/octopus/secrets.yaml. See Octopus
Temporal workers App secret/fzymgc-house/cluster/temporal/github app_id, installation_id, private_key argocd/cluster-app/templates/temporal-workers.yaml
HCP Terraform VCS None. HCP Terraform holds the OAuth client data.tfe_oauth_client.github in tf/hcp-terraform/data.tf
Cloudflare Access GitHub login None. Two sensitive workspace variables on main-cluster-cloudflare github_oauth_client_id, github_oauth_client_secret tf/cloudflare/access.tf. The identity provider exists only when the id is set

Actions Runner Controller runs as two ArgoCD applications. arc-controller installs the gha-runner-scale-set-controller chart in arc-systems. arc-runners installs the gha-runner-scale-set chart in arc-runners with the scale set fzymgc-house-cluster-runners, minRunners: 0, maxRunners: 3 and Docker-in-Docker mode. argocd/app-configs/arc-runners/README.md carries the chart notes.

The scale set authenticates as a GitHub App installed on the fzymgc-house organization. The ExternalSecret reads the three keys from Vault every 15 seconds and renders the github-token Secret that githubConfigSecret names in values.yaml.

The runner pods run the image pinned in argocd/app-configs/arc-runners/values.yaml, built from images/actions-runner/Dockerfile by the workflow .github/workflows/build-runner-image.yaml.

  1. On the App’s settings page under the fzymgc-house organization, generate a new private key. The .pem downloads once.

  2. Write it to Vault from the file, never as an inline literal:

    Terminal window
    vault kv patch secret/fzymgc-house/cluster/github github_org_app_private_key=@"$HOME/Downloads/<file>.pem"
    rm -f "$HOME/Downloads/<file>.pem"
  3. Make sure that the ExternalSecret synced and the listener re-registered:

    Terminal window
    kubectl --context fzymgc-house -n arc-runners get externalsecret github-token # Ready True
    kubectl --context fzymgc-house -n arc-systems get pods # controller and listener Running
    gh api /orgs/fzymgc-house/actions/runners --jq '.runners[] | {name, status}'
  4. Revoke the old key on the App’s settings page.

A successful registration does not mean a runner can receive jobs. GitHub retires runner versions, and a retired runner is rejected with Runner version <v> is deprecated and cannot receive messages while the controller and listener stay healthy. That is an image version failure, not a credential failure.

  1. Raise ARG RUNNER_VERSION in images/actions-runner/Dockerfile, or run the build workflow with its runner_version input.

  2. After the multi-architecture build succeeds, update the image pin in argocd/app-configs/arc-runners/values.yaml in a pull request. Building the image does not update the cluster.

  3. After sync, make sure that a queued job starts on the new version:

    Terminal window
    gh run list --limit 3 --json databaseId,status,conclusion
Terminal window
kubectl --context fzymgc-house -n arc-systems get pods
kubectl --context fzymgc-house -n arc-systems logs <controller pod> --tail=100
kubectl --context fzymgc-house -n arc-runners describe externalsecret github-token
  • 401: the App’s private key or id is wrong in Vault.
  • 403: the App lacks a permission, or the installation does not cover the repository.
  • ExternalSecret not Ready: the Vault path or a property name is wrong, or the vault ClusterSecretStore is unhealthy.

GitHub App: fzymgc-terraform (Terraform write identity)

Section titled “GitHub App: fzymgc-terraform (Terraform write identity)”

This App is the github provider’s credential in tf/core-services. It manages github_repository_ruleset.main_branch_protection, the protection on main, and the alert issue label in github_issue_label.alert. Terraform reads the three keys ephemerally at plan time and never writes them, so the App is created, installed and seeded by hand.

The path sits outside cluster/ on purpose. The cluster’s ExternalSecret store reads only under cluster/, so no ExternalSecret can copy this key. The policy terraform-core-services-admin in tf/vault/policy-terraform-workspaces.tf grants the workspace read on the data and read and list on the metadata, and nothing more.

Do not confuse this path with secret/fzymgc-house/cluster/github. That is the runner App, a different identity with a different blast radius and different key names. Rotating the wrong one breaks the other.

Both prerequisites must hold before the plan that reads the secret runs. The secret must exist, and tf/vault must have applied the grant. Until the grant applies, the ephemeral read fails with a 403 that reads like a missing secret. The workspaces carry no run triggers between them, so land the grant alone and first.

  1. Create the App at https://github.com/organizations/fzymgc-house/settings/apps/new, named fzymgc-terraform. Turn the webhook off. Record the Client ID. The Client ID, not the numeric App ID, is what Vault stores and what the provider sends as the JWT iss claim.

  2. Set the repository permissions Administration and Issues to Read & write, and no other. Metadata read-only is selected automatically. Do not soften Administration to read-only: every ruleset endpoint, the reads included, sits under Administration at write access, so a read-only App fails the import’s GET with a 403. The label endpoints need Issues at write access. Without it the plan passes, and the apply on merge fails with a 403.

  3. Install the App on fzymgc-house/selfhosted-cluster only. Record the installation id, the trailing number in the URL after install. The App holds write authority over the protection that gates every other change in this repository. Do not widen the installation.

  4. Generate a private key. The .pem downloads once. A lost key means a new key and a new Vault write, not a new App.

  5. Write the three keys to Vault and verify by metadata only:

    Terminal window
    vault token lookup
    PEM="$HOME/Downloads/fzymgc-terraform.<date>.private-key.pem"
    vault kv put secret/fzymgc-house/infrastructure/github-terraform \
    client_id="<client-id>" \
    installation_id="<installation-id>" \
    private_key=@"$PEM"
    vault kv metadata get secret/fzymgc-house/infrastructure/github-terraform # current_version rose, destroyed false
    rm -f "$PEM"

    Do not vault kv get this path. That prints the private key. The key must never reach an HCP Terraform workspace variable, a pull request body, a commit or an inline shell argument.

Generate a new key on the App, run the vault kv put above with the same client_id and installation_id and the new file, then revoke the old key on the App. The next plan in main-cluster-core-services reads the new version.

403 Resource not accessible by integration (GitHub App auth)

Section titled “403 Resource not accessible by integration (GitHub App auth)”

Two causes produce this message from the github provider. Read them in this order:

  1. The App has Administration: Read-only. Set it to Read & write, then accept the permission change on the installation.
  2. The provider block omits owner. Under App auth the provider cannot infer the owner from an installation token. tf/core-services/github.tf sets owner = "fzymgc-house". Keep it.

If both are correct, make sure that the installation covers fzymgc-house/selfhosted-cluster. An App installed with the wrong repository selected fails the same way.

  • Never commit a token or key to Git. Vault is the only home.
  • Grant each App the minimum permission its consumer uses, and install it on the smallest set of repositories that works.
  • Write a private key to Vault from a file with @<file>, never as an inline literal, so it stays out of shell history and the process table.
  • Verify a Vault write with vault kv metadata get, never with vault kv get, on any path that holds a private key.
  • Rotate at once when a key is exposed, and revoke the old key on the App after the new one is in Vault and its consumer has re-read it.