GitHub Credential Operations
Every GitHub credential this estate uses is a GitHub App, except the HCP Terraform VCS connection, which is an OAuth client that HCP Terraform holds. Each App’s private key lives in Vault and reaches its consumer through an ExternalSecret or an ephemeral Terraform read. No personal access token is in use. This page states which credentials exist, where each lives, and the procedures that create and rotate them.
Quick Reference
Section titled “Quick Reference”| Credential | Vault path | Keys | Consumer |
|---|---|---|---|
| Actions runner App (org) | secret/fzymgc-house/cluster/github |
github_org_app_id, github_org_app_private_key, github_org_app_installation_id |
ExternalSecret github-token in arc-runners (argocd/app-configs/arc-runners/github-token-secret.yaml) |
fzymgc-terraform App |
secret/fzymgc-house/infrastructure/github-terraform |
client_id, installation_id, private_key |
The github provider in tf/core-services/github.tf, read ephemerally |
| Renovate App | secret/fzymgc-house/cluster/renovate |
github_app_id, github_app_private_key, github_webhook_secret |
argocd/app-configs/renovate/external-secret.yaml. See Renovate |
| Octopus App | secret/fzymgc-house/cluster/octopus |
github_app_id, github_app_private_key, github_webhook_secret, github_state_secret, github_client_id, github_client_secret |
argocd/app-configs/octopus/secrets.yaml. See Octopus |
| Temporal workers App | secret/fzymgc-house/cluster/temporal/github |
app_id, installation_id, private_key |
argocd/cluster-app/templates/temporal-workers.yaml |
| HCP Terraform VCS | None. HCP Terraform holds the OAuth client | data.tfe_oauth_client.github in tf/hcp-terraform/data.tf |
|
| Cloudflare Access GitHub login | None. Two sensitive workspace variables on main-cluster-cloudflare |
github_oauth_client_id, github_oauth_client_secret |
tf/cloudflare/access.tf. The identity provider exists only when the id is set |
The Actions runner App
Section titled “The Actions runner App”Actions Runner Controller runs as two ArgoCD applications. arc-controller installs the
gha-runner-scale-set-controller chart in arc-systems. arc-runners installs the
gha-runner-scale-set chart in arc-runners with the scale set fzymgc-house-cluster-runners,
minRunners: 0, maxRunners: 3 and Docker-in-Docker mode. argocd/app-configs/arc-runners/README.md
carries the chart notes.
The scale set authenticates as a GitHub App installed on the fzymgc-house organization. The
ExternalSecret reads the three keys from Vault every 15 seconds and renders the github-token
Secret that githubConfigSecret names in values.yaml.
The runner pods run the image pinned in argocd/app-configs/arc-runners/values.yaml, built from
images/actions-runner/Dockerfile by the workflow .github/workflows/build-runner-image.yaml.
Rotate the runner App’s private key
Section titled “Rotate the runner App’s private key”-
On the App’s settings page under the
fzymgc-houseorganization, generate a new private key. The.pemdownloads once. -
Write it to Vault from the file, never as an inline literal:
Terminal window vault kv patch secret/fzymgc-house/cluster/github github_org_app_private_key=@"$HOME/Downloads/<file>.pem"rm -f "$HOME/Downloads/<file>.pem" -
Make sure that the ExternalSecret synced and the listener re-registered:
Terminal window kubectl --context fzymgc-house -n arc-runners get externalsecret github-token # Ready Truekubectl --context fzymgc-house -n arc-systems get pods # controller and listener Runninggh api /orgs/fzymgc-house/actions/runners --jq '.runners[] | {name, status}' -
Revoke the old key on the App’s settings page.
Update the runner version
Section titled “Update the runner version”A successful registration does not mean a runner can receive jobs. GitHub retires runner
versions, and a retired runner is rejected with Runner version <v> is deprecated and cannot receive messages while the controller and listener stay healthy. That is an image version
failure, not a credential failure.
-
Raise
ARG RUNNER_VERSIONinimages/actions-runner/Dockerfile, or run the build workflow with itsrunner_versioninput. -
After the multi-architecture build succeeds, update the image pin in
argocd/app-configs/arc-runners/values.yamlin a pull request. Building the image does not update the cluster. -
After sync, make sure that a queued job starts on the new version:
Terminal window gh run list --limit 3 --json databaseId,status,conclusion
Troubleshoot the runners
Section titled “Troubleshoot the runners”kubectl --context fzymgc-house -n arc-systems get podskubectl --context fzymgc-house -n arc-systems logs <controller pod> --tail=100kubectl --context fzymgc-house -n arc-runners describe externalsecret github-token401: the App’s private key or id is wrong in Vault.403: the App lacks a permission, or the installation does not cover the repository.- ExternalSecret not
Ready: the Vault path or a property name is wrong, or thevaultClusterSecretStore is unhealthy.
GitHub App: fzymgc-terraform (Terraform write identity)
Section titled “GitHub App: fzymgc-terraform (Terraform write identity)”This App is the github provider’s credential in tf/core-services. It manages
github_repository_ruleset.main_branch_protection, the protection on main, and the alert issue
label in github_issue_label.alert. Terraform reads the
three keys ephemerally at plan time and never writes them, so the App is created, installed and
seeded by hand.
The path sits outside cluster/ on purpose. The cluster’s ExternalSecret store reads only under
cluster/, so no ExternalSecret can copy this key. The policy terraform-core-services-admin in
tf/vault/policy-terraform-workspaces.tf grants the workspace read on the data and read and
list on the metadata, and nothing more.
Do not confuse this path with secret/fzymgc-house/cluster/github. That is the runner App, a
different identity with a different blast radius and different key names. Rotating the wrong one
breaks the other.
Create or recreate the App
Section titled “Create or recreate the App”Both prerequisites must hold before the plan that reads the secret runs. The secret must exist,
and tf/vault must have applied the grant. Until the grant applies, the ephemeral read fails with
a 403 that reads like a missing secret. The workspaces carry no run triggers between them, so land
the grant alone and first.
-
Create the App at https://github.com/organizations/fzymgc-house/settings/apps/new, named
fzymgc-terraform. Turn the webhook off. Record the Client ID. The Client ID, not the numeric App ID, is what Vault stores and what the provider sends as the JWTissclaim. -
Set the repository permissions
AdministrationandIssuestoRead & write, and no other. Metadata read-only is selected automatically. Do not softenAdministrationto read-only: every ruleset endpoint, the reads included, sits under Administration at write access, so a read-only App fails the import’s GET with a 403. The label endpoints needIssuesat write access. Without it the plan passes, and the apply on merge fails with a 403. -
Install the App on
fzymgc-house/selfhosted-clusteronly. Record the installation id, the trailing number in the URL after install. The App holds write authority over the protection that gates every other change in this repository. Do not widen the installation. -
Generate a private key. The
.pemdownloads once. A lost key means a new key and a new Vault write, not a new App. -
Write the three keys to Vault and verify by metadata only:
Terminal window vault token lookupPEM="$HOME/Downloads/fzymgc-terraform.<date>.private-key.pem"vault kv put secret/fzymgc-house/infrastructure/github-terraform \client_id="<client-id>" \installation_id="<installation-id>" \private_key=@"$PEM"vault kv metadata get secret/fzymgc-house/infrastructure/github-terraform # current_version rose, destroyed falserm -f "$PEM"Do not
vault kv getthis path. That prints the private key. The key must never reach an HCP Terraform workspace variable, a pull request body, a commit or an inline shell argument.
Rotate the App’s private key
Section titled “Rotate the App’s private key”Generate a new key on the App, run the vault kv put above with the same client_id and
installation_id and the new file, then revoke the old key on the App. The next plan in
main-cluster-core-services reads the new version.
403 Resource not accessible by integration (GitHub App auth)
Section titled “403 Resource not accessible by integration (GitHub App auth)”Two causes produce this message from the github provider. Read them in this order:
- The App has
Administration: Read-only. Set it toRead & write, then accept the permission change on the installation. - The provider block omits
owner. Under App auth the provider cannot infer the owner from an installation token.tf/core-services/github.tfsetsowner = "fzymgc-house". Keep it.
If both are correct, make sure that the installation covers fzymgc-house/selfhosted-cluster. An
App installed with the wrong repository selected fails the same way.
Constraints
Section titled “Constraints”- Never commit a token or key to Git. Vault is the only home.
- Grant each App the minimum permission its consumer uses, and install it on the smallest set of repositories that works.
- Write a private key to Vault from a file with
@<file>, never as an inline literal, so it stays out of shell history and the process table. - Verify a Vault write with
vault kv metadata get, never withvault kv get, on any path that holds a private key. - Rotate at once when a key is exposed, and revoke the old key on the App after the new one is in Vault and its consumer has re-read it.
See Also
Section titled “See Also”- Vault Operations: secret storage
- Secrets Reference: every Vault path and its keys
- HCP Terraform Operations: the break-glass path past a blocked required check on
main - Actions Runner Controller