Skip to content

Temporal Operations

Temporal runs the cluster’s workflows. This page states what exists, where each part is defined, and the procedures that touch it. Commands assume --context fzymgc-house.

Property Value
Web UI https://temporal.fzymgc.house, behind the shared oauth2-proxy forward-auth middleware oauth-auth-temporal, which admits the Keycloak group temporal-users
Frontend temporal-frontend.temporal.svc.cluster.local:7233
Node gateway 192.168.250.146:7233 and [fddb:f665:73f7:250::146]:7233, the mTLS entry for the resolver Pis. See The node gateway
Namespace temporal
Temporal namespaces workflows for the cluster workers, dns-control for dns-control and the node workers
Server chart temporal from https://go.temporal.io/helm-charts, pinned in argocd/cluster-app/templates/temporal-server.yaml with the server, web and admin-tools image tags
Manifests argocd/app-configs/temporal-server/ (namespace, ingress, certificate, database secret, CA chain)
Workers repository fzymgc-house/temporal-workers

The group temporal-users and the oauth2-proxy client come from tf/keycloak/oauth2_proxy.tf. The design is in the forward-auth ADR.

Temporal uses two databases on the CNPG cluster main in the postgres namespace. Each is a Database object under argocd/app-configs/cnpg/, and each has its own schema_version table, so the two schemas cannot collide.

Database Object Purpose
temporal db-temporal.yaml Workflow state and history
temporal_visibility db-temporal-visibility.yaml Workflow search and listing

Both datastores connect to main-rw.postgres.svc.cluster.local:5432 as user temporal over TLS, with the CA from the vault-ca-chain Secret. The password is temporal-db-secret, read by an ExternalSecret from fzymgc-house/cluster/postgres/users/main-temporal. Both datastores set manageSchema: true, so the chart’s temporal-schema Job runs as an ArgoCD PreSync hook and updates the schema before the server pods change.

Each server role runs one replica. The chart values set numHistoryShards: 128 and turn on the frontend.WorkerHeartbeatsEnabled and frontend.ListWorkersEnabled dynamic configuration.

Component Purpose
temporal-frontend API gateway and client connections
temporal-history Workflow state
temporal-matching Task queue routing
temporal-worker Internal workflows
temporal-web Web UI
temporal-admintools The tctl and temporal CLIs
Application Chart What it does
temporal-worker-controller and temporal-worker-controller-crds docker.io/temporalio charts, the same version in both templates The controller reconciles WorkerDeployment and Connection objects and rolls workers progressively
temporal-workers ghcr.io/fzymgc-house/charts/temporal-workers, pinned in argocd/cluster-app/templates/temporal-workers.yaml. The values are in argocd/app-configs/temporal-workers/values.yaml The terraform worker: it plans the main-cluster-vault and main-cluster-core-services workspaces, posts to Discord, and applies tf/hcp-terraform under a guard that needs approval on every plan
temporal-webhook-receiver ghcr.io/fzymgc-house/charts/temporal-webhook-receiver, the same version as the workers chart Receives webhooks and starts workflows on the frontend

The two controller charts must stay on one version, because the CRDs and the controller change together. A workers chart that changes CRD kinds needs the matching controller first. A new chart deploys only after a pull request here moves targetRevision.

A new worker image deploys without a person. Within five minutes of a release in the workers repository, Argo CD Image Updater opens a pull request. It moves workers.terraform.image.tag in argocd/app-configs/temporal-workers/values.yaml, and it merges after its checks pass. The image is ghcr.io/fzymgc-house/temporal-workers/terraform. The chart reads workers.<worker>.image from chart 3.3.0 on. A new worker needs its own workers.<worker>.image keys in the values file and its own image in argocd/app-configs/argocd-image-updater/temporal-workers.yaml.

After a worker release, the old worker version keeps running until each workflow pinned to it ends. A pinned workflow stays on the worker version that started it. The controller then waits one hour (scaledownDelay: 1h) and stops the old version. Both images are live for that time. A drift check that waits for approval holds its version for up to four hours. If an old version runs longer, see An old worker version does not stop.

The workers run as the temporal-worker ServiceAccount with the ghcr-pull-secret (argocd/app-configs/temporal-workers/). They send OTLP to the ClickStack collector with the ingest key from fzymgc-house/cluster/clickstack.

A drift check that finds drift posts Approve and Reject buttons to Discord. Each button holds an approval ID, a ULID that the check also sets as the search attribute ApprovalId. The webhook receiver finds the waiting workflow by that attribute and signals it. A button ID is at most 54 characters, and Discord allows 100.

The search attribute must exist in the namespace workflows before a worker sets it. A workflow that sets an unknown search attribute fails its workflow task on every retry. If you rebuild the namespace, create the attribute again before the workers start:

Terminal window
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal operator search-attribute create -n workflows \
--address temporal-frontend.temporal.svc.cluster.local:7233 \
--name ApprovalId --type Keyword

Image Updater moves the worker image. The receiver moves only when a pull request here moves the targetRevision of temporal-webhook-receiver, so a change to the button format needs both. Until the receiver moves, it answers “Invalid request” to a new button. The workflow keeps waiting, so a click after the move works.

dns-control and the node workers on the resolver Pis use the Temporal namespace dns-control. The chart creates it. The values in argocd/cluster-app/templates/temporal-server.yaml list it under server.config.namespaces with a retention of 7 days. The retention is the time that Temporal keeps a closed workflow.

The chart Job temporal-namespace runs after each install and upgrade. Its init container create-dns-control-namespace runs this command, which is safe to run twice:

Terminal window
temporal operator namespace describe -n dns-control || temporal operator namespace create -n dns-control --retention 7d

The Job only creates the namespace. A change of the retention in the values does not change the namespace that exists. To change the retention, change the values and run temporal operator namespace update in the same window. See Manage namespaces.

Task queue Poller What runs on it
dns-control The control worker in the dns-control pod The workflows ops.action and ops.status and their control activities
dns-node-sandiego The node worker on sandiego The six node activities of sandiego
dns-node-portland The node worker on portland The six node activities of portland

dns-control reaches the frontend in the cluster, at temporal-frontend.temporal.svc.cluster.local:7233. The Pis reach it only through the node gateway.

The resolver Pis are outside the cluster. They reach Temporal through a gateway on a Lab VIP. The gateway ends the mTLS of the Pis, asks a policy engine about each call, and forwards an allowed call to the frontend. The in-cluster tenants keep the plaintext frontend and do not change.

Property Value
Address 192.168.250.146:7233 and [fddb:f665:73f7:250::146]:7233, routed VIPs announced over BGP
Service temporal/temporal-node-gateway-vip, type LoadBalancer, externalTrafficPolicy: Local
Workload Deployment/temporal-node-gateway, two replicas, each with the containers envoy and opa
Manifests argocd/app-configs/temporal-server/node-gateway/, synced by the application temporal-server
Server certificate temporal-node-gateway-tls for temporal-nodes.fzymgc.house, from vault-issuer, RSA 2048
Client trust The CA chain vault-ca-chain, with a URI SAN under the prefix spiffe://fzymgc.house/dns-node-worker/
Ingress The CiliumNetworkPolicy admits the resolver Pis only: 192.168.40.6/32 (sandiego) and 192.168.40.7/32 (portland), their Core addresses. A Pi reaches the routed VIP through its default route on Core

Envoy listens on port 7233. The listener requires a client certificate and TLS 1.3. The trust bundle is the full ICA1 chain, so every certificate that ICA1 issues completes the chain check. The SAN match then admits only a certificate with a URI SAN under spiffe://fzymgc.house/dns-node-worker/. Only the PKI roles of the node workers in Vault allow that URI SAN. The cert-manager role issues any DNS name, so a match on a DNS SAN admits a Certificate from any namespace.

Envoy forwards an allowed call over plaintext HTTP/2 to the frontend. The route timeout is 0 seconds, because a poll holds its stream for about a minute. Every hour Envoy closes each connection with GOAWAY. The Pi then connects again, and the new handshake loads a renewed client certificate.

The server certificate is RSA 2048 and not ECDSA. The ClusterIssuer vault-issuer signs through the Vault role fzymgc-house/v1/ica1/v1/roles/cert-manager, which sets key_type rsa. Vault refuses a request whose key type differs from the role. Reloader rolls the pods when the certificate renews.

Envoy calls OPA through ext_authz for each gRPC call, with the request body. OPA is the Open Policy Agent, a policy engine. OPA decodes the protobuf body with the descriptor set temporal-api.binpb, so the policy reads the namespace from the body. If OPA does not answer, Envoy denies the call. DNS answers never depend on the gateway.

The policy is in policy/policy.rego, and policy/data.json binds each node to its namespace. Envoy takes the principal from the first URI SAN of the client certificate. The policy allows a WorkflowService call when all of these conditions are true:

  1. The principal is spiffe://fzymgc.house/dns-node-worker/<node>, and data.json names the node.
  2. The method is one of the calls of an activity worker: DescribeNamespace, PollActivityTaskQueue, RespondActivityTaskCompleted, RespondActivityTaskFailed, RespondActivityTaskCanceled, RecordActivityTaskHeartbeat, RecordWorkerHeartbeat or ShutdownWorker.
  3. Envoy passed the full body, not a part of it.
  4. The namespace in the body equals the namespace that the node is bound to, dns-control.
  5. If the call carries the header temporal-namespace, the header equals the body.
  6. PollActivityTaskQueue and ShutdownWorker name the task queue of the node, dns-node-<node>. ShutdownWorker names no sticky queue, because the node runs no workflow worker.

The health check and GetSystemInfo pass for a known node. The policy denies every other method, the *ById variants of the activity calls, OperatorService and AdminService. A stolen Pi certificate therefore reaches no other namespace and no other task queue.

The policy has two limits:

  • The Respond calls and RecordActivityTaskHeartbeat pass on the namespace alone. The policy does not read their task token.
  • A later change that turns on frontend.WorkerCommandsEnabled makes the SDK poll PollNexusTaskQueue. The gateway denies that call, and the worker logs a warning every second. Add the call to the policy in the same change.

Run the policy tests before a change:

Terminal window
mise //argocd/app-configs/temporal-server/node-gateway:policy:test

The pr-gates member temporal-gateway-policy runs the same tests when a pull request changes a file under argocd/app-configs/temporal-server/node-gateway/policy/. The nightly pr-gates run also runs them. After a change of the Temporal API version, rebuild the descriptor set with mise //argocd/app-configs/temporal-server/node-gateway:descriptor:build.

OPA writes each decision to the log of its container, with the principal of the caller. The rule in policy/log_mask.rego drops the request body from each entry and keeps only its namespace and taskQueue. A body can carry activity results and failures of up to 4 MiB. Read the decision log and the Envoy log:

Terminal window
kubectl logs -n temporal deploy/temporal-node-gateway -c opa --tail=100
kubectl logs -n temporal deploy/temporal-node-gateway -c envoy --tail=100

The node workers send uncompressed gRPC frames. OPA cannot decode a compressed body, so it denies every call from a client that compresses its frames.

Each resolver Pi runs the node worker dns-node-worker. The Ansible role dns-node-worker installs it. The worker runs activities only and no workflow. It polls the task queue dns-node-<node> in the namespace dns-control: dns-node-sandiego and dns-node-portland.

Property Value
Dial 192.168.250.146:7233 by IP literal, with the server name temporal-nodes.fzymgc.house
TLS TLS 1.3, with the CA file as its only roots
Client certificate EC P-256 from the Vault PKI role dns-node-worker-<node>, with the URI SAN spiffe://fzymgc.house/dns-node-worker/<node> and a TTL of 30 days, in /var/lib/dns-worker-pki/bundle.pem
Renewal Vault Agent in dns-node-worker-pki.service, with the AppRole dns-node-worker-<node>
Frames Uncompressed

The worker dials the IP literal and not a name. A Pi resolves names through its own Blocky, and the literal keeps the connection up while that resolver is drained or down. The worker reads the bundle again when the file changes, so a renewed certificate reaches the next handshake with no restart.

A worker appears as a poller of its queue. Read the pollers in the admin tools pod:

Terminal window
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal task-queue describe --task-queue dns-node-sandiego --task-queue-type activity -n dns-control

The output lists the worker of the node with its last poll time. If no poller shows, follow A node worker does not answer in the dns-control runbook.

The gateway reads the node from the URI SAN spiffe://fzymgc.house/dns-node-worker/<node>. A certificate from before this change has a DNS SAN only. Roll out the change in this order:

  1. Apply tf/vault first. The PKI role of each node then allows the URI SAN. If the role does not allow it, the certificate request of Vault Agent fails with URI Subject Alternative Names are not allowed.

  2. Run the Ansible role on one node per window:

    Terminal window
    scripts/dns-playbook.sh --limit <node> --tags dns-node-worker
  3. On that node, remove the bundle and restart Vault Agent. Vault Agent keeps a certificate that is still valid, so without this step the node keeps its old certificate:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.file -a 'path=/var/lib/dns-worker-pki/bundle.pem state=absent'
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl restart dns-node-worker-pki.service'
  4. Make sure that the new certificate has the URI SAN:

    Terminal window
    scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'openssl x509 -noout -ext subjectAltName -in /var/lib/dns-worker-pki/bundle.pem'
  5. Repeat steps 2 to 4 for the other node in another window.

  6. Deploy the gateway change last.

When the gateway change is live, a certificate without the URI SAN fails the handshake. The worker of that node then does not answer, and its actions refuse with node_not_answering. DNS answers do not change.

Path Reader
fzymgc-house/cluster/postgres/users/main-temporal The temporal-db-secret ExternalSecret
fzymgc-house/cluster/hcp-terraform (api_token) The terraform worker, through Vault role temporal-worker-terraform
fzymgc-house/cluster/discord/cluster-notifications-app (bot_token) The terraform worker
fzymgc-house/cluster/temporal/github The terraform worker
fzymgc-house/cluster/ghcr/pull-secret The ghcr-pull-secret ExternalSecret
fzymgc-house/cluster/clickstack (otel_ingest_api_key) The temporal-workers-otlp-headers ExternalSecret

The policy that grants the worker paths is tf/vault/policy-temporal.tf.

Terminal window
# Interactive shell
kubectl exec -it -n temporal deploy/temporal-admintools -- bash
# Single command
kubectl exec -n temporal deploy/temporal-admintools -- tctl <command>

Run the temporal CLI in the admin tools pod.

Terminal window
# List namespaces
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal operator namespace list
# Describe a namespace
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal operator namespace describe -n dns-control
# Create a namespace
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal operator namespace create -n <namespace> --retention 7d
# Change the retention of a namespace
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal operator namespace update -n dns-control --retention 14d

Declare a new namespace under server.config.namespaces in argocd/cluster-app/templates/temporal-server.yaml, as dns-control is. The chart Job then creates it. A retention change needs the update command, because the Job does not change a namespace that exists. Change the value in git in the same pull request.

Terminal window
# List workflows
kubectl exec -n temporal deploy/temporal-admintools -- \
tctl --ns workflows workflow list
# Execute workflow
kubectl exec -n temporal deploy/temporal-admintools -- \
temporal workflow execute \
--namespace workflows \
--task-queue <task-queue> \
--type <WorkflowType> \
--input '<json-input>'
# Get workflow history
kubectl exec -n temporal deploy/temporal-admintools -- \
tctl --ns workflows workflow showid <workflow-id>
# Terminate workflow
kubectl exec -n temporal deploy/temporal-admintools -- \
tctl --ns workflows workflow terminate --workflow_id <id> --reason "manual termination"
Terminal window
kubectl exec -n temporal deploy/temporal-admintools -- \
tctl --ns workflows taskqueue describe --taskqueue default

The output lists the workers that poll the queue.

Terminal window
kubectl exec -n temporal deploy/temporal-admintools -- \
tctl cluster health

The output is temporal.api.workflowservice.v1.WorkflowService: SERVING.

The commands run on the CNPG primary. Read its pod name first:

Terminal window
P=$(kubectl get pod -n postgres -l cnpg.io/cluster=main,cnpg.io/instanceRole=primary -o name | head -1)
# Default store
kubectl exec -n postgres $P -- \
psql -U postgres -d temporal -c "SELECT * FROM schema_version;"
# Visibility store
kubectl exec -n postgres $P -- \
psql -U postgres -d temporal_visibility -c "SELECT * FROM schema_version;"

The visibility store holds the tables executions_visibility, schema_update_history and schema_version:

Terminal window
kubectl exec -n postgres $P -- \
psql -U postgres -d temporal_visibility -c "\dt"

Temporal supports an upgrade of one minor server version at a time, for example 1.30 to 1.31. Each step is one PR to argocd/cluster-app/templates/temporal-server.yaml.

  1. Make sure that Docker Hub has the temporalio/server and temporalio/admin-tools tags. A GitHub release can exist days before its images. Run crane ls docker.io/temporalio/server.
  2. Set server.image.tag and admintools.image.tag to the same patch release of the next minor version.
  3. After ArgoCD syncs, make sure that tctl cluster health reports SERVING and that the worker pods poll their task queues. Then start the next step.

The temporal-schema PreSync Job runs update-schema with the new admin-tools image before the server pods change. If the Job fails, the sync stops and the old pods keep running.

The visibility store reports a missing relation

Section titled “The visibility store reports a missing relation”

The error is ListWorkflowExecutions operation failed. Select failed: pq: relation "executions_visibility" does not exist. The schema Job did not run against the visibility database.

  1. Make sure that both Database objects exist:

    Terminal window
    kubectl get database -n postgres | grep temporal

    The output lists temporal and temporal-visibility.

  2. Delete the schema Job and sync again:

    Terminal window
    kubectl delete job -n temporal -l app.kubernetes.io/component=schema
    argocd app sync temporal-server
  3. Make sure that the visibility tables exist, with the \dt command above.

Workflows stay in Running and make no progress.

  1. Make sure that the worker pod runs:

    Terminal window
    kubectl get pods -n temporal -l app.kubernetes.io/name=temporal-worker
  2. Make sure that the worker polls the task queue:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    tctl --ns workflows taskqueue describe --taskqueue default
  3. Read the worker logs:

    Terminal window
    kubectl logs -n temporal -l app.kubernetes.io/name=temporal-worker --tail=100
Terminal window
kubectl logs -n temporal deploy/temporal-frontend --tail=100
kubectl logs -n temporal deploy/temporal-history --tail=100

The usual causes are the database connection (read the CNPG cluster status), the temporal-db-secret ExternalSecret, and the schema Job.

Terminal window
kubectl get jobs -n temporal
kubectl describe job temporal-schema -n temporal
kubectl logs -n temporal job/temporal-schema -c manage-schema-default-store
kubectl logs -n temporal job/temporal-schema -c manage-schema-visibility-store

A Job is immutable. To run it again:

Terminal window
kubectl delete job temporal-schema -n temporal
argocd app sync temporal-server

A worker version stops only after every workflow pinned to it ends. A workflow that never ends keeps the old version and its pod running. Image Updater then reads the old tag as live and logs 422 Validation Failed on every cycle. The drift-detection schedule skips a run while the last run is open, so no drift check runs either.

  1. Read the state of each old version. Draining with no drainedSince means that a pinned workflow is still open:

    Terminal window
    kubectl get workerdeployment worker-terraform -n temporal \
    -o jsonpath='{range .status.deprecatedVersions[*]}{.buildID} {.status} {.drainedSince}{"\n"}{end}'
  2. List the running workflows. A drift check that is older than its approval timeout of four hours is stuck:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal workflow list -n workflows \
    --address temporal-frontend.temporal.svc.cluster.local:7233 \
    --query 'ExecutionStatus="Running"'
  3. Read the pending activity of the stuck workflow and its last failure:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal workflow describe -n workflows \
    --address temporal-frontend.temporal.svc.cluster.local:7233 \
    -w <workflow id>
  4. Terminate the parent drift run, drift-detection-run-<time>. Its child drift checks end with it:

    Terminal window
    kubectl exec -n temporal deploy/temporal-admintools -- \
    temporal workflow terminate -n workflows \
    --address temporal-frontend.temporal.svc.cluster.local:7233 \
    -w <parent workflow id> --reason "<reason>"

The old version reports Drained a few minutes later and stops one hour after that.