Temporal Operations
Temporal runs the cluster’s workflows. This page states what exists, where each part is defined,
and the procedures that touch it. Commands assume --context fzymgc-house.
What exists
Section titled “What exists”| Property | Value |
|---|---|
| Web UI | https://temporal.fzymgc.house, behind the shared oauth2-proxy forward-auth middleware oauth-auth-temporal, which admits the Keycloak group temporal-users |
| Frontend | temporal-frontend.temporal.svc.cluster.local:7233 |
| Node gateway | 192.168.250.146:7233 and [fddb:f665:73f7:250::146]:7233, the mTLS entry for the resolver Pis. See The node gateway |
| Namespace | temporal |
| Temporal namespaces | workflows for the cluster workers, dns-control for dns-control and the node workers |
| Server chart | temporal from https://go.temporal.io/helm-charts, pinned in argocd/cluster-app/templates/temporal-server.yaml with the server, web and admin-tools image tags |
| Manifests | argocd/app-configs/temporal-server/ (namespace, ingress, certificate, database secret, CA chain) |
| Workers repository | fzymgc-house/temporal-workers |
The group temporal-users and the oauth2-proxy client come from tf/keycloak/oauth2_proxy.tf.
The design is in
the forward-auth ADR.
Databases
Section titled “Databases”Temporal uses two databases on the CNPG cluster main in the postgres namespace. Each is a
Database object under argocd/app-configs/cnpg/, and each has its own schema_version table,
so the two schemas cannot collide.
| Database | Object | Purpose |
|---|---|---|
temporal |
db-temporal.yaml |
Workflow state and history |
temporal_visibility |
db-temporal-visibility.yaml |
Workflow search and listing |
Both datastores connect to main-rw.postgres.svc.cluster.local:5432 as user temporal over TLS,
with the CA from the vault-ca-chain Secret. The password is temporal-db-secret, read by an
ExternalSecret from fzymgc-house/cluster/postgres/users/main-temporal. Both datastores set
manageSchema: true, so the chart’s temporal-schema Job runs as an ArgoCD PreSync hook and
updates the schema before the server pods change.
Server components
Section titled “Server components”Each server role runs one replica. The chart values set numHistoryShards: 128 and turn on the
frontend.WorkerHeartbeatsEnabled and frontend.ListWorkersEnabled dynamic configuration.
| Component | Purpose |
|---|---|
temporal-frontend |
API gateway and client connections |
temporal-history |
Workflow state |
temporal-matching |
Task queue routing |
temporal-worker |
Internal workflows |
temporal-web |
Web UI |
temporal-admintools |
The tctl and temporal CLIs |
Workers
Section titled “Workers”| Application | Chart | What it does |
|---|---|---|
temporal-worker-controller and temporal-worker-controller-crds |
docker.io/temporalio charts, the same version in both templates |
The controller reconciles WorkerDeployment and Connection objects and rolls workers progressively |
temporal-workers |
ghcr.io/fzymgc-house/charts/temporal-workers, pinned in argocd/cluster-app/templates/temporal-workers.yaml. The values are in argocd/app-configs/temporal-workers/values.yaml |
The terraform worker: it plans the main-cluster-vault and main-cluster-core-services workspaces, posts to Discord, and applies tf/hcp-terraform under a guard that needs approval on every plan |
temporal-webhook-receiver |
ghcr.io/fzymgc-house/charts/temporal-webhook-receiver, the same version as the workers chart |
Receives webhooks and starts workflows on the frontend |
The two controller charts must stay on one version, because the CRDs and the controller change
together. A workers chart that changes CRD kinds needs the matching controller first. A new chart
deploys only after a pull request here moves targetRevision.
A new worker image deploys without a person. Within five minutes of a release in the workers
repository, Argo CD Image Updater opens a pull request. It moves
workers.terraform.image.tag in argocd/app-configs/temporal-workers/values.yaml, and it merges
after its checks pass. The image is ghcr.io/fzymgc-house/temporal-workers/terraform. The chart
reads workers.<worker>.image from chart 3.3.0 on. A new worker needs its own
workers.<worker>.image keys in the values file and its own image in
argocd/app-configs/argocd-image-updater/temporal-workers.yaml.
After a worker release, the old worker version keeps running until each workflow pinned to it
ends. A pinned workflow stays on the worker version that started it. The controller then waits one
hour (scaledownDelay: 1h) and stops the old version. Both images are live for that time. A drift
check that waits for approval holds its version for up to four hours. If an old version runs
longer, see An old worker version does not stop.
The workers run as the temporal-worker ServiceAccount with the ghcr-pull-secret
(argocd/app-configs/temporal-workers/). They send OTLP to the ClickStack collector with the
ingest key from fzymgc-house/cluster/clickstack.
Approval buttons
Section titled “Approval buttons”A drift check that finds drift posts Approve and Reject buttons to Discord. Each button holds an
approval ID, a ULID that the check also sets as the search attribute ApprovalId. The webhook
receiver finds the waiting workflow by that attribute and signals it. A button ID is at most 54
characters, and Discord allows 100.
The search attribute must exist in the namespace workflows before a worker sets it. A workflow
that sets an unknown search attribute fails its workflow task on every retry. If you rebuild the
namespace, create the attribute again before the workers start:
kubectl exec -n temporal deploy/temporal-admintools -- \ temporal operator search-attribute create -n workflows \ --address temporal-frontend.temporal.svc.cluster.local:7233 \ --name ApprovalId --type KeywordImage Updater moves the worker image. The receiver moves only when a pull request here moves the
targetRevision of temporal-webhook-receiver, so a change to the button format needs both. Until
the receiver moves, it answers “Invalid request” to a new button. The workflow keeps waiting, so a
click after the move works.
The namespace dns-control
Section titled “The namespace dns-control”dns-control and the node workers on the resolver Pis use the Temporal namespace dns-control.
The chart creates it. The values in argocd/cluster-app/templates/temporal-server.yaml list it
under server.config.namespaces with a retention of 7 days. The retention is the time that
Temporal keeps a closed workflow.
The chart Job temporal-namespace runs after each install and upgrade. Its init container
create-dns-control-namespace runs this command, which is safe to run twice:
temporal operator namespace describe -n dns-control || temporal operator namespace create -n dns-control --retention 7dThe Job only creates the namespace. A change of the retention in the values does not change the
namespace that exists. To change the retention, change the values and run
temporal operator namespace update in the same window. See
Manage namespaces.
| Task queue | Poller | What runs on it |
|---|---|---|
dns-control |
The control worker in the dns-control pod | The workflows ops.action and ops.status and their control activities |
dns-node-sandiego |
The node worker on sandiego |
The six node activities of sandiego |
dns-node-portland |
The node worker on portland |
The six node activities of portland |
dns-control reaches the frontend in the cluster, at temporal-frontend.temporal.svc.cluster.local:7233.
The Pis reach it only through the node gateway.
The node gateway
Section titled “The node gateway”The resolver Pis are outside the cluster. They reach Temporal through a gateway on a Lab VIP. The gateway ends the mTLS of the Pis, asks a policy engine about each call, and forwards an allowed call to the frontend. The in-cluster tenants keep the plaintext frontend and do not change.
| Property | Value |
|---|---|
| Address | 192.168.250.146:7233 and [fddb:f665:73f7:250::146]:7233, routed VIPs announced over BGP |
| Service | temporal/temporal-node-gateway-vip, type LoadBalancer, externalTrafficPolicy: Local |
| Workload | Deployment/temporal-node-gateway, two replicas, each with the containers envoy and opa |
| Manifests | argocd/app-configs/temporal-server/node-gateway/, synced by the application temporal-server |
| Server certificate | temporal-node-gateway-tls for temporal-nodes.fzymgc.house, from vault-issuer, RSA 2048 |
| Client trust | The CA chain vault-ca-chain, with a URI SAN under the prefix spiffe://fzymgc.house/dns-node-worker/ |
| Ingress | The CiliumNetworkPolicy admits the resolver Pis only: 192.168.40.6/32 (sandiego) and 192.168.40.7/32 (portland), their Core addresses. A Pi reaches the routed VIP through its default route on Core |
Envoy listens on port 7233. The listener requires a client certificate and TLS 1.3. The trust
bundle is the full ICA1 chain, so every certificate that ICA1 issues completes the chain check.
The SAN match then admits only a certificate with a URI SAN under
spiffe://fzymgc.house/dns-node-worker/. Only the PKI roles of the node workers in Vault allow
that URI SAN. The cert-manager role issues any DNS name, so a match on a DNS SAN admits a
Certificate from any namespace.
Envoy forwards an allowed call over plaintext HTTP/2 to the frontend. The route timeout is 0 seconds, because a poll holds its stream for about a minute. Every hour Envoy closes each connection with GOAWAY. The Pi then connects again, and the new handshake loads a renewed client certificate.
The server certificate is RSA 2048 and not ECDSA. The ClusterIssuer vault-issuer signs through
the Vault role fzymgc-house/v1/ica1/v1/roles/cert-manager, which sets key_type rsa. Vault
refuses a request whose key type differs from the role. Reloader rolls the pods when the
certificate renews.
The policy
Section titled “The policy”Envoy calls OPA through ext_authz for each gRPC call, with the request body. OPA is the Open
Policy Agent, a policy engine. OPA decodes the protobuf body with the descriptor set
temporal-api.binpb, so the policy reads the namespace from the body. If OPA does not answer,
Envoy denies the call. DNS answers never depend on the gateway.
The policy is in policy/policy.rego, and policy/data.json binds each node to its namespace.
Envoy takes the principal from the first URI SAN of the client certificate. The policy allows a
WorkflowService call when all of these conditions are true:
- The principal is
spiffe://fzymgc.house/dns-node-worker/<node>, anddata.jsonnames the node. - The method is one of the calls of an activity worker:
DescribeNamespace,PollActivityTaskQueue,RespondActivityTaskCompleted,RespondActivityTaskFailed,RespondActivityTaskCanceled,RecordActivityTaskHeartbeat,RecordWorkerHeartbeatorShutdownWorker. - Envoy passed the full body, not a part of it.
- The namespace in the body equals the namespace that the node is bound to,
dns-control. - If the call carries the header
temporal-namespace, the header equals the body. PollActivityTaskQueueandShutdownWorkername the task queue of the node,dns-node-<node>.ShutdownWorkernames no sticky queue, because the node runs no workflow worker.
The health check and GetSystemInfo pass for a known node. The policy denies every other method,
the *ById variants of the activity calls, OperatorService and AdminService. A stolen Pi
certificate therefore reaches no other namespace and no other task queue.
The policy has two limits:
- The Respond calls and
RecordActivityTaskHeartbeatpass on the namespace alone. The policy does not read their task token. - A later change that turns on
frontend.WorkerCommandsEnabledmakes the SDK pollPollNexusTaskQueue. The gateway denies that call, and the worker logs a warning every second. Add the call to the policy in the same change.
Run the policy tests before a change:
mise //argocd/app-configs/temporal-server/node-gateway:policy:testThe pr-gates member temporal-gateway-policy runs the same tests when a pull request changes a file
under argocd/app-configs/temporal-server/node-gateway/policy/. The nightly pr-gates run also runs
them. After a change of the Temporal API version, rebuild the descriptor set with
mise //argocd/app-configs/temporal-server/node-gateway:descriptor:build.
OPA writes each decision to the log of its container, with the principal of the caller. The rule
in policy/log_mask.rego drops the request body from each entry and keeps only its namespace
and taskQueue. A body can carry activity results and failures of up to 4 MiB. Read the decision
log and the Envoy log:
kubectl logs -n temporal deploy/temporal-node-gateway -c opa --tail=100kubectl logs -n temporal deploy/temporal-node-gateway -c envoy --tail=100The node workers send uncompressed gRPC frames. OPA cannot decode a compressed body, so it denies every call from a client that compresses its frames.
The node workers
Section titled “The node workers”Each resolver Pi runs the node worker dns-node-worker. The Ansible role dns-node-worker
installs it. The worker runs activities only and no workflow. It polls the task queue
dns-node-<node> in the namespace dns-control: dns-node-sandiego and dns-node-portland.
| Property | Value |
|---|---|
| Dial | 192.168.250.146:7233 by IP literal, with the server name temporal-nodes.fzymgc.house |
| TLS | TLS 1.3, with the CA file as its only roots |
| Client certificate | EC P-256 from the Vault PKI role dns-node-worker-<node>, with the URI SAN spiffe://fzymgc.house/dns-node-worker/<node> and a TTL of 30 days, in /var/lib/dns-worker-pki/bundle.pem |
| Renewal | Vault Agent in dns-node-worker-pki.service, with the AppRole dns-node-worker-<node> |
| Frames | Uncompressed |
The worker dials the IP literal and not a name. A Pi resolves names through its own Blocky, and the literal keeps the connection up while that resolver is drained or down. The worker reads the bundle again when the file changes, so a renewed certificate reaches the next handshake with no restart.
A worker appears as a poller of its queue. Read the pollers in the admin tools pod:
kubectl exec -n temporal deploy/temporal-admintools -- \ temporal task-queue describe --task-queue dns-node-sandiego --task-queue-type activity -n dns-controlThe output lists the worker of the node with its last poll time. If no poller shows, follow A node worker does not answer in the dns-control runbook.
Roll out the node identity
Section titled “Roll out the node identity”The gateway reads the node from the URI SAN spiffe://fzymgc.house/dns-node-worker/<node>. A
certificate from before this change has a DNS SAN only. Roll out the change in this order:
-
Apply
tf/vaultfirst. The PKI role of each node then allows the URI SAN. If the role does not allow it, the certificate request of Vault Agent fails withURI Subject Alternative Names are not allowed. -
Run the Ansible role on one node per window:
Terminal window scripts/dns-playbook.sh --limit <node> --tags dns-node-worker -
On that node, remove the bundle and restart Vault Agent. Vault Agent keeps a certificate that is still valid, so without this step the node keeps its old certificate:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.file -a 'path=/var/lib/dns-worker-pki/bundle.pem state=absent'scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'systemctl restart dns-node-worker-pki.service' -
Make sure that the new certificate has the URI SAN:
Terminal window scripts/dns-adhoc.sh <node> -b -m ansible.builtin.command -a 'openssl x509 -noout -ext subjectAltName -in /var/lib/dns-worker-pki/bundle.pem' -
Repeat steps 2 to 4 for the other node in another window.
-
Deploy the gateway change last.
When the gateway change is live, a certificate without the URI SAN fails the handshake. The worker
of that node then does not answer, and its actions refuse with node_not_answering. DNS answers do
not change.
Vault paths
Section titled “Vault paths”| Path | Reader |
|---|---|
fzymgc-house/cluster/postgres/users/main-temporal |
The temporal-db-secret ExternalSecret |
fzymgc-house/cluster/hcp-terraform (api_token) |
The terraform worker, through Vault role temporal-worker-terraform |
fzymgc-house/cluster/discord/cluster-notifications-app (bot_token) |
The terraform worker |
fzymgc-house/cluster/temporal/github |
The terraform worker |
fzymgc-house/cluster/ghcr/pull-secret |
The ghcr-pull-secret ExternalSecret |
fzymgc-house/cluster/clickstack (otel_ingest_api_key) |
The temporal-workers-otlp-headers ExternalSecret |
The policy that grants the worker paths is tf/vault/policy-temporal.tf.
Procedures
Section titled “Procedures”Use the admin tools
Section titled “Use the admin tools”# Interactive shellkubectl exec -it -n temporal deploy/temporal-admintools -- bash
# Single commandkubectl exec -n temporal deploy/temporal-admintools -- tctl <command>Manage namespaces
Section titled “Manage namespaces”Run the temporal CLI in the admin tools pod.
# List namespaceskubectl exec -n temporal deploy/temporal-admintools -- \ temporal operator namespace list
# Describe a namespacekubectl exec -n temporal deploy/temporal-admintools -- \ temporal operator namespace describe -n dns-control
# Create a namespacekubectl exec -n temporal deploy/temporal-admintools -- \ temporal operator namespace create -n <namespace> --retention 7d
# Change the retention of a namespacekubectl exec -n temporal deploy/temporal-admintools -- \ temporal operator namespace update -n dns-control --retention 14dDeclare a new namespace under server.config.namespaces in
argocd/cluster-app/templates/temporal-server.yaml, as dns-control is. The chart Job then
creates it. A retention change needs the update command, because the Job does not change a
namespace that exists. Change the value in git in the same pull request.
Workflow Operations
Section titled “Workflow Operations”# List workflowskubectl exec -n temporal deploy/temporal-admintools -- \ tctl --ns workflows workflow list
# Execute workflowkubectl exec -n temporal deploy/temporal-admintools -- \ temporal workflow execute \ --namespace workflows \ --task-queue <task-queue> \ --type <WorkflowType> \ --input '<json-input>'
# Get workflow historykubectl exec -n temporal deploy/temporal-admintools -- \ tctl --ns workflows workflow showid <workflow-id>
# Terminate workflowkubectl exec -n temporal deploy/temporal-admintools -- \ tctl --ns workflows workflow terminate --workflow_id <id> --reason "manual termination"Read a task queue
Section titled “Read a task queue”kubectl exec -n temporal deploy/temporal-admintools -- \ tctl --ns workflows taskqueue describe --taskqueue defaultThe output lists the workers that poll the queue.
Check Cluster Health
Section titled “Check Cluster Health”kubectl exec -n temporal deploy/temporal-admintools -- \ tctl cluster healthThe output is temporal.api.workflowservice.v1.WorkflowService: SERVING.
Read the schema versions
Section titled “Read the schema versions”The commands run on the CNPG primary. Read its pod name first:
P=$(kubectl get pod -n postgres -l cnpg.io/cluster=main,cnpg.io/instanceRole=primary -o name | head -1)
# Default storekubectl exec -n postgres $P -- \ psql -U postgres -d temporal -c "SELECT * FROM schema_version;"
# Visibility storekubectl exec -n postgres $P -- \ psql -U postgres -d temporal_visibility -c "SELECT * FROM schema_version;"The visibility store holds the tables executions_visibility, schema_update_history and
schema_version:
kubectl exec -n postgres $P -- \ psql -U postgres -d temporal_visibility -c "\dt"Upgrade the server
Section titled “Upgrade the server”Temporal supports an upgrade of one minor server version at a time, for example 1.30 to 1.31.
Each step is one PR to argocd/cluster-app/templates/temporal-server.yaml.
- Make sure that Docker Hub has the
temporalio/serverandtemporalio/admin-toolstags. A GitHub release can exist days before its images. Runcrane ls docker.io/temporalio/server. - Set
server.image.tagandadmintools.image.tagto the same patch release of the next minor version. - After ArgoCD syncs, make sure that
tctl cluster healthreportsSERVINGand that the worker pods poll their task queues. Then start the next step.
The temporal-schema PreSync Job runs update-schema with the new admin-tools image before the
server pods change. If the Job fails, the sync stops and the old pods keep running.
Troubleshooting
Section titled “Troubleshooting”The visibility store reports a missing relation
Section titled “The visibility store reports a missing relation”The error is
ListWorkflowExecutions operation failed. Select failed: pq: relation "executions_visibility" does not exist.
The schema Job did not run against the visibility database.
-
Make sure that both
Databaseobjects exist:Terminal window kubectl get database -n postgres | grep temporalThe output lists
temporalandtemporal-visibility. -
Delete the schema Job and sync again:
Terminal window kubectl delete job -n temporal -l app.kubernetes.io/component=schemaargocd app sync temporal-server -
Make sure that the visibility tables exist, with the
\dtcommand above.
A worker does not process tasks
Section titled “A worker does not process tasks”Workflows stay in Running and make no progress.
-
Make sure that the worker pod runs:
Terminal window kubectl get pods -n temporal -l app.kubernetes.io/name=temporal-worker -
Make sure that the worker polls the task queue:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \tctl --ns workflows taskqueue describe --taskqueue default -
Read the worker logs:
Terminal window kubectl logs -n temporal -l app.kubernetes.io/name=temporal-worker --tail=100
A server pod crash-loops
Section titled “A server pod crash-loops”kubectl logs -n temporal deploy/temporal-frontend --tail=100kubectl logs -n temporal deploy/temporal-history --tail=100The usual causes are the database connection (read the CNPG cluster status), the
temporal-db-secret ExternalSecret, and the schema Job.
The schema Job is stuck
Section titled “The schema Job is stuck”kubectl get jobs -n temporalkubectl describe job temporal-schema -n temporalkubectl logs -n temporal job/temporal-schema -c manage-schema-default-storekubectl logs -n temporal job/temporal-schema -c manage-schema-visibility-storeA Job is immutable. To run it again:
kubectl delete job temporal-schema -n temporalargocd app sync temporal-serverAn old worker version does not stop
Section titled “An old worker version does not stop”A worker version stops only after every workflow pinned to it ends. A workflow that never ends keeps
the old version and its pod running. Image Updater then reads the old tag as live and logs
422 Validation Failed on every cycle. The drift-detection schedule skips a run while the last run
is open, so no drift check runs either.
-
Read the state of each old version.
Drainingwith nodrainedSincemeans that a pinned workflow is still open:Terminal window kubectl get workerdeployment worker-terraform -n temporal \-o jsonpath='{range .status.deprecatedVersions[*]}{.buildID} {.status} {.drainedSince}{"\n"}{end}' -
List the running workflows. A drift check that is older than its approval timeout of four hours is stuck:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal workflow list -n workflows \--address temporal-frontend.temporal.svc.cluster.local:7233 \--query 'ExecutionStatus="Running"' -
Read the pending activity of the stuck workflow and its last failure:
Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal workflow describe -n workflows \--address temporal-frontend.temporal.svc.cluster.local:7233 \-w <workflow id> -
Terminate the parent drift run,
drift-detection-run-<time>. Its child drift checks end with it:Terminal window kubectl exec -n temporal deploy/temporal-admintools -- \temporal workflow terminate -n workflows \--address temporal-frontend.temporal.svc.cluster.local:7233 \-w <parent workflow id> --reason "<reason>"
The old version reports Drained a few minutes later and stops one hour after that.
Related
Section titled “Related”- Services Catalog - Temporal
- dns-control: the operations actions that run on the namespace
dns-control, and the recovery of a stuck action and of a silent node worker - temporal-workers Repository
- Temporal Documentation