Temporal.io Migration Design

Date: 2026-01-01 Status: Draft Author: Sean + Claude

Overview

Migration from Windmill to Temporal.io for workflow orchestration, with a focus on:

1. AI Agent Platform — Durable execution for LLM-based agents
2. General Workflow Engine — Scheduled tasks, approvals, notifications
3. Code/GitOps/IaC friendly — Code-first over UI-first

Architecture

High-Level Overview

graph TB
    subgraph "selfhosted-cluster repo"
        TF[tf/vault/policy-temporal.tf]
        ARGO_SERVER[argocd/.../temporal-server.yaml]
        ARGO_CTRL[argocd/.../temporal-worker-controller.yaml]
        ARGO_WORKERS[argocd/.../temporal-workers.yaml]
    end

    subgraph "temporal-workers repo"
        COMMON[packages/common]
        W_CORE[packages/worker-core]
        W_TF[packages/worker-terraform]
        W_LLM[packages/worker-llm]
        WEBHOOK[packages/webhook-receiver]
        DOCKER[docker/Dockerfile.*]
        K8S_CRDS[k8s/crds/*.yaml]
    end

    subgraph "K8s Namespace: temporal"
        SERVER[Temporal Server<br/>frontend, history, matching, web]
        CTRL[Worker Controller]
        DB[(PostgreSQL<br/>CNPG)]

        subgraph "Managed by Controller"
            WD_CORE[worker-core-v1.2.3]
            WD_TF[worker-terraform-v2.0.1]
            WD_LLM[worker-llm-v0.1.0]
        end

        WH_SVC[Webhook Receiver]
    end

    subgraph "External"
        VAULT[Vault]
        DISCORD[Discord]
        GITHUB[GitHub]
        GRAFANA[Grafana Stack]
    end

    TF --> VAULT
    ARGO_SERVER --> SERVER
    ARGO_CTRL --> CTRL
    ARGO_WORKERS --> K8S_CRDS
    K8S_CRDS --> CTRL
    CTRL --> WD_CORE & WD_TF & WD_LLM

    SERVER --> DB
    WD_CORE & WD_TF & WD_LLM --> SERVER
    WD_CORE & WD_TF & WD_LLM --> VAULT
    WD_CORE & WD_TF & WD_LLM --> GRAFANA

    WH_SVC --> SERVER
    DISCORD --> WH_SVC
    GITHUB --> WH_SVC

Deployment Flow

sequenceDiagram
    participant DEV as Developer
    participant GH as GitHub Actions
    participant GHCR as GHCR
    participant ARGO as ArgoCD
    participant CTRL as Worker Controller
    participant K8S as K8s Deployments

    DEV->>GH: Push to temporal-workers
    GH->>GH: Build worker images (matrix)
    GH->>GHCR: Push images with version tags

    ARGO->>ARGO: Detect CRD changes
    ARGO->>K8S: Apply TemporalWorkerDeployment CRDs

    CTRL->>GHCR: Detect new image version
    CTRL->>K8S: Create versioned Deployment
    CTRL->>CTRL: Progressive rollout (10% → 50% → 100%)
    CTRL->>K8S: Drain and delete old versions

Repository Structure

selfhosted-cluster (Infrastructure)

selfhosted-cluster/
├── tf/vault/
│   └── policy-temporal.tf              # Vault policies only
│
├── argocd/cluster-app/templates/
│   ├── temporal-server.yaml            # Helm chart for Temporal server
│   ├── temporal-worker-controller.yaml # Helm chart for controller
│   └── temporal-workers.yaml           # Points to worker CRDs
│
└── argocd/app-configs/
    ├── temporal-server/
    │   ├── kustomization.yaml
    │   ├── namespace.yaml
    │   ├── database.yaml               # CNPG Database resource
    │   ├── secrets.yaml                # ExternalSecrets
    │   ├── certificate.yaml
    │   └── ingress.yaml
    │
    └── temporal-workers/
        ├── kustomization.yaml
        └── secrets.yaml                # ExternalSecrets for workers

temporal-workers (Application)

fzymgc-house/temporal-workers/
├── pyproject.toml                      # uv workspace root
├── uv.lock
│
├── packages/
│   ├── common/                         # Shared utilities
│   │   ├── pyproject.toml
│   │   └── src/temporal_common/
│   │       ├── settings.py             # Pydantic settings
│   │       ├── client.py               # Temporal client factory
│   │       ├── observability.py        # OTel setup
│   │       └── vault.py                # hvac client
│   │
│   ├── worker-core/                    # Discord, GitHub, S3
│   │   ├── pyproject.toml
│   │   └── src/worker_core/
│   │       ├── activities/
│   │       ├── workflows/
│   │       └── main.py
│   │
│   ├── worker-terraform/               # Terraform operations
│   │   ├── pyproject.toml
│   │   └── src/worker_terraform/
│   │       ├── activities/
│   │       ├── workflows/
│   │       └── main.py
│   │
│   ├── worker-llm/                     # LLM/Agent (future)
│   │   ├── pyproject.toml
│   │   └── src/worker_llm/
│   │
│   └── webhook-receiver/               # FastAPI webhook service
│       ├── pyproject.toml
│       └── src/webhook_receiver/
│           ├── main.py
│           └── routes/
│
├── docker/
│   ├── Dockerfile.core
│   ├── Dockerfile.terraform
│   ├── Dockerfile.llm
│   └── Dockerfile.webhook
│
└── k8s/
    ├── base/
    │   └── serviceaccount.yaml
    ├── crds/
    │   ├── worker-core.yaml
    │   ├── worker-terraform.yaml
    │   └── worker-llm.yaml
    └── webhook/
        ├── deployment.yaml
        ├── service.yaml
        └── ingress.yaml

First Workflow: Terraform Deploy

Workflow Diagram

stateDiagram-v2
    [*] --> TerraformPlan

    TerraformPlan --> NotifyPlanReady: Plan succeeded
    TerraformPlan --> NotifyPlanFailed: Plan failed
    NotifyPlanFailed --> [*]

    NotifyPlanReady --> WaitForApproval

    WaitForApproval --> Approved: 👍 reaction
    WaitForApproval --> Rejected: 👎 reaction
    WaitForApproval --> TimedOut: 24h timeout

    Rejected --> NotifyRejected
    NotifyRejected --> [*]

    TimedOut --> NotifyTimeout
    NotifyTimeout --> [*]

    Approved --> TerraformApply
    TerraformApply --> NotifyComplete: Apply succeeded
    TerraformApply --> NotifyApplyFailed: Apply failed

    NotifyComplete --> [*]
    NotifyApplyFailed --> [*]

Discord Approval Flow

sequenceDiagram
    participant WF as Workflow
    participant ACT as Activity
    participant DISCORD as Discord
    participant USER as User
    participant WH as Webhook Receiver

    WF->>ACT: terraform_plan()
    ACT-->>WF: plan_output

    WF->>ACT: notify_discord(plan_summary)
    ACT->>DISCORD: Send embed with 👍/👎 prompt

    WF->>WF: workflow.wait_condition(signal or timeout)

    USER->>DISCORD: Reacts with 👍
    DISCORD->>WH: Interaction webhook
    WH->>WF: Signal: approval

    WF->>ACT: terraform_apply()
    ACT-->>WF: apply_result

    WF->>ACT: notify_discord(success)

Activities

Activity	Description	Retry Policy
terraform_plan	Execute terraform plan, capture output	3 attempts, exponential backoff
terraform_apply	Execute terraform apply	1 attempt (no retry)
notify_discord	Send message/embed to Discord channel	5 attempts

Signals

Signal	Source	Payload
approval	Webhook receiver	{"user": "...", "timestamp": "..."}
rejection	Webhook receiver	{"user": "...", "reason": "..."}

Observability

Telemetry Flow

graph LR
    subgraph "Worker Process"
        SDK[OTel SDK]
        TRACES[Traces]
        METRICS[Metrics]
        LOGS[Logs]
    end

    subgraph "Collection"
        COLLECTOR[OTel Collector]
    end

    subgraph "Grafana Stack"
        TEMPO[Tempo]
        PROM[Prometheus]
        LOKI[Loki]
        GRAFANA[Grafana]
    end

    SDK --> TRACES & METRICS & LOGS
    TRACES & METRICS & LOGS --> COLLECTOR

    COLLECTOR -->|traces| TEMPO
    COLLECTOR -->|metrics| PROM
    COLLECTOR -->|logs| LOKI

    TEMPO & PROM & LOKI --> GRAFANA

Key Metrics

Metric	Purpose
temporal_workflow_started_total	Workflow start rate
temporal_workflow_completed_total	Success/failure rate
temporal_activity_duration_seconds	Activity latency histogram
llm_request_duration_seconds	LLM API call latency (future)
llm_tokens_total	Token consumption (future)

Secrets Management

Hybrid Approach

graph TB
    subgraph "Static Secrets (startup)"
        VAULT_STATIC[Vault]
        ESO[External Secrets Operator]
        K8S_SECRET[K8s Secret]
        ENV[Environment Variables]
    end

    subgraph "Dynamic Secrets (runtime)"
        VAULT_DYN[Vault]
        HVAC[hvac client]
        ACTIVITY[Activity code]
    end

    VAULT_STATIC --> ESO --> K8S_SECRET --> ENV
    VAULT_DYN --> HVAC --> ACTIVITY

Type	Use Case	Mechanism
Static	Discord token, DB creds, OTel endpoint	ExternalSecrets → env vars
Dynamic	Per-workflow secrets, rotated credentials	Vault SDK (hvac) in activity

Implementation Phases

Phase 1: Foundation

gantt
    title Phase 1: Foundation
    dateFormat  YYYY-MM-DD
    section Repos
    Create temporal-workers repo     :p1a, 2026-01-02, 1d
    Initialize uv workspace          :p1b, after p1a, 1d
    Add common package               :p1c, after p1b, 1d

    section Infrastructure
    CNPG Database manifest           :p1d, 2026-01-02, 1d
    Temporal server ArgoCD app       :p1e, after p1d, 1d
    Worker Controller ArgoCD app     :p1f, after p1e, 1d
    Vault policies                   :p1g, after p1f, 1d

Phase 2: First Worker

• worker-core package (Discord activity)
• TemporalWorkerDeployment CRD
• Validate controller picks up image
• Hello-world workflow via CLI

Phase 3: Terraform Workflow

• worker-terraform package
• TerraformDeploy workflow
• Discord approval flow
• Webhook receiver deployment
• Schedule creation

Phase 4: Production Hardening

• Grafana dashboards
• Alerting rules
• Documentation
• Windmill deprecation

Phase 5: AI Agent Foundation (Future)

• worker-llm package
• Anthropic activity
• Tool execution activity
• Agent orchestration workflow

Decisions Log

Decision	Choice	Rationale
Temporal namespace	workflows	Descriptive, K8s ns remains temporal
Worker repo	Separate repo	Different release cadence, blast radius isolation
Dev workflow	Local Temporal	Fast iteration, single prod namespace
Database	CNPG main cluster	Consistent with existing pattern
Secrets	Hybrid (ESO + hvac)	Static at startup, dynamic when needed
Observability	Full OTel	Critical for AI agent debugging
First approval	Discord reactions	Existing tokens, immediate value
Deployment	Worker Controller	Rainbow deploys, graceful drain

Open Questions

1. Worker Controller version — Which Helm chart version to start with?
2. OTel Collector — Use existing or deploy dedicated for Temporal?
3. Discord bot setup — New application or reuse existing?

References

• Temporal Python SDK
• Temporal Worker Controller
• Worker Versioning
• Temporal Helm Charts