Windmill Staging/Production Workflow Migration Plan¶

Executive Summary¶

Migrate from a single-workspace manual deployment model to Windmill's recommended two-workspace GitOps pattern with automated staging → production promotion via pull requests.

Timeline: Estimated 2-3 hours for initial setup and testing.

Risk Level: Low - Non-destructive migration with rollback capability.

Prerequisites: - The staging branch must exist before implementing GitHub Actions workflows - Windmill Enterprise Edition must be deployed (already complete)

Current State Analysis¶

Existing Configuration¶

Single Workspace: terraform-gitops
Deployment Method: Manual CLI sync via wmill sync push
Git Integration: Manual commits, no automated sync
Branch Strategy: Working directly on feat/windmill-migration and main
Configuration: windmill/wmill.yaml with basic sync settings
Image: Windmill EE v1.589.1 (EE features available)

Limitations¶

No separation between development/testing and production
Manual promotion process prone to human error
No automated testing before production deployment
Direct production changes without review process
Violates GitOps principles for critical infrastructure

Target State Design¶

Two-Workspace Architecture¶

Workspace 1: terraform-gitops-staging (Development/Testing) - Purpose: Development and testing of Terraform automation workflows - Git Sync Mode: Sync Mode (auto-commits on deploy) - Git Branch: staging branch - Access: Developers have full access - Promotion: Creates branches targeting main via Promotion Mode

Workspace 2: terraform-gitops-prod (Production) - Purpose: Production Terraform deployments - Git Sync Mode: Pull Mode (receives deployments via GitHub Actions) - Git Branch: main branch - Access: Read-only via UI, deployed only through PR merges - Protection: GitHub branch protection on main

Git Branch Strategy¶

staging branch (auto-updated by Windmill)
    ↓
wm_deploy/staging/<item-path> (auto-created per-item)
    ↓
Pull Request (auto-created by GitHub Actions)
    ↓
main branch (after PR approval)
    ↓
Auto-deploy to prod workspace (GitHub Actions)

GitHub Actions Workflows¶

Workflow 1: windmill-open-pr.yaml - Triggers on push to staging branch - Creates PR from staging → main for review - Labels: windmill, autogenerated

Workflow 2: windmill-deploy-prod.yaml - Triggers on PR merge to main - Runs wmill sync push to deploy to prod workspace - Uses Windmill token stored in GitHub secrets

Repository Structure¶

windmill/
├── wmill.yaml                          # Workspace configuration
├── f/                                  # Folder: terraform
│   ├── terraform/
│   │   ├── deploy_vault.flow/
│   │   ├── terraform_init.script/
│   │   ├── terraform_plan.script/
│   │   └── ...
│   ├── resources/                      # Shared resources
│   └── bots/                           # Discord bot configs
├── u/                                  # User scripts (if any)
└── .github/
    └── workflows/
        ├── windmill-open-pr.yaml
        └── windmill-deploy-prod.yaml

Migration Steps¶

Phase 1: Preparation (30 minutes)¶

1.1 Create Staging Branch¶

git checkout main
git pull
git checkout -b staging
git push -u origin staging

1.2 Create Staging Workspace in Windmill¶

Navigate to Windmill UI: https://windmill.fzymgc.house
Create new workspace: terraform-gitops-staging
Configure workspace settings:
Default runtime: Python 3
Default TS: Bun
Enable all features

1.3 Update wmill.yaml for Staging¶

Update windmill/wmill.yaml:

workspace: terraform-gitops-staging  # Changed from terraform-gitops
version: v2
defaultRuntime: python3
defaultTs: bun

includes:
  - "**"

excludes: []

# Sync everything for staging workspace
skipVariables: false
skipResources: false
skipResourceTypes: false
skipSecrets: false
skipScripts: false
skipFlows: false
skipApps: false
skipFolders: false

includeSchedules: true
includeTriggers: true
includeUsers: true
includeGroups: true
includeSettings: true
includeKey: false

codebases:
  - relativePath: "."
    defaultRuntime: python3
    defaultTs: bun

1.4 Initial Deployment to Staging¶

cd windmill
npx wmill workspace add terraform-gitops-staging terraform-gitops-staging https://windmill.fzymgc.house <token>
npx wmill sync push

Phase 2: Configure Git Sync (45 minutes)¶

2.1 Enable Git Sync on Staging Workspace¶

In Windmill UI (Workspace Settings → Git Sync):

Repository: fzymgc-house/selfhosted-cluster
Branch: staging
Path: windmill
Mode: Sync + Promotion
Git Username: GitHub username
Git Email: GitHub email
Git Token: Create fine-grained PAT with:
Repository: selfhosted-cluster
Permissions: contents: write, pull_requests: write
Promotion Settings:
Enable promotion mode
Target branch: main
Branch prefix: wm_deploy/staging/

2.2 Verify Git Sync¶

Make a small change in staging workspace (add a comment to a script)
Click "Deploy" in Windmill UI
Verify commit appears in staging branch
Verify promotion branch created: wm_deploy/staging/<item-path>

Phase 3: GitHub Actions Setup (30 minutes)¶

3.1 Create GitHub Secrets¶

Store in repository settings → Secrets and variables → Actions:

WMILL_TOKEN_PROD: <prod-workspace-token>

Note: WMILL_URL is hardcoded in the workflow environment variables, not stored as a secret.

To get prod workspace token: 1. Create terraform-gitops-prod workspace in Windmill 2. Generate token: Settings → Tokens → Create Token 3. Token permissions: Admin access for sync operations

3.2 Create Workflow: Auto-PR on Staging Commit¶

Create .github/workflows/windmill-open-pr.yaml:

name: Windmill - Open PR on Staging Commit

on:
  push:
    branches:
      - staging
    paths:
      - 'windmill/**'

jobs:
  open-pr:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Check for existing PR
        id: check-pr
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          PR_COUNT=$(gh pr list --base main --head staging --json number --jq 'length')
          echo "count=$PR_COUNT" >> $GITHUB_OUTPUT

      - name: Create Pull Request
        if: steps.check-pr.outputs.count == '0'
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr create \
            --title "Windmill: Deploy staging → prod" \
            --body "Automated PR to deploy Windmill changes from staging to production.

          **Changed Files:**
          $(git diff --name-only origin/main...origin/staging -- windmill/)

          **Review checklist:**
          - [ ] Verify all scripts are tested in staging
          - [ ] Check for secrets/credentials exposure
          - [ ] Confirm Terraform changes are safe
          - [ ] Validate approval workflows" \
            --base main \
            --head staging \
            --label windmill \
            --label autogenerated

3.3 Create Workflow: Auto-Deploy on PR Merge¶

Create .github/workflows/windmill-deploy-prod.yaml:

name: Windmill - Deploy to Production

on:
  pull_request:
    types: [closed]
    branches:
      - main
    paths:
      - 'windmill/**'

jobs:
  deploy:
    if: github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'windmill')
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          ref: main

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install Windmill CLI
        run: npm install -g windmill-cli

      - name: Configure Windmill workspace
        env:
          WMILL_TOKEN: ${{ secrets.WMILL_TOKEN_PROD }}
        run: |
          cd windmill
          wmill workspace add ${{ env.WINDMILL_WORKSPACE }} ${{ env.WINDMILL_WORKSPACE }} "${{ env.WINDMILL_URL }}" "$WMILL_TOKEN"

      - name: Deploy to production
        run: |
          cd windmill
          wmill sync push --fail-on-error

      - name: Comment on PR
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh pr comment ${{ github.event.pull_request.number }} \
            --body "✅ Successfully deployed to production workspace"

Phase 4: Production Workspace Setup (15 minutes)¶

4.1 Create Production Workspace¶

Navigate to Windmill UI
Create workspace: terraform-gitops-prod
Configure identical settings to staging (default runtime, etc.)
Important: Do NOT enable Git sync on prod workspace

4.2 Initial Production Deployment¶

cd windmill

# Configure prod workspace
npx wmill workspace add terraform-gitops-prod terraform-gitops-prod https://windmill.fzymgc.house <token>

# Deploy current state to prod
npx wmill sync push

4.3 Configure Resources and Variables¶

Production-specific resources/variables that should differ:

Discord webhook URLs: Different channels for staging vs prod
Vault tokens: Consider separate tokens with different policies
S3 buckets: Different prefixes or buckets for Terraform state

These should be configured manually in each workspace and will NOT sync between workspaces (skipVariables/skipResources settings).

Phase 5: Testing and Validation (30 minutes)¶

5.1 Test Staging Deployment¶

Make a non-critical change in staging workspace
Deploy the change
Verify:
Commit appears in staging branch
Change works as expected in staging

5.2 Test PR Workflow¶

Push to staging branch (should happen automatically from Windmill)
Verify GitHub Action creates PR
Review PR for correctness
Merge PR

5.3 Test Production Deployment¶

Verify GitHub Action triggers on PR merge
Check action logs for successful deployment
Verify change appears in prod workspace
Test the deployed workflow in prod

5.4 Test Approval Workflow¶

Once deployed to prod, test the fixed approval workflow:

Trigger deploy_vault flow in staging
Verify approval step works (will need the fixed approval script)
Promote to prod via PR
Verify approval works in prod

Rollback Plan¶

If Issues Occur During Migration¶

Staging workspace issues:

# Disable Git sync in Windmill UI
# Continue using terraform-gitops workspace
git checkout main

GitHub Actions failures:

# Disable workflows
git checkout main -- .github/workflows/windmill-*.yaml
# Deploy manually to prod
cd windmill
wmill workspace add terraform-gitops-prod terraform-gitops-prod https://windmill.fzymgc.house <token>
wmill sync push

Production deployment breaks:

# Revert main branch
git revert <commit-hash>
# Redeploy to prod
cd windmill
wmill sync push

Post-Migration Tasks¶

1. Update Documentation¶

Update windmill/README.md with new workflow
Document staging vs prod differences
Add troubleshooting guide

2. Configure Branch Protection¶

In GitHub repository settings:

Branch: main
- Require pull request reviews (1 reviewer)
- Require status checks to pass
- Require conversation resolution
- Do not allow bypassing the above settings

3. Decommission Old Workspace¶

Once stable for 1 week:

Archive terraform-gitops workspace
Remove from Windmill UI (or mark as deprecated)

4. Team Training¶

Document the new workflow:

Development Flow:
1. Make changes in staging workspace UI
2. Deploy → auto-commits to staging branch
3. Review auto-created PR
4. Merge PR → auto-deploys to prod
5. Verify in prod workspace

Emergency Fix:
1. Make change in staging
2. Deploy and test
3. Request expedited PR review
4. Merge and verify prod deployment

Success Criteria¶

[ ] Staging workspace created and functional
[ ] Git sync configured on staging (sync + promotion modes)
[ ] Production workspace created
[ ] GitHub Actions workflows deployed and tested
[ ] Successful staging → prod deployment via PR
[ ] Branch protection enabled on main
[ ] Approval workflow fixed and tested
[ ] Team trained on new workflow
[ ] Documentation updated

Troubleshooting¶

Git Sync Not Committing¶

Symptoms: Changes deployed but no commit in staging branch

Fix: 1. Check Git sync settings in Windmill UI 2. Verify token has contents: write permission 3. Check Windmill logs: kubectl --context fzymgc-house logs -n windmill -l app.kubernetes.io/name=windmill-app

PR Not Auto-Creating¶

Symptoms: Commit appears in staging but no PR

Fix: 1. Check GitHub Actions tab for workflow run errors 2. Verify workflow file syntax 3. Ensure GH_TOKEN has PR creation permissions

Production Deployment Fails¶

Symptoms: PR merges but prod workspace not updated

Fix: 1. Check GitHub Actions logs for wmill sync push errors 2. Verify WMILL_TOKEN_PROD secret is valid 3. Check Windmill prod workspace logs for permission errors

Approval Workflow Still Failing¶

Symptoms: unknown variant 'approval' error in prod

Fix: 1. The approval script fix needs to be deployed through the staging → prod workflow 2. Create the fixed approval script in staging 3. Deploy and test in staging 4. Promote to prod via PR

References¶

Appendix: Configuration Examples¶

wmill.yaml (Final Configuration)¶

workspace: staging  # or prod, depending on which workspace
version: v2
defaultRuntime: python3
defaultTs: bun

includes:
  - "**"

excludes: []

# Full sync for both workspaces
skipVariables: false
skipResources: false
skipResourceTypes: false
skipSecrets: false
skipScripts: false
skipFlows: false
skipApps: false
skipFolders: false

includeSchedules: true
includeTriggers: true
includeUsers: true
includeGroups: true
includeSettings: true
includeKey: false  # Never sync workspace encryption key

codebases:
  - relativePath: "."
    defaultRuntime: python3
    defaultTs: bun

GitHub PAT Permissions¶

Fine-grained personal access token for Git sync:

Repository: fzymgc-house/selfhosted-cluster
Permissions:
  - Contents: Read and write
  - Pull requests: Read and write
  - Metadata: Read-only (automatic)