Skip to content

Turing Pi boards

The two Turing Pi 2 boards, tpi-alpha and tpi-beta, each hold four k3s nodes and one BMC. This page covers the uplink of a board, the legs of its nodes and the address of its BMC. It also holds the steps that move a board between networks.

Each board carries an onboard switch that joins the BMC and the four nodes to one uplink. The switch has no VLAN filtering: it passes a tagged frame untouched, and it puts every untagged frame on the native network of the uplink. So the native network of the uplink decides where the BMC lands, and each node picks its own network by tagging.

Item Value
tpi-alpha uplink switch-office port 12, profile TpiUplink
tpi-beta uplink switch-office port 11, profile TpiUplink
TpiUplink Core native, Main and Lab tagged, every other network excluded (tf/unifi/port-profiles.tf)
Node legs end0.3000 for Main and end0.1000 for Lab, rendered by tp2-bootstrap-node. end0 holds no address
BMC Untagged, so on Core: tpi-alpha-bmc at 192.168.40.40 and tpi-beta-bmc at 192.168.40.41, from their dns-control entries

The group_vars of each board set tp2_node_main_tagged: true. With the default false, the template renders Main on end0 untagged, which is the shape for an uplink that is native Main. The two values change together with the port, in the same pull request.

A port reaches TpiUplink from another profile, never from its own network keys. The provider does not clear the profile of a port that moves to its own keys, so the controller keeps the old profile and ignores the keys (UniFi).

The BMC presents a self-signed certificate. Log in with a JSON POST and pin the public key:

Terminal window
host=tpi-beta-bmc.fzymgc.house
pin=$(echo | openssl s_client -connect "$host:443" 2>/dev/null | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64)
export BMC_PASS=$(vault kv get -field=password secret/fzymgc-house/infrastructure/bmc/tpi-beta)
tok=$(curl -sS --insecure --pinnedpubkey "sha256//$pin" -H 'Content-Type: application/json' -d "{\"username\":\"root\",\"password\":\"$BMC_PASS\"}" "https://$host/api/bmc/authenticate" | jq -r .id)
curl -sS --insecure --pinnedpubkey "sha256//$pin" -H "Authorization: Bearer $tok" "https://$host/api/bmc?opt=get&type=about" | jq .

The alpha password is at bmc/tpi-alpha under the same mount. Both boards run an upstream development build of the BMC firmware, which type=about reports.

The order matters. Cilium’s VLAN filter on end0 drops a tagged frame for a VLAN device that Cilium does not own, and a restart of the agent never adopts an unlisted device.

  1. Make sure that cilium_devices of the cilium role lists end0.3000, and converge the role (k3s-playbook.yml --tags cilium). Cilium selects a listed device when it appears, so the list can lead the netplan change.

  2. Create or change the dns-control entry of the BMC with its Core address.

  3. In one pull request, set tp2_node_main_tagged: true in the group_vars of the board and give the uplink port the profile TpiUplink. Merge it and confirm the main-cluster-unifi run. At that moment the untagged Main leg of each node stops, and the four nodes go NotReady.

  4. For each node, through its Lab address, render and apply the tagged Main leg:

    Terminal window
    cd ansible
    uv run ansible-playbook -i inventory/hosts.yml bootstrap-nodes-playbook.yml --tags network --limit tpi-beta-4 -e ansible_host=192.168.200.154

    The node is Ready again within about a minute. Take the next node when it is.

  5. Read the port back from the controller. It holds the profile id of TpiUplink, not of AllMainDefault.

The Lab leg is the management path during the move, because it is tagged and does not change. If a node is lost, the BMC serial console (tpi uart -n <slot>) is the recovery path.

After the flip, the BMC still holds its Main lease, which is now on the wrong network, until its DHCP client renews. That is up to six hours. To force it, put a throwaway container on Core with a temporary address inside the old subnet. The container then shares a segment with the BMC. Send the DHCP client of the BMC the renew signal. Kea refuses the old lease on the Core leg, so the client falls back to discovery and takes the Core reservation.

Terminal window
T=.claude/skills/nas-throwaway-lxc/scripts/throwaway_lxc.py
uv run $T up bmcfix --existing-vlan 1 --packages sshpass
uv run $T exec bmcfix -- ip addr add 192.168.23.230/22 dev eth0
export BMC_PASS=$(vault kv get -field=password secret/fzymgc-house/infrastructure/bmc/tpi-beta)
printf '%s\n' "$BMC_PASS" | uv run $T exec bmcfix --stdin -- sshpass -d 0 ssh -o StrictHostKeyChecking=no root@192.168.23.229 'kill -USR1 $(pidof udhcpc)'
uv run $T down

The BMC answers on its Core address within seconds. Do not bounce the uplink port for this. The bridge port of the BMC on the onboard switch never goes down, so the client sees no link change.

tpi-alpha-bmc refuses password login over SSH and accepts the operator’s key only, so the password path above fails on it with Permission denied (publickey,keyboard-interactive). For that board, give the container an SSH server and the operator’s public keys, and reach the BMC from a workstation with agent forwarding. The temporary address is a /32 with one host route to the BMC, because a /22 on the container pulls every Main-subnet reply onto the Core segment, where the workstation is not.

Terminal window
T=.claude/skills/nas-throwaway-lxc/scripts/throwaway_lxc.py
uv run $T up jump --existing-vlan 1 --packages openssh-server
ssh-add -L | uv run $T exec jump --stdin -- sh -c 'mkdir -p /root/.ssh && cat > /root/.ssh/authorized_keys && chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys'
uv run $T exec jump -- sh -c 'ip addr add 192.168.23.230/32 dev eth0; ip route add 192.168.22.140/32 dev eth0 src 192.168.23.230'
ssh -A root@<the Core address of the container> # then: ssh root@192.168.22.140, then: kill -USR1 $(pidof udhcpc)
uv run $T down

A rebuilt node boots with the untagged DHCP configuration of its image, so it takes a Core lease until tp2-bootstrap-node renders the tagged configuration. Core has Kea, and the role connects by the address you pass it, so pass that Core address as ansible_host for the first run.