tpi-beta-4 — console ring buffer, 2026-08-23

Captured via Turing Pi beta BMC (192.168.23.229): tpi uart -n 4 get.

Why this file exists: Armbian runs armbian-ramlog.service — /var/log lives in RAM and syncs periodically. The root filesystem went read-only on 2026-07-27, so nothing has persisted since. This console capture is very likely the ONLY forensic record of the failure, and a reboot destroys the ring buffer. Do not discard it.

Kernel messages (verbatim, trailing portion after login prompt)

Armbian 26.2.1 noble ttyS0
tpi-beta-4 login:
[ 596049.379023] rk_gmac-dwmac fe1c0000.ethernet end0: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5384 ms
[4271483.934162] EXT4-fs error (device dm-1): ext4_put_super:1308: comm vector: Couldn't clean up the journal
[4271483.945317] EXT4-fs (dm-1): Remounting filesystem read-only
[6589557.793336] BTRFS: error (device nvme0n1p2) in btrfs_commit_transaction:2536: errno=-5 IO failure (Error while writing out transaction)
[6589557.798748] BTRFS: error (device nvme0n1p2 state A) in __btrfs_update_delayed_inode:1044: errno=-5 IO failure
[6589557.819156] BTRFS: error (device nvme0n1p2 state EA) in cleanup_transaction:2021: errno=-5 IO failure
[6589557.819298] EXT4-fs error (device sdf): ext4_journal_check_start:87: comm postgres: Detected aborted journal
[6589557.846679] EXT4-fs (sdf): Remounting filesystem read-only
[6590123.768537] EXT4-fs (dm-0): failed to convert unwritten extents to written extents -- potential data loss!  (inode 524325, error -5)
[6590123.783419] EXT4-fs (dm-0): failed to convert unwritten extents to written extents -- potential data loss!  (inode 524325, error -5)
[6590132.322046] EXT4-fs error (device dm-0): ext4_journal_check_start:87: comm nats-server: Detected aborted journal
[6590132.334718] EXT4-fs (dm-0): Remounting filesystem read-only
[7298531.067966] rk_gmac-dwmac fe1c0000.ethernet end0: NETDEV WATCHDOG: CPU: 5: transmit queue 0 timed out 5268 ms
[7299485.945283] rk_gmac-dwmac fe1c0000.ethernet end0: NETDEV WATCHDOG: CPU: 4: transmit queue 0 timed out 5036 ms

Timeline

Kernel timestamps are seconds since boot. Node booted ~2026-05-12; the BTRFS failure at uptime 6589557s (76.27d) lands on 2026-07-27, matching the kubelet’s last posted heartbeat 2026-07-27T08:00:56Z exactly. Relative offsets below are anchor-independent and hold regardless of the exact boot timestamp.

Uptime	Event	Relative to failure
596049s	NETDEV WATCHDOG end0 tx queue timeout (CPU 2)	−69.4d
4271483s	ext4 dm-1 → read-only, comm vector, “Couldn’t clean up the journal”	−26.8d
6589557s	BTRFS errno=-5 IO failure on nvme0n1p2 (commit_transaction, update_delayed_inode, cleanup_transaction) + ext4 sdf → read-only (comm postgres)	0 — node dies
6590123s	ext4 dm-0: “failed to convert unwritten extents — potential data loss!” inode 524325 ×2	+9.4 min
6590132s	ext4 dm-0 → read-only, comm nats-server	+9.6 min
7298531s	NETDEV WATCHDOG (CPU 5)	+8.2d
7299485s	NETDEV WATCHDOG (CPU 4)	+8.2d

Interpretation

errno=-5 is EIO — the block layer returned an error to the filesystem. This is not filesystem corruption that fsck repairs; the device failed to complete writes.

ansible/inventory/host_vars/tpi-beta-4.yml sets host_btrfs_root_volume: /dev/nvme0n1p2 — the failing device is the root volume. The console shows /var/lib/kubelet, /var/lib/longhorn, and /var/lib/rancher mounted as btrfs subvolumes from it, so when it went read-only, kubelet, k3s-agent, and Longhorn all lost their state store at once. That is why all of userspace died while the kernel kept answering ICMP from page cache and issuing RSTs on closed ports.

dm-0, dm-1, and sdf are Longhorn volumes attached to the node — their filesystems went read-only too, one with an explicit potential data loss flag.

The disk had been failing for ~27 days before it took the node down (the dm-1 ext4 read-only event). Nothing alerted on it.

Open question

The NVMe is reportedly recent, so media wear is an unlikely explanation. On RK1 modules the NVMe sits on PCIe, where link-training instability, ASPM behaviour, and power delivery are more common culprits than failing flash. btrfs device stats keeps persistent cumulative error counters and is the highest-value post-reboot check.