RustFS (NAS object storage)
RustFS is the S3-compatible object store on the TrueNAS host nas (192.168.20.200). It runs as
the TrueNAS custom app rustfs. The Ansible role ansible/roles/nas-rustfs manages it through
the TrueNAS middleware API, the same way as the other NAS apps (see NAS).
Endpoints and consumers
Section titled “Endpoints and consumers”| Name | Address | TLS | Path |
|---|---|---|---|
s3-direct.nas.fzymgc.house |
192.168.20.203 and fddb:f665:73f7:1:aaaa::203, port 443 |
Estate ICA1 certificate from the sidecar vault-agent |
nginx terminator rustfs-tls, then RustFS port 9000 |
s3.nas.fzymgc.house |
NAS Traefik | Let’s Encrypt through Cloudflare | Traefik, then RustFS port 9000 |
s3-console.nas.fzymgc.house |
NAS Traefik | Let’s Encrypt through Cloudflare | Traefik, then the RustFS console on port 9001 |
LightningStream on the DNS resolvers sandiego and portland replicates the PowerDNS LMDB
through the bucket lightningstream-auth51 on s3-direct. If RustFS stops, DNS answers do not
change, but record changes stop reaching the other resolver.
To make sure that both S3 endpoints answer, run this from a host on the LAN, for example a resolver:
scripts/dns-adhoc.sh sandiego -m shell \ -a 'curl -sS -o /dev/null -w "%{http_code}\n" https://s3-direct.nas.fzymgc.house/health; curl -sS -o /dev/null -w "%{http_code}\n" https://s3.nas.fzymgc.house/health'Both lines print 200.
Credentials
Section titled “Credentials”| Vault path (KV v2) | Contents |
|---|---|
secret/fzymgc-house/infrastructure/nas/rustfs-admin |
The root identity: access_key, secret_key |
secret/fzymgc-house/infrastructure/nas/rustfs-s3 |
The scoped LightningStream key for lightningstream-auth51 |
secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key |
A copy of the encryption key for the data dataset |
The role writes the root identity to files under /mnt/main/rustfs/root-cred. The compose
passes only the file paths to RustFS, so docker inspect and app.config do not show the secret.
Apply the role
Section titled “Apply the role”export VAULT_TOKEN="$(vault print token)"scripts/nas-playbook.sh --tags nas-rustfs --check --diff # dry runscripts/nas-playbook.sh --tags nas-rustfs # applyThe control-node token must read the role id of the AppRole cert-agent-nas-rustfs and create its
secret id. The first run creates the secret id, and later runs reuse the one on the NAS.
The role restarts the app only when the compose, the terminator configuration, the root credential
or a file of the sidecar vault-agent changed. A restart stops LightningStream replication for a
few seconds.
The certificate of s3-direct
Section titled “The certificate of s3-direct”The app has a third container, the sidecar vault-agent. It runs Vault Agent from the image
hashicorp/vault at vault_agent_version (ansible/inventory/group_vars/all.yml) with the AppRole
cert-agent-nas-rustfs. The agent renders the certificate of the Vault PKI role
nas-rustfs-s3-server (720h, CN s3-direct.nas.fzymgc.house) as one bundle,
/mnt/main/rustfs/certs/tls.pem, and renews it at half of its life. The bundle holds the
certificate, the CA chain and the key, and nginx reads all three from it.
- The sidecar runs as uid 568, the owner of its AppRole files and of the bundle, with no
capability and a read-only root file system. Its configuration, its AppRole files and its CA
are in
/mnt/main/rustfs/vault-agent. - The terminator
rustfs-tlsstarts after the healthcheck of the sidecar finds the bundle. A loop in its command reloads nginx every hour, so a renewal needs no restart. - Host certificates describes the agent, the expiry alert, the rotation of the secret id and a manual issue.
To make sure that the terminator serves the certificate in the bundle, read both serials. The two serials are equal, except for up to one hour after a renewal:
scripts/nas-adhoc.sh nas -b -m command \ -a 'openssl x509 -noout -serial -enddate -in /mnt/main/rustfs/certs/tls.pem'scripts/dns-adhoc.sh sandiego -m shell \ -a 'echo | openssl s_client -connect 192.168.20.203:443 -servername s3-direct.nas.fzymgc.house 2>/dev/null | openssl x509 -noout -serial -enddate'Storage
Section titled “Storage”RustFS stores all objects and its IAM state in the dataset main/rustfs/data, mounted at
/mnt/main/rustfs/data. The dataset is its own encryption root. TrueNAS generated its key, holds
it in the appliance keystore and unlocks the dataset at boot. Do not change it to a passphrase
key. A passphrase key does not unlock at boot, and RustFS then stays down until an operator
unlocks it.
The role creates the dataset when it is absent. If the dataset exists and is not encrypted, the role stops with an error and does not touch the app.
To make sure that the dataset is encrypted, mounted and unlocked:
scripts/nas-adhoc.sh nas -b -m command \ -a 'zfs get -H -o property,value encryption,keyformat,keystatus,mounted main/rustfs/data'The output shows aes-256-gcm, hex, available and yes.
Back up the dataset key
Section titled “Back up the dataset key”Run this once after the role creates the dataset. Vault holds the copy for a rebuild of the NAS:
scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j pool.dataset.export_key main/rustfs/data'vault kv put secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key key=<hex>Move the data to an encrypted dataset
Section titled “Move the data to an encrypted dataset”Use this procedure when the role stops because main/rustfs/data exists without encryption.
ZFS cannot encrypt an existing dataset, so the procedure copies the data into a new one. RustFS
is down from step 1 to step 5.
-
Stop the app:
Terminal window scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j app.stop rustfs' -
Rename the unencrypted dataset. Its snapshots stay with it. The middleware refuses a rename without
force, because it does not look for other users of the dataset. Make sure that the app is stopped and that no sandbox mounts the dataset before you run it:Terminal window scripts/nas-adhoc.sh nas -b -m command \-a "midclt call pool.dataset.rename main/rustfs/data '{\"new_name\":\"main/rustfs/data-unencrypted\",\"force\":true}'" -
Run the role. It creates the encrypted
main/rustfs/dataand gives it to uid 568. The app stays stopped, because the compose did not change:Terminal window scripts/nas-playbook.sh --tags nas-rustfs -
Copy the data. The trailing slashes are necessary:
Terminal window scripts/nas-adhoc.sh nas -b -m command \-a 'rsync -aHAX --numeric-ids /mnt/main/rustfs/data-unencrypted/ /mnt/main/rustfs/data/' -
Start the app:
Terminal window scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j app.start rustfs' -
Make sure that LightningStream replicates again, one resolver at a time:
Terminal window scripts/dns-playbook.sh --limit sandiego --tags lightningstream-verifyscripts/dns-playbook.sh --limit portland --tags lightningstream-verify -
Back up the dataset key to Vault. See Back up the dataset key.
-
When replication is healthy on both resolvers, delete the old dataset:
Terminal window scripts/nas-adhoc.sh nas -b -m command \-a "midclt call pool.dataset.delete main/rustfs/data-unencrypted '{\"recursive\":true}'"
Backups
Section titled “Backups”Two copies protect the data:
- The recursive snapshot tasks on the
mainpool take daily, weekly, monthly and yearly ZFS snapshots ofmain/rustfs/data. These are the consistent local copies. - Kopia copies
/mnt/main/rustfs/datato Backblaze B2 every night, at the time innas_kopia_backup_oncalendar, with the other leaves innas_kopia_leaves. Kopia encrypts the data before it leaves the NAS.
Kopia reads live files. If RustFS writes an object during the backup, the B2 copy of that object can be incomplete. The next nightly run copies it again.
After the dataset changes, give Kopia access to the new dataset:
-
Converge the sandbox, so that the new bind mount exists:
Terminal window scripts/nas-playbook.sh --tags nas-sandbox-instance -
Restart
nas-support. A new FILESYSTEM device mounts only at container start. Read the container id first, then use it as<id>:Terminal window scripts/nas-adhoc.sh nas -b -m command \-a "midclt call container.query '[[\"name\",\"=\",\"nas-support\"]]' '{\"select\":[\"id\"]}'"scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j container.stop <id>'scripts/nas-adhoc.sh nas -b -m command -a 'midclt call container.start <id>' -
Grant Kopia read access:
Terminal window scripts/nas-playbook.sh --tags nas-kopia-acls -
Render the backup script again. The
nas-kopia-backuprole writes the list of leaves into/usr/local/bin/kopia-b2-backuponnas-support. Until this role runs, the nightly backup skips the new leaf and reports success:Terminal window scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backupThe play connects to
nas-supportat 192.168.40.32. A workstation that reaches the NAS over Tailscale while a worker is the primary advertiser of Core connects to that address. Whiletruenas-scaleis primary, the path through the gateway also works, because the NAS holds no Core address. If the connection still fails, add-e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none -o ProxyCommand=\"ssh -W %h:%p <user>@<lan-host>\"'"with a host on the LAN. -
After the next nightly run, make sure that the snapshot exists. This command runs
kopiain the sandbox from the NAS host, so it does not need a connection to 192.168.40.32:Terminal window scripts/nas-adhoc.sh nas -b -m shell -a 'pid=$(midclt call container.query "[[\"name\",\"=\",\"nas-support\"]]" "{\"select\":[\"status\"]}" | python3 -c "import json,sys;print(json.load(sys.stdin)[0][\"status\"][\"pid\"])"); nsenter -t $pid -a -- env KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia snapshot list --all | grep -A2 "rustfs/data$"'
Restore
Section titled “Restore”A copy of the RustFS data directory is a complete server: .rustfs.sys holds the IAM state,
service accounts included, and each bucket is a directory. The root credential is not in the
copy. RustFS reads it from the environment, and Vault holds it at
secret/fzymgc-house/infrastructure/nas/rustfs-admin.
Restore on the NAS
Section titled “Restore on the NAS”Use this procedure when the live data is lost or damaged. Pick one source:
- A ZFS snapshot, when the pool is intact. The snapshots are under
/mnt/main/rustfs/data/.zfs/snapshot/. - The Kopia snapshot in B2, when the pool is lost. List it with the command in step 5 of Backups.
Then:
- Stop the app:
midclt call -j app.stop rustfs. - If the pool is rebuilt, run the role. It creates the encrypted dataset. If the dataset key
must be imported instead, use the copy in Vault at
secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key. - Copy the snapshot content into
/mnt/main/rustfs/datawithrsync -aHAX --numeric-ids, or restore it there withkopia snapshot restore <snapshot id> /mnt/main/rustfs/data. - Give the files to uid 568:
chown -R 568:568 /mnt/main/rustfs/data. - Start the app, then run the LightningStream verify on each resolver, one at a time.
LightningStream holds the full LMDB on each resolver and writes new snapshots all the time, so a restored bucket that is a few hours old catches up by itself.
Rehearse a restore
Section titled “Rehearse a restore”A rehearsal restores a copy away from the NAS and serves it with a scratch RustFS on loopback. It does not touch the live server or DNS.
-
Connect a read-only Kopia client with its own configuration. The commands read each secret from
secret/fzymgc-house/infrastructure/nas/kopia-b2into the environment. Do not type a secret or pass it as a flag, so that it stays out of the shell history and the process list:Terminal window V=secret/fzymgc-house/infrastructure/nas/kopia-b2export AWS_ACCESS_KEY_ID=$(vault kv get -field=b2_key_id $V)export AWS_SECRET_ACCESS_KEY=$(vault kv get -field=b2_application_key $V)export KOPIA_PASSWORD=$(vault kv get -field=repo_password $V)export KOPIA_CONFIG_PATH=$PWD/kopia.configkopia repository connect s3 --readonly --bucket=$(vault kv get -field=b2_bucket $V) \--endpoint=s3.$(vault kv get -field=b2_region $V).backblazeb2.com --cache-directory=$PWD/cachekopia snapshot list --all | grep -A2 'rustfs/data$'kopia snapshot restore <snapshot id> $PWD/restoreFor a ZFS snapshot, stream it instead:
ssh nas-automation@192.168.20.200 "sudo -n tar -C /mnt/main/rustfs/data/.zfs/snapshot/<name> -cf - ." | tar -C restore -xf -. Use thenas-automationkey from Vault. -
Start the same RustFS version on the copy, on loopback, with the root credential from Vault:
Terminal window R=secret/fzymgc-house/infrastructure/nas/rustfs-adminRUSTFS_ACCESS_KEY=$(vault kv get -field=access_key $R) \RUSTFS_SECRET_KEY=$(vault kv get -field=secret_key $R) \RUSTFS_VOLUMES=$PWD/restore RUSTFS_ADDRESS=127.0.0.1:19000 \RUSTFS_CONSOLE_ENABLE=false RUSTFS_OBS_ENDPOINT= ./rustfs -
Make sure that:
- The root credential lists
lightningstream-auth51and can GET every object. RustFS checks a bitrot hash on each read and refuses a damaged object with503, so a full set of GETs proves that the data is intact. - The LightningStream key from
secret/fzymgc-house/infrastructure/nas/rustfs-s3can list and GET, and getsAccessDeniedfor a bucket create. This proves that the service account and its policy came back. - An object that exists in both the copy and the live bucket has the same ETag and size.
- The root credential lists
-
Stop the scratch server, run
kopia repository disconnect, and delete the copy. It holds the IAM state.
Buckets and IAM (Terraform)
Section titled “Buckets and IAM (Terraform)”tf/rustfs manages the buckets on RustFS, in the HCP Terraform workspace
main-cluster-rustfs. It uses the aminueza/minio provider, which speaks the MinIO admin API
that RustFS implements. The run executes on the agent pool in k3s, reaches RustFS through
s3.nas.fzymgc.house, and reads the root credential from Vault with an ephemeral read, so the
credential never enters state. A merge to main applies the change.
tf/rustfs manages the bucket lightningstream-auth51. Versioning is off for it, because
LightningStream deletes its old snapshots and versioning would keep each one.
To make sure that Terraform and RustFS agree, read the latest run of main-cluster-rustfs in HCP
Terraform. A plan with 0 to add, 0 to change, 0 to destroy means no drift.
To add a bucket for important data:
- Add a
minio_s3_bucketwithlifecycle { prevent_destroy = true }. - Add a
minio_s3_bucket_versioningwithstatus = "Enabled", so that an overwrite or a delete keeps the old version. - Give the consumer its own IAM user and policy, and store its key in Vault. Do not use the root credential or a service account of root for a consumer.
- Open a PR. The speculative plan on the PR is the last look before the apply.
Do not set acl on a bucket. The provider applies an acl change, and the first write of a
new bucket, as a write of the whole bucket policy. An acl next to a bucket policy deletes that
policy.
The LightningStream access key is a service account of the root user, and tf/rustfs does not
manage it.
Telemetry and alerts
Section titled “Telemetry and alerts”RustFS sends metrics and logs over OTLP HTTP to the NAS collector otelcol-nas, at
http://otelcol:4318 on the nas-internal Docker network. In ClickStack, the data has
ServiceName = 'rustfs' and ResourceAttributes['host.name'] = 'nas'. Traces and profiling are
off. See the Telemetry section of NAS for the collector side.
The ClickStack dashboard “RustFS - health” holds these tiles and alerts. The bootstrap script
argocd/app-configs/clickstack-alerts/bootstrap-script.yaml defines them.
| Alert | Threshold and interval | Catches |
|---|---|---|
RustFS telemetry MISSING (app or OTLP path dark) |
count rustfs_process_uptime_seconds, below 1, 15m |
The app stopped, or the OTLP path to the collector broke |
RustFS drive OFFLINE |
max rustfs_cluster_health_drives_offline_count, above_exclusive 0, 15m |
The data dataset is not mounted or its key is not loaded |
The dashboard also shows free and used capacity. Free capacity is the shared main pool.
RustFS has no capacity alert, because TrueNAS sends pool capacity alerts by email.
To make sure that data arrives:
SELECT MetricName, count() FROM default.otel_metrics_gaugeWHERE ServiceName = 'rustfs' AND TimeUnix > now() - INTERVAL 10 MINUTEGROUP BY MetricName ORDER BY MetricNameUpgrade RustFS
Section titled “Upgrade RustFS”Renovate tracks nas_rustfs_version in ansible/roles/nas-rustfs/defaults/main.yml. The tag
is pinned exactly and never floats.
-
Read the upstream release notes for S3 or IAM behavior changes.
-
Make sure that the tag exists:
crane digest rustfs/rustfs:<version>. -
Change
nas_rustfs_versionin a PR and merge it. -
From
main, apply the role. Then run the LightningStream verify on each resolver, one at a time:Terminal window scripts/dns-playbook.sh --limit sandiego --tags lightningstream-verifyscripts/dns-playbook.sh --limit portland --tags lightningstream-verify