Skip to content

RustFS (NAS object storage)

RustFS is the S3-compatible object store on the TrueNAS host nas (192.168.20.200). It runs as the TrueNAS custom app rustfs. The Ansible role ansible/roles/nas-rustfs manages it through the TrueNAS middleware API, the same way as the other NAS apps (see NAS).

Name Address TLS Path
s3-direct.nas.fzymgc.house 192.168.20.203 and fddb:f665:73f7:1:aaaa::203, port 443 Estate ICA1 certificate from the sidecar vault-agent nginx terminator rustfs-tls, then RustFS port 9000
s3.nas.fzymgc.house NAS Traefik Let’s Encrypt through Cloudflare Traefik, then RustFS port 9000
s3-console.nas.fzymgc.house NAS Traefik Let’s Encrypt through Cloudflare Traefik, then the RustFS console on port 9001

LightningStream on the DNS resolvers sandiego and portland replicates the PowerDNS LMDB through the bucket lightningstream-auth51 on s3-direct. If RustFS stops, DNS answers do not change, but record changes stop reaching the other resolver.

To make sure that both S3 endpoints answer, run this from a host on the LAN, for example a resolver:

Terminal window
scripts/dns-adhoc.sh sandiego -m shell \
-a 'curl -sS -o /dev/null -w "%{http_code}\n" https://s3-direct.nas.fzymgc.house/health; curl -sS -o /dev/null -w "%{http_code}\n" https://s3.nas.fzymgc.house/health'

Both lines print 200.

Vault path (KV v2) Contents
secret/fzymgc-house/infrastructure/nas/rustfs-admin The root identity: access_key, secret_key
secret/fzymgc-house/infrastructure/nas/rustfs-s3 The scoped LightningStream key for lightningstream-auth51
secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key A copy of the encryption key for the data dataset

The role writes the root identity to files under /mnt/main/rustfs/root-cred. The compose passes only the file paths to RustFS, so docker inspect and app.config do not show the secret.

Terminal window
export VAULT_TOKEN="$(vault print token)"
scripts/nas-playbook.sh --tags nas-rustfs --check --diff # dry run
scripts/nas-playbook.sh --tags nas-rustfs # apply

The control-node token must read the role id of the AppRole cert-agent-nas-rustfs and create its secret id. The first run creates the secret id, and later runs reuse the one on the NAS.

The role restarts the app only when the compose, the terminator configuration, the root credential or a file of the sidecar vault-agent changed. A restart stops LightningStream replication for a few seconds.

The app has a third container, the sidecar vault-agent. It runs Vault Agent from the image hashicorp/vault at vault_agent_version (ansible/inventory/group_vars/all.yml) with the AppRole cert-agent-nas-rustfs. The agent renders the certificate of the Vault PKI role nas-rustfs-s3-server (720h, CN s3-direct.nas.fzymgc.house) as one bundle, /mnt/main/rustfs/certs/tls.pem, and renews it at half of its life. The bundle holds the certificate, the CA chain and the key, and nginx reads all three from it.

  • The sidecar runs as uid 568, the owner of its AppRole files and of the bundle, with no capability and a read-only root file system. Its configuration, its AppRole files and its CA are in /mnt/main/rustfs/vault-agent.
  • The terminator rustfs-tls starts after the healthcheck of the sidecar finds the bundle. A loop in its command reloads nginx every hour, so a renewal needs no restart.
  • Host certificates describes the agent, the expiry alert, the rotation of the secret id and a manual issue.

To make sure that the terminator serves the certificate in the bundle, read both serials. The two serials are equal, except for up to one hour after a renewal:

Terminal window
scripts/nas-adhoc.sh nas -b -m command \
-a 'openssl x509 -noout -serial -enddate -in /mnt/main/rustfs/certs/tls.pem'
scripts/dns-adhoc.sh sandiego -m shell \
-a 'echo | openssl s_client -connect 192.168.20.203:443 -servername s3-direct.nas.fzymgc.house 2>/dev/null | openssl x509 -noout -serial -enddate'

RustFS stores all objects and its IAM state in the dataset main/rustfs/data, mounted at /mnt/main/rustfs/data. The dataset is its own encryption root. TrueNAS generated its key, holds it in the appliance keystore and unlocks the dataset at boot. Do not change it to a passphrase key. A passphrase key does not unlock at boot, and RustFS then stays down until an operator unlocks it.

The role creates the dataset when it is absent. If the dataset exists and is not encrypted, the role stops with an error and does not touch the app.

To make sure that the dataset is encrypted, mounted and unlocked:

Terminal window
scripts/nas-adhoc.sh nas -b -m command \
-a 'zfs get -H -o property,value encryption,keyformat,keystatus,mounted main/rustfs/data'

The output shows aes-256-gcm, hex, available and yes.

Run this once after the role creates the dataset. Vault holds the copy for a rebuild of the NAS:

Terminal window
scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j pool.dataset.export_key main/rustfs/data'
vault kv put secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key key=<hex>

Use this procedure when the role stops because main/rustfs/data exists without encryption. ZFS cannot encrypt an existing dataset, so the procedure copies the data into a new one. RustFS is down from step 1 to step 5.

  1. Stop the app:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j app.stop rustfs'
  2. Rename the unencrypted dataset. Its snapshots stay with it. The middleware refuses a rename without force, because it does not look for other users of the dataset. Make sure that the app is stopped and that no sandbox mounts the dataset before you run it:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command \
    -a "midclt call pool.dataset.rename main/rustfs/data '{\"new_name\":\"main/rustfs/data-unencrypted\",\"force\":true}'"
  3. Run the role. It creates the encrypted main/rustfs/data and gives it to uid 568. The app stays stopped, because the compose did not change:

    Terminal window
    scripts/nas-playbook.sh --tags nas-rustfs
  4. Copy the data. The trailing slashes are necessary:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command \
    -a 'rsync -aHAX --numeric-ids /mnt/main/rustfs/data-unencrypted/ /mnt/main/rustfs/data/'
  5. Start the app:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j app.start rustfs'
  6. Make sure that LightningStream replicates again, one resolver at a time:

    Terminal window
    scripts/dns-playbook.sh --limit sandiego --tags lightningstream-verify
    scripts/dns-playbook.sh --limit portland --tags lightningstream-verify
  7. Back up the dataset key to Vault. See Back up the dataset key.

  8. When replication is healthy on both resolvers, delete the old dataset:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command \
    -a "midclt call pool.dataset.delete main/rustfs/data-unencrypted '{\"recursive\":true}'"

Two copies protect the data:

  • The recursive snapshot tasks on the main pool take daily, weekly, monthly and yearly ZFS snapshots of main/rustfs/data. These are the consistent local copies.
  • Kopia copies /mnt/main/rustfs/data to Backblaze B2 every night, at the time in nas_kopia_backup_oncalendar, with the other leaves in nas_kopia_leaves. Kopia encrypts the data before it leaves the NAS.

Kopia reads live files. If RustFS writes an object during the backup, the B2 copy of that object can be incomplete. The next nightly run copies it again.

After the dataset changes, give Kopia access to the new dataset:

  1. Converge the sandbox, so that the new bind mount exists:

    Terminal window
    scripts/nas-playbook.sh --tags nas-sandbox-instance
  2. Restart nas-support. A new FILESYSTEM device mounts only at container start. Read the container id first, then use it as <id>:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m command \
    -a "midclt call container.query '[[\"name\",\"=\",\"nas-support\"]]' '{\"select\":[\"id\"]}'"
    scripts/nas-adhoc.sh nas -b -m command -a 'midclt call -j container.stop <id>'
    scripts/nas-adhoc.sh nas -b -m command -a 'midclt call container.start <id>'
  3. Grant Kopia read access:

    Terminal window
    scripts/nas-playbook.sh --tags nas-kopia-acls
  4. Render the backup script again. The nas-kopia-backup role writes the list of leaves into /usr/local/bin/kopia-b2-backup on nas-support. Until this role runs, the nightly backup skips the new leaf and reports success:

    Terminal window
    scripts/nas-playbook.sh --limit nas-support --tags nas-kopia-backup

    The play connects to nas-support at 192.168.40.32. A workstation that reaches the NAS over Tailscale while a worker is the primary advertiser of Core connects to that address. While truenas-scale is primary, the path through the gateway also works, because the NAS holds no Core address. If the connection still fails, add -e "ansible_ssh_common_args='-o IdentitiesOnly=yes -o IdentityAgent=none -o ProxyCommand=\"ssh -W %h:%p <user>@<lan-host>\"'" with a host on the LAN.

  5. After the next nightly run, make sure that the snapshot exists. This command runs kopia in the sandbox from the NAS host, so it does not need a connection to 192.168.40.32:

    Terminal window
    scripts/nas-adhoc.sh nas -b -m shell -a 'pid=$(midclt call container.query "[[\"name\",\"=\",\"nas-support\"]]" "{\"select\":[\"status\"]}" | python3 -c "import json,sys;print(json.load(sys.stdin)[0][\"status\"][\"pid\"])"); nsenter -t $pid -a -- env KOPIA_CONFIG_PATH=/etc/nas-kopia-b2/repository.config /home/linuxbrew/.linuxbrew/bin/kopia snapshot list --all | grep -A2 "rustfs/data$"'

A copy of the RustFS data directory is a complete server: .rustfs.sys holds the IAM state, service accounts included, and each bucket is a directory. The root credential is not in the copy. RustFS reads it from the environment, and Vault holds it at secret/fzymgc-house/infrastructure/nas/rustfs-admin.

Use this procedure when the live data is lost or damaged. Pick one source:

  • A ZFS snapshot, when the pool is intact. The snapshots are under /mnt/main/rustfs/data/.zfs/snapshot/.
  • The Kopia snapshot in B2, when the pool is lost. List it with the command in step 5 of Backups.

Then:

  1. Stop the app: midclt call -j app.stop rustfs.
  2. If the pool is rebuilt, run the role. It creates the encrypted dataset. If the dataset key must be imported instead, use the copy in Vault at secret/fzymgc-house/infrastructure/nas/rustfs-zfs-key.
  3. Copy the snapshot content into /mnt/main/rustfs/data with rsync -aHAX --numeric-ids, or restore it there with kopia snapshot restore <snapshot id> /mnt/main/rustfs/data.
  4. Give the files to uid 568: chown -R 568:568 /mnt/main/rustfs/data.
  5. Start the app, then run the LightningStream verify on each resolver, one at a time.

LightningStream holds the full LMDB on each resolver and writes new snapshots all the time, so a restored bucket that is a few hours old catches up by itself.

A rehearsal restores a copy away from the NAS and serves it with a scratch RustFS on loopback. It does not touch the live server or DNS.

  1. Connect a read-only Kopia client with its own configuration. The commands read each secret from secret/fzymgc-house/infrastructure/nas/kopia-b2 into the environment. Do not type a secret or pass it as a flag, so that it stays out of the shell history and the process list:

    Terminal window
    V=secret/fzymgc-house/infrastructure/nas/kopia-b2
    export AWS_ACCESS_KEY_ID=$(vault kv get -field=b2_key_id $V)
    export AWS_SECRET_ACCESS_KEY=$(vault kv get -field=b2_application_key $V)
    export KOPIA_PASSWORD=$(vault kv get -field=repo_password $V)
    export KOPIA_CONFIG_PATH=$PWD/kopia.config
    kopia repository connect s3 --readonly --bucket=$(vault kv get -field=b2_bucket $V) \
    --endpoint=s3.$(vault kv get -field=b2_region $V).backblazeb2.com --cache-directory=$PWD/cache
    kopia snapshot list --all | grep -A2 'rustfs/data$'
    kopia snapshot restore <snapshot id> $PWD/restore

    For a ZFS snapshot, stream it instead: ssh nas-automation@192.168.20.200 "sudo -n tar -C /mnt/main/rustfs/data/.zfs/snapshot/<name> -cf - ." | tar -C restore -xf -. Use the nas-automation key from Vault.

  2. Start the same RustFS version on the copy, on loopback, with the root credential from Vault:

    Terminal window
    R=secret/fzymgc-house/infrastructure/nas/rustfs-admin
    RUSTFS_ACCESS_KEY=$(vault kv get -field=access_key $R) \
    RUSTFS_SECRET_KEY=$(vault kv get -field=secret_key $R) \
    RUSTFS_VOLUMES=$PWD/restore RUSTFS_ADDRESS=127.0.0.1:19000 \
    RUSTFS_CONSOLE_ENABLE=false RUSTFS_OBS_ENDPOINT= ./rustfs
  3. Make sure that:

    • The root credential lists lightningstream-auth51 and can GET every object. RustFS checks a bitrot hash on each read and refuses a damaged object with 503, so a full set of GETs proves that the data is intact.
    • The LightningStream key from secret/fzymgc-house/infrastructure/nas/rustfs-s3 can list and GET, and gets AccessDenied for a bucket create. This proves that the service account and its policy came back.
    • An object that exists in both the copy and the live bucket has the same ETag and size.
  4. Stop the scratch server, run kopia repository disconnect, and delete the copy. It holds the IAM state.

tf/rustfs manages the buckets on RustFS, in the HCP Terraform workspace main-cluster-rustfs. It uses the aminueza/minio provider, which speaks the MinIO admin API that RustFS implements. The run executes on the agent pool in k3s, reaches RustFS through s3.nas.fzymgc.house, and reads the root credential from Vault with an ephemeral read, so the credential never enters state. A merge to main applies the change.

tf/rustfs manages the bucket lightningstream-auth51. Versioning is off for it, because LightningStream deletes its old snapshots and versioning would keep each one.

To make sure that Terraform and RustFS agree, read the latest run of main-cluster-rustfs in HCP Terraform. A plan with 0 to add, 0 to change, 0 to destroy means no drift.

To add a bucket for important data:

  1. Add a minio_s3_bucket with lifecycle { prevent_destroy = true }.
  2. Add a minio_s3_bucket_versioning with status = "Enabled", so that an overwrite or a delete keeps the old version.
  3. Give the consumer its own IAM user and policy, and store its key in Vault. Do not use the root credential or a service account of root for a consumer.
  4. Open a PR. The speculative plan on the PR is the last look before the apply.

Do not set acl on a bucket. The provider applies an acl change, and the first write of a new bucket, as a write of the whole bucket policy. An acl next to a bucket policy deletes that policy.

The LightningStream access key is a service account of the root user, and tf/rustfs does not manage it.

RustFS sends metrics and logs over OTLP HTTP to the NAS collector otelcol-nas, at http://otelcol:4318 on the nas-internal Docker network. In ClickStack, the data has ServiceName = 'rustfs' and ResourceAttributes['host.name'] = 'nas'. Traces and profiling are off. See the Telemetry section of NAS for the collector side.

The ClickStack dashboard “RustFS - health” holds these tiles and alerts. The bootstrap script argocd/app-configs/clickstack-alerts/bootstrap-script.yaml defines them.

Alert Threshold and interval Catches
RustFS telemetry MISSING (app or OTLP path dark) count rustfs_process_uptime_seconds, below 1, 15m The app stopped, or the OTLP path to the collector broke
RustFS drive OFFLINE max rustfs_cluster_health_drives_offline_count, above_exclusive 0, 15m The data dataset is not mounted or its key is not loaded

The dashboard also shows free and used capacity. Free capacity is the shared main pool. RustFS has no capacity alert, because TrueNAS sends pool capacity alerts by email.

To make sure that data arrives:

SELECT MetricName, count() FROM default.otel_metrics_gauge
WHERE ServiceName = 'rustfs' AND TimeUnix > now() - INTERVAL 10 MINUTE
GROUP BY MetricName ORDER BY MetricName

Renovate tracks nas_rustfs_version in ansible/roles/nas-rustfs/defaults/main.yml. The tag is pinned exactly and never floats.

  1. Read the upstream release notes for S3 or IAM behavior changes.

  2. Make sure that the tag exists: crane digest rustfs/rustfs:<version>.

  3. Change nas_rustfs_version in a PR and merge it.

  4. From main, apply the role. Then run the LightningStream verify on each resolver, one at a time:

    Terminal window
    scripts/dns-playbook.sh --limit sandiego --tags lightningstream-verify
    scripts/dns-playbook.sh --limit portland --tags lightningstream-verify