Helios64 reloaded: Armbian Trixie w/ stock Samba

#helios64 #armbian #debian #nas #samba #tailscale

Back in 2022 I wrote up my Openmediavault & Nextcloud on Armbian Server Helios64 setup. It served me well. But it aged badly: Debian 10 Buster, Armbian kernel 4.4, OMV 5, NextcloudPi w/ PHP 7.4 from the Sury repo. When the PHP2Shell vulnerabilities (CVE-2026-109618/109619) came along, there simply was no way to patch: the Buster repos are gone (404), and so is any PHP 7.x update. 💀 (Sure, I could have compiled a patched PHP 7.4 myself—on a Buster box w/o security updates for anything else. No thanks.)

Time for a clean slate. My goals:

Kobol is long gone (RIP 🪦), but Armbian still builds images for the Helios64—community supported, but they boot. Mostly. More on that below. 🙈

Like with the MacBook, I did this pair-debugging w/ Claude Code—over SSH, and over the serial console when SSH wasn’t an option.

Before touching anything

I did NOT do a full backup of the 2.3 TB on the RAID (RAID10 out of 5× 4 TB IronWolf, mdadm → LVM → ext4). The plan was to never write to it, only re-mount it. What I did instead:

SD card first

Armbian 26.8.1 Trixie minimal (kernel 6.18.43) on an SD card, checksum + GPG verified. The Helios64 boots SPI → eMMC → SD, and jumper P10 skips the eMMC. TIL: w/ P10 set, the eMMC is invisible to Linux, too (mmc1: Failed to initialize a non-removable card)—not just to the boot ROM.

Booted right away. The RAID showed up on its own as md127 (auto-read-only), LVM stayed inactive w/o lvm2 installed. Good: nothing touched it.

Getting Armbian onto the eMMC

The Kobol wiki assumes an empty eMMC. Mine had a bootloader on it, so the Helios64 always booted from the eMMC, SD card or not. What worked:

  1. From the still-running old system: back up the first 16 MiB of the eMMC, then wipe only the bootloader area (sectors 64–32767; the first partition starts at 32768). With no bootloader on the eMMC, the boot ROM falls back to SD. No jumper needed.
  2. SD card w/ Kobol’s helios64_sdcard_u-boot-only.img.xz. That’s a U-Boot that runs ums mmc 0 and hands the eMMC over USB-C as a mass storage device.
  3. Full image of the old eMMC to the Mac first (15.6 GB, 3.9 GB gzipped). UMS is slow: ~6–7 MB/s, ~40 min. ☕
  4. Then flash Armbian w/ the Armbian Imager straight onto the eMMC.

Two gotchas:

The boss level: random kernel panics at boot

Installed, booted… and then roughly every other boot died ~3 s in. Red LED blinking, or the blue LED steady on w/o heartbeat and the fan going full blast. On the serial console:

SError Interrupt on CPU…
… rockchip_pcie_rd_conf …
Kernel panic - not syncing: Asynchronous SError Interrupt

That’s the PCIe bus scan—where the JMB585 SATA controller sits. Score w/ 6.18.43: 2 good boots out of 7.

What I tried, and what didn’t help:

So it’s not the kernel version. It’s a known RK3399 hardware quirk (RockPro64 and Rock Pi 4 are affected as well): a config read on a PCIe device that isn’t ready gets reported to the CPU as an abort. Timing-dependent, hence random.

The recipe: don’t prevent it, recover from it

The crash only happens during the PCIe scan at boot, before the SATA driver loads. The disks are never touched. So instead of fighting the hardware, I let the box heal itself and otherwise stay on stock Armbian: current kernel, no overlay, no apt-mark hold.

Exactly two deviations from stock:

1. Panic → reboot in /boot/armbianEnv.txt:

extraargs=panic=10 oops=panic

oops=panic matters: it turns the silent freeze (“synchronous external abort”) into a panic as well, and panic=10 reboots 10 s later. Armbian’s stock watchdog daemon catches whatever is left—no need for your own watchdog config. (I added one at first. It stole /dev/watchdog from the stock daemon. Removed. 🙃)

2. Reboot if the SATA controller is missing. Rarely the scan doesn’t crash but fails differently (PCIe link training gen1 timeout! … probe failed with error -110). Linux comes up fine—w/o disks. A tiny oneshot service catches that, /usr/local/sbin/helios64-pcie-check:

#!/bin/sh
# Helios64: reboot if the SATA controller (JMB585 on PCIe) didn't come up at boot.
# At most 3 attempts in a row.
C=/var/lib/helios64-pcie-check.count
if [ "$(cat /sys/bus/pci/devices/0000:01:00.0/vendor 2>/dev/null)" = "0x197b" ]; then
	rm -f "$C"
	exit 0
fi
n=$(cat "$C" 2>/dev/null || echo 0)
if [ "$n" -ge 3 ]; then
	logger -t helios64-pcie-check "SATA controller still missing, giving up after $n reboots"
	exit 1
fi
echo $((n + 1)) > "$C"
logger -t helios64-pcie-check "SATA controller missing, reboot $((n + 1))/3"
systemctl reboot

/etc/systemd/system/helios64-pcie-check.service:

[Unit]
Description=Helios64: reboot if the SATA controller is missing
After=local-fs.target
Before=mdmonitor.service smbd.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/helios64-pcie-check

[Install]
WantedBy=multi-user.target

systemctl enable helios64-pcie-check, done.

Result: two rounds of 10 reboots each, 20/20 back on their own, all five disks every time. In between, 4–7 caught panics—some boots took one or two extra laps. 💪

Tip: journalctl --list-boots (the journal is persistent on Armbian) shows the panicked boots as gaps between boots. Way more reliable than the serial console on warm reboots, b/c the FTDI chip doesn’t always re-enumerate in time.

From OMV to plain Debian

With OMV gone, the NAS part is a handful of config files:

Samba. The share sections came over 1:1 from OMV’s generated smb.conf (create mask 0664, directory mask 0775, inherit acls, hide dot files, valid users…). [global] is slim: workgroup, standalone, fruit:aapl = yes, multicast dns register = yes. Plus wsdd2 (Windows) and avahi-daemon (Finder).

The nicest trick of the whole reinstall: take the SMB passwords along. I unpacked the old eMMC image onto the RAID, mounted its root partition read-only (losetup -r -o 16777216, mount -o ro,noload) and copied /var/lib/samba/private/passdb.tdb and secrets.tdb over. The clients reconnected w/ their stored credentials, no password prompt. 🤓

Tailscale serve: one HTTPS name for everything

Tailscale from the official repo. Before tailscale up I deleted the old node in the admin console and afterwards set its old IP again → name and IP 1:1.

Then tailscale serve does what the Apache vhosts + cert script did in 2022:

tailscale serve --bg --https=443 --set-path=/cockpit http://127.0.0.1:9090/cockpit

→ https://helios64.<tailnet>.ts.net/cockpit/ is the dashboard, w/ a valid Let’s Encrypt cert. In the LAN it’s reachable directly as well.

TIL: an nftables rule on tailscale0 does nothing for tailscale serve. tailscaled accepts those connections itself, in userspace, before netfilter ever sees them. Access control belongs in the tailnet policy (ACLs). You can check what actually applies on the node w/ tailscale debug netmap (→ PacketFilter).

Cockpit

Stock Debian, w/o the recommends: cockpit-ws cockpit-system cockpit-bridge cockpit-storaged. To run under a path behind tailscale serve, /etc/cockpit/cockpit.conf:

[WebService]
UrlRoot = /cockpit
Origins = https://helios64.<tailnet>.ts.net https://helios64:9090 https://helios64.local:9090
ProtocolHeader = X-Forwarded-Proto
AllowUnencrypted = true

Two things Cockpit complained about afterwards:

Careful: the Storage page can also delete RAIDs and volumes. Read only. 😬

Mail, SMART & Docker

Log noise you can ignore

On a healthy box, journalctl -p err still shows a few things, all from stock packages:

Bottom line

Boot in ~19 s (when it doesn’t take an extra lap 😉), SoC at ~41 °C, RAID [UUUUU]. Gone are OMV, Nextcloud, MariaDB, PHP and Apache. Instead it’s Samba, Cockpit and a few config files I actually understand. And the only Helios64-specific bits are one kernel parameter line and a tiny shell script.

The Helios64 lives on. 🤞