Two Raspberry Pi 5 run at my home. The main one, with 16 GB, carries seven applications in Docker containers: AEGIS, VIVA, ATLAS (which manages my bank accounts), myGastroBook, myGastroCellar, this site's audience dashboard, and one last one I keep to myself. The second, with 8 GB, serves as a backup target and spare machine. No application is exposed to the Internet: everything goes through a private Tailscale network, over HTTPS.
A dead switch port
It all started with a mundane wish. At the end of July, I just wanted up-to-date technical documentation for the two machines. That same afternoon, the main Pi disappeared: no SSH, no applications, neither on the local network nor on Tailscale. My first reaction was to suspect the changes made a little earlier. Wrong track: we had only touched the second Pi, give or take a container restart. The diagnosis, run from the second Pi turned observation post, eventually pointed to a dead switch port. Port changed at around 6:54 pm, back to normal. About two hours of downtime.
That day, one thought redirected everything: if it had been the SSD rather than the switch, the data would have been lost. Backups jumped ahead of everything else. No pre-production environment either. I develop on my workstation, let things settle, then they go to the server.
Three copies, one of them untouchable
The chain was built on 29 July, in one afternoon. Restic, three destinations every night at 3:30 am: the second Pi, the NAS, Backblaze B2 in Europe. For the off-site copy, the real concern was intrusion. Anyone who took over the main Pi would hold every key, and so the ability to erase the local copies. Hence Object Lock on the B2 bucket, 30 days, in Governance mode to keep control if I make a mistake. The price is real: automatic pruning becomes impossible, so I will do it by hand once or twice a year. Backblaze's interface does not let you create an ‘append-only’ key; immutability comes from the bucket. Restore tested the same day from all three destinations: databases intact, 240 recipes recovered. No test since.
One night in three
Now for the unpleasant part. While preparing this article, I extracted the end-of-backup lines from the log, night by night, since the very first. 77 nights from 30 July to 14 October. The backup ran every night, without exception. But 26 of those nights ended in error. One in three.
Most often, the cause was the same. After a restart, the NAS share would not remount, and nothing retried the mount. The script did exactly what I had asked of it: skip the missing destination, carry on with the others, exit with code 1. Twenty-three nights with the other two copies made. One night, 20 September, with only the off-site copy. And two nights, 23 August and 30 August, with no successful copy at all, the Pi cut off from the network, very probably both times. The tolerance that protected the other destinations is precisely what hid the failure. Nobody reads a log.
The September episode, 14 nights, I found by chance on 23 September while adding ATLAS to the backup. The August one, 11 nights from 13 to 23 August, I discovered while rereading the log to write this piece. It was in no documentation. No data was lost, simply because I never needed to restore. That is luck, not design.
The fix of 23 September comes down to one mount option: the NAS mounts on first access, so at backup time, whatever the boot order. 21 clean nights in a row since. What is still missing is the alert. It has been on my list since September. An error code in a file nobody opens is not an alert.
What held
Meanwhile, the rest held up fairly well. A Wi-Fi access point that switched itself back into router mode, turning the two Pis into an isolated island for a whole night. An automatic Tailscale update that demanded a fresh authorisation and made every app unreachable; automatic updates are now switched off. Seven hours without Internet on 28 September, 5,867 network errors in the log, fixed by a cold restart. In none of these incidents were the Pis' hardware or data affected.
On 8 October, Malta lost power for about two and a half hours. Pis, switch, router, NAS: everything went dark at once. When it came back, the Pi was ready in 25 seconds, the containers restarted on their own, and Tailscale reconnected as soon as the router had finished booting. Applications reachable about three minutes after the power returned, without me touching anything. It is not a feat; it is the sum of the earlier fixes. Only the NAS did not come back: a ‘start after power loss’ option, off by default, now switched on. And a reading trap along the way: the Pi has no clock battery, and its log showed 8:01 am when it was actually 10:05 am. Read too quickly, the outage would have ‘lasted a minute’.
A network watchdog tests DNS and an outbound connection every 5 minutes, restarts the services after two failures and reboots the machine after four, three times a day at most. Twelve days in simulation before I switched it to live mode on 10 October. It has had nothing to do yet. It played no part on 8 October: it was in simulation, and the connection was back before its threshold. A UPS is on order, not installed.
The alert first
What I would do differently: the alert before the third destination. I spent an afternoon building a chain of three encrypted copies with off-site immutability, then eight weeks before I saw the problem, and eleven before I grasped its extent: one night in three was ending in error. A message whenever the exit code is not zero would have cost ten minutes. I would also rerun the restore test at a fixed interval rather than by feel: the only one dates back to 29 July. The rest of the figures are drama-free. A full cycle takes 27 seconds, the repository on the NAS weighs 431 MiB for 13 retained backups, the 1 TB SSD is 4% full, and both boards run at around 50 °C.