Hacker News Reader

I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our servers and updating. As well as an onion roll-out process, with exponentially sized groups prioritized by distance from our office, and risk of customer loss due to downtime.

I think this all is a fascinating problem. I'm running a digital signage service for the Raspberry Pi (see profile) and also offer OTA updates. I've initially built my OTA process around the same idea: Keep a bare minimum bootable kernel and minimal userland that is always able to connect to the update servers. In case of a broken update I would simply rely on this part to fetch a fixed version. Unfortunately it turned out that most updates I did also updated the kernel (for new features/bug fixes) and it felt like I superfluously split my system into two parts that both need maintenance and testing. I've since changed to an A/B booting system that automatically falls back to the previous version if it isn't able to reach the servers within a few minutes. It feels way more unified and all I need for it to fall back to the previous version is a working kernel (which, on the Pi, is easy to test), the busybox shell and the first couple of lines of my init script. Everything else might break and the system will revert. It wasn't easy to reach this point but thinking about OTA updates doesn't feel so scary now.

we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again

Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)?

I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.

Userland. Basically during upgrades we installed a shell script in /etc/init.d:

  sleep 1200
  if ping -c 5 central-services.anybots.com; then
    rm <this file>
  else
    cd /home/dist && git checkout STABLE && make install
    reboot
  fi

(We used git to distribute binaries, so we could checkout and install a previous version easily).

This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.

An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.

I believe GRUB can be told to fallback to a safe mode.

I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.

This will be the time where setup HW watch dog should help.

After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.

If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.

As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.

Hmm, you can set a watchdog before rebooting? I assumed that it would be cleared automatically.

U-Boot has a mechanism for offering fallback by counting boots (and saving that to local EEPROM).

If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.

It's fun to poke around in these embeded firmwares. Most of them are strung together with shell scripts for everything but the actual application, this kind of stuff is generally just another script.

Gradual rollout also helps (whenever possible), especially with stuff that has to work on many hardware platforms etc, so you can roll out only a few to a new hardware platform, see that it works, then roll out the rest.

All hardware devices that allow firmware updates should have two things:

1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)

2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.

Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.

our app updates via the app itself. This gets regularly bricked but we have a Cronjob where we can put bash commands in a .txt file and upload that to our server, and devices will execute it regularly. This has always saved the day but apparently the past developers before me used to drive around with a USB stick on a regular basis, to re-image bricked devices.

I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are

Your end devices execute arbitrary code put in a specific url on your website? I assume this is a purely in-house device and not one in the hands of consumers?

Executing arbitrary code put at a specific URL on a website is exactly how software updates work. As long as it's properly authenticated, it's fine.

Normally there is a bit of code signing involved.

Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)

If someone is not competent enough to keep their domain renewed, what's to say they're competent enough to keep private keys secured, and not lose or leak them. Then again I saw something on here recently about someone hijacking the whole .io TLD, so what you're saying is a real issue.

Slightly, but not exactly. Any remote code execution could lead to "example.com/run_this.txt" being filled with "evil_hacker.sh". However if "run_this.txt" were somehow signed and verified prior to downloading (with signing keys kept off the distribution box) then it's much safer.

Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.

Our main issue is if someone tampers with content stored on the server, which gets copied to devices. For example someone replaces a JPG with the wrong JPG that is something offensive.

If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry.

To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.

At RentingLock.com we solved such issue by... not using internet connection, thus not relaying on something that might fail anytime.