Hacker News Reader

Comment by stdclass | Hacker News Reader

That one bit me really hard today.

We're using hundreds of raspberry's as digital signage devices and before deploying them I checked a few of them and found out that sometimes the source files of the app were empty.

After a lot of headscratching and googling we boiled it down to a write delay to the SD card.

If you do echo "test" > test.txt and power off the device like 5 sec. after, and power it on again, there was a file called test.txt with 0 bytes.

I'm still baffled how something like this can be a common problem especially since raspberry pi usage is so widespread

View on HN

I operate a digital signage service (https://info-beamer.com). It's using a custom built Linux OS that is highly optimized to deal with file system corruption and unexpected reboots:

* The OS is is always read-only. While this doesn't prevent all possible corruption scenarios, it's pretty close and you can be quite sure the system always boots.

* System updates are implemented as A/B. So even during updates, the OS the system booted from isn't touched.

* All content data is on a third partition and together with the OS files is constantly checked for indications of corruption.

* If the OS detects any bit error on the OS files, it is reinstalled (using the A/B mechanism above)

* If there's a bit error on any of the content files, they are deleted and downloaded again

* If the content filesystem is broken beyond repair the system reformats everything and restores everything automatically by redownloading

* Various watchdogs and checks constantly monitor system health and notify you on the dashboard.

The service has multi-million hours of service by now and the only case of filesystem corruption was due to a really crappy SD card. When done properly, the Pi can be reliable. That's also the reason why there is no way to properly shut down the system. Just unplugging power shouldn't be an unexpected circumstance but something you optimize for. Starting with a normal Raspbian and expecting that to work is a bad idea.

Is your updater self rolled or based on something existing like mender.io?

Self rolled. Built in Python.

I have a use-case in mind where the power might disappear suddenly and often. Is there a different SBC board and OS which is optimal for this use-case?

You might want to look at some of the 'Pi UPS' systems which have been developed. Once they indicate the power has failed you can start a graceful shut down. One example can be found here:

https://pi.gate.ac.uk/pages/mopi.html

In my opinion it seems the Pi was too good at fast and cheap prototyping and people started shipping products based on it, overlooking the fact that the hobbyist and unreliable nature of the board wouldn't necessarily scale as hoped.

It always looked to me like the Pi was supposed to be used some steps before hardware integration and shipping.

I am glad products are built on it but I was a bit worried when I had to evaluate a Pi based solution for digital signaling last year. Maintenance was a key factor in the proposals.

When you say you powered it off, was that a safe shutdown or did you just pull the power?

Most file system drivers delay writes, I suspect some variant of your test (if it was unsafe shutdown) could be reproduced on almost any system.

It was an unsafe shutdown, meaning I just pulled the plug. But it was at least several seconds after saving it. I've never seen behaviour like this before.

Most sd cards are formatted fat32, and do not have a virtual filesystem cache and write scheduler buffering work in ram for them.

Also raspis do NOT like big fast SD cards with tight tolerances. I've had the best luck with old class 4 cards. Class 10 cards don't last nearly as long with regular power cycling.

Did you ever run linux with ext2 (not ext3 or ext4) on a desktop computer in the early days? It used to fail spectacularly on unclean shutdowns. IIRC linux used to delay/buffer writes up to a minute or so. (Maybe it still does?)

You can force a partition to be mounted "synchronously", where writes aren't buffered in RAM and then flushed later -- every write blocks until it is written to disk.

The options are the third column in your /etc/fstab file. For example, if you had "errors=remount-ro", you'd change it to "errors=remount-ro,sync".

(Or is the Raspi problem actually in hardware?)

Yes, in fact you'll find lots of source code out there that will write test.txt.tmp and then close and rename it to test.txt, exactly because this is an issue in all filesystems. From the sounds of it the time window during which RPi is vulnerable is longer than usual. A careful choice of filesystem journaling/integrity flags would probably help.

The fact that there's a 0-byte file instead of a completely broken file system is already the result of a journal. You can reliably observe this with ext4.

On unix systems you can always do a sync(2) to commit the buffers to disk.

https://linux.die.net/man/2/sync

or sync 3 times and then pull the plug

https://utcc.utoronto.ca/~cks/space/blog/unix/TheLegendOfSyn...

I would never SD card boot a Pi in a production environment unless your software can handle a read only file system, the card corruption issues are so widely documented at this point. Thankfully USB boot volumes are supported now.

We used to type sync; sync; halt. Funny how these things come back around.

Reply on news.ycombinator.com