Hacker News Reader

What was interesting to me about this was that they had decided not to enforce a deletion policy on /wrk, because they had so much space and the filesystem hadn't ever failed. But a rolling deletion policy would have gone a long way to containing the damage by encouraging the users to move their data to a filesystem optimized for reliability instead of availability. Still, I appreciate the heroics involved in restoring the data.

View on HN

Author here. We have had an automated deletion policy on our previous filesystems but opted out this time: There are users that have temporary files that they want to persist on the /wrk and we have plenty of capacity. We definitely learned our lesson, though. :)

Better solution for the future: client side scripts that push those files back after every purge.

The (inevitable?) consequence of deletion policies that typically delete based on mtime, tend to be users putting touch scripts in cron, making the metadata servers even more of a bottleneck than they already are. Been there, done that.

Perhaps the solution is some netflix-like chaos monkey that randomly deletes files..;) Or for each user over its soft quota, delete the oldest files until under the quota. Or something like that..

Reply on news.ycombinator.com