You can never have too many backups
theregister.com
theregister.com
The Chief Engineer and crew went to inspect another factory for inspiration. There was a direct flight to the city, but by corporate policy they had to use the preferred airline, requiring a transfer. The Chief Engineer didn't want to lug his heavy laptop around, so he checked it in, with the thumbdrive still plugged in.
Chief Engineer and crew arrived successfully at the factory. His suitcase did not. Enquiries with the airline told them the suitcase had been shipped to the wrong place and would be returned very soon. And that was the last they or the airline ever saw of it. There was no central server, no versioning, no backups -- the plans were now gone.
Fortunately, since this was a Japanese company, the plans were printed out weekly for review at the status update meeting. So the plans were manually reconstructed from paper printouts, page by page, line by line, character by character, and one of the many, many people typing it in was the friend who told me this story. Unsurprisingly, they no longer work at Megacorp.
Another good one is Maersk. When they suffered a ransomware attack that deleted all access to all backups/replication servers, the only surviving backup was a replication server that happened to be offline during the attack, because it was situated in a country with very unstable internet and power supply.
> the plans were printed out weekly for review at the status update meeting.
This is so quintessentially Japanese. Have a single point of failure in a rigid chain of command structure, but bake in a ton of paper redundancy. Nonetheless, I bet the lost laptop was an extremely advanced model.
In the early 2000s, my dad worked with some Japanese collaborators. They brought over some impossibly thin laptops that seemed a decade ahead of what you could get in the US at the time…and proceeded to use them to send faxes.
In one company, I used GitLab Board, and had special tags for the recurring and background tasks, which were sticky in Kanban columns right below the generally one-time development tasks. The deadline dates for recurring tasks kept changing to the next time the task had to be done. The Issue description had the concise immediate instructions for the steps of the task (with a link to the wiki for related and background info). And every time you did the task, you added a very quick Comment with any notes, and bumped the due date.
It was very lightweight, and fit the simplicity I was advocating (e.g., the GitLab Kanban board says what everyone is doing, all info goes into GitLab in one form or another, minimal other tools and silos to juggle, etc.).
Does it have specific meaning in software engineering? Isn't Kanban just billboard in Japanese?
The "Blocked" is important to highlight on the board: when someone can't do their thing, no sense floundering quietly -- highlight that, see who can unblock it, and what anyone else can do in the meantime.
"Abandoned" is another one that I think is important, but might be unusual. If the task no longer makes sense, stop immediately, move it to Abandoned, and move on to a more worthwhile task. Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore. Later, we can reflect on what info we had, when, and what we can learn from that.
(I usually really-really want a Gantt model as well, on the same database that the board view is coming from, but the tools aren't quite there yet.)
Even on personal projects I try to not remove TODO's or tasks without writing why it is no longer a goal. The amount of knowledge that is thrown away in many organizations or by people by discarding tasks without saying why is astounding, the why for discarding it can be as productive as a task completed.
Example: test our database backup restore process. Everyone hops on; leader logs into the cloud provider; confirms that the backup strategy is what we expect (which is documented in the description of the meeting invite); confirms that database backups exist in line with that policy; confirms they're of a reasonably expected size; we click one for staging -> restore to a new deployment -> wait for the cloud provider to set everything up, use that 5 minutes to talk with one another about stuff -> set up the database username/password; log in to AWS Secrets Manager -> update credentials -> roll the deployment to pull in the new credentials; confirm everything looks good; and tear the old (again: staging) deployment down. A second person on the call is tasked with documenting everything we did, taking screenshots of the screen share, and getting it all into a wiki. For us, it takes less than a half hour, do it every quarter or so, usually scheduled for a Friday afternoon where we can also talk about weekend plans and enjoy some drinks.
Eyeballing dashboards is a harder one; a similar thing we do, similar to the last example, is more of a Performance Review, which is bi-weekly. We've got a collection of dashboards, database index metrics, etc; we all get together for a half hour and review all of them similarly. If anything looks unacceptable, we generally backlog two tickets: one to adjust our Slack/Datadog alerts for it so the next time we don't find out during the review, and another to remediate.
We ended up being saved by our traditionally "crappy" local dev process; me and the other dev had copies of the prod db on our development machines. And, naturally, we rarely updated them; the last time I had sync'ed it down was, conveniently, about four weeks ago. So they had some data loss between that "backup" and the oopsie-whoopsie, but far less major than their issues with the current state.
The most interesting thing to really come out of that was a set of tooling to precision restore an individual customer's data from a backup of this multi-tenanted database. That was something I see as pretty rare for engineers to consider in their backup strategy; you may have the backup, it may be easily restored, but it will take the world with it and maybe you don't want the whole world. We ended up using that a half dozen or more times over the next few years... against a far more comprehensive set of backups of course. Yeah, maybe it makes sense to store a history of every change made to every row; but for our needs it made a lot more sense to leverage backups and some glue scripts.
And some of those restores should happen in the application layer not the db backup.
0% has been because of hardware failures.
This is why I love that PostgreSQL backup files are plain text files. It is so easy to extract a single table (or parts of a table) using a simple perl/python/whatever script.
There is even a tower-of-Hanoi tape rotation scheme:
https://www.techtarget.com/searchdatabackup/tip/An-introduct...
* https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_o...
Another comment mentioned an offline backup. And I agree. But it's also easy to go from having a month old offline backup to a year old to...
One that seems close is ForeFlight (demo: https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s), but it's just one feature of a much larger app, is mobile only, and obviously targeted at aviation.
I'm looking for similar functionality but more IT-oriented (or at least generic, not referring to airplane models, etc.) and preferably with an offline desktop "client" (it is especially critical that you have your checklist library available if the whole corporate network is down, for example) or, barring that, at least a desktop-friendly web UI.
Honestly, if I knew there was a market for this, I would build it. It's either a great opportunity or the market isn't big enough.
It's not a software engineering challenge. The minimal features are textbook basic CRUD. The essential UI will be well-supported by any framework. Even the challenges when you branch out are solved problems. Device sync, sharing, backups, etc all have some off-the shelf solution or are solved in another space you can look at.
It's a product design / UX / whatever you want to call it problem. Within the set of choices that are easily available to you in the software engineering world, you have to make the right ones! Most people won't naturally make good checklists. How do you help them? One of the biggest (maybe single biggest) issues with checklist is using them at all. How do you get compliance up? How do you handle "skipping" items? Do you go with read/do or do/confirm? How do you keep checklists up-to-date (critical for emergency checklists). It would be very easy to get this stuff wrong.
It also hits a weird spot in the market. What would people pay for this? I don't think you hit enough users for "free" and then my guess is it's a really weird curve with some people paying $1 and then a minority paying like a lot ($100?). I don't know, that's a total guess. Just bringing this up to cover "why isn't a company already doing this?".
Not to be discouraging! Just don't take it as a software engineering challenge. If you're experienced and working on your home turf, it will be easy. The hard thing will be making the right easy thing. Best way to do that is to solve your own problem btw. Maybe start with a "I crashed the server" checklist or something.
If anyone wants to take a stab, the essay that the book "The Checklist Manifesto" is based on would be a good start. The book itself is a bit verbose, but would be worth it if you get serious. I'd also chat for 30 minutes or answer random questions for anyone who's going to make something. My email is my last name at gmail.
Perfect for do-check lists.
[0]: https://apps.apple.com/de/app/daily-checklist-checkyourlist/...
Throw in collaboration features like sharing a single checklist ("are we bringing everything we need for the roadtrip? mark items that you've packed" "someone please buy milk, or else I will if it's still on the list when I stop at the grocery store tonight," "has everyone grabbed a task for the next sprint?"), or delegating items to other people ("for tomorrow's presentation, make sure someone has set up the projector"), and it could add the magic that I felt back in 2009 when I realized how amazing git became once GitHub was layered on top of it.
The reusable-list idea works for me when I'm traveling. I have a list of stuff I need to do when I leave the house -- turn off the thermostat, make sure the plants are watered, etc. -- and I find it annoying to have to uncheck the old list's items in Google Keep. I'd rather keep the old checked lists and instantiate new ones. The list itself evolves over time, so having the old completed (or not-quite-completed) lists is interesting.
Also see this recent item https://news.ycombinator.com/item?id=32552782 proposing a spec for to-do lists.
1. Type it into a file.
2. Print it onto a piece of paper
3. Put a mark by each item with a pen as you finish it.
I have a small sticky note pad on my desk. When we have a meeting about features, I jot down my todos. In some cases, I may do a screengrab, or submit a bug report (to myself), but, most times, it’s the PostIt.
When I’m done with a task, I crumple the note, and drag it into the trash can.
The number of times I've missed a step on a paper checklist because I was on autopilot is ... non-zero
You can create an issue with a checklist in it by pasting in the following:
- [ ] Item one
- [ ] Item two
And if you want to duplicate a checklist, you can copy and paste in a full block of markdown.
This should also lead to the question of how up-to-date your backups need to be and is your solution setup for that (is it 1s, 1m, 1h, 1d, etc.)? Define an actual SLA for how much data you can loose and test for that.
I have no idea why major drive users don't demand this from the makers.
At the SOHO server level, this might be useful. But the alternatives are already available: unmount the disk, or unmount and unplug it, or unmount, unplug, and carry over to the safe, or unmount, unplug, and carry to the backup site...
At the datacenter level, nobody has time to physically touch things.
Besides, for the bozos, ship the drives with the write-enable turned on. They won't notice the difference.
The systems that send backups to that S3 bucket don't have permissions to modify previous versions. If one or all of those systems are compromised then they can't overwrite previous backups because the original data will still exist in old versions.
Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this.
I am confused by these sentences. Sometimes the law does it, but you're not aware of a law that does it?
Policies are of course a different thing, and they may or may not apply to you, from industry regulations or cyber security insurance or just company or trade group policies that might apply.
If data is on backup tapes you seem to be able to inform them of how long your retention policies are.
If you decide to go down this route, you should bear in mind that other supervisory authorities might be stricter and that you must be able to demonstrate that it’s impractical to delete backup data.
This will require, at the very least, a risk assessment, business impact assessment and data protection impact assessment.
Working in security I have been advising clients to make sure that they consider their obligations for retention periods. Not just so that you release user information you no longer need at the point you no longer need it, but also to reduce litigation if there were to be a data breach.
Until a court or politician says otherwise, "Deleting data" means inaccesible, not physically destroyed.
AFAIK GDPR, CCPA don't specify deletion means physical destruction.
You can kill one or one-thousand backups with the same effort.
- write all commands, ever entered to any shell, down. Of course, this also includes the procedure to create backups
- do not treat all data equal. There's 'fast' (but small) data that needs more backups more often and there is 'slow' (but large) data that is infrequently updated, thus less often backed up.
- 3-2-1 rule is a minimal setup; extend it wherever possible. Also, use 'migration periods' to replace one backup system with another; during this period, you use both backup systems until you gained enough trust in the new one to deprecate the old one. This should include actually testing restoring systems from backups in VMs at least once.
My private 3-2-1+:
- immediate sync of files to my internal nextcloud server, hosted offsite. That means any files exist at least once in to different locations
- daily ZFS automatic snapshots at the server side and deprecation of old ones
- weekly, my server starts another Odroid CS2 at another offsite location and rsync's any changes; the Odroid is shutdown afterwards
- yearly, I attach a 16TB usb drive and send ZFS snapshots.
- in addition to these 'basic' 3-2-1, the '+' means: daily, Nextcloud's document folder is duplicated to a different ZFS dataset on the server which is not actively write-referenced in any VM; Also, for some very important files, I keep more versions: password manager, dotfiles, config files etc. are a) versioned in my private Gitlab, also b) Nextcloud will keep versions and c) some of these files are synced to my phone; I also print out the most precious keys yearly and store those at a 'secure' location
- here's a special tip: Backup your OTP codes! I realized this recently: Google Authenticator will not create backups. You can migrate your OTP codes to Aegis Authenticator [1], which allows you to create automatic, encrypted backups that can be automatically synced using Nextcloud
[1]: https://f-droid.org/en/packages/com.beemdevelopment.aegis/
Sometimes, just a couple of pages of documents, or a setup script, is worth weeks of work, ressources, money, etc. And it's nothing compare to the costs of creating multiple backups.
If your RTO is very short, then full backups with fast restores are best.
3 recent copies of important data, in 2 different media formats, with 1 copy off-site
Just curious - how often do you test this?
I write an MD5 checksum for each file so they can be verified at any time in the future.
Big media files are a bit trickier