Hacker News Reader

Comment by LinuxBender | Hacker News Reader

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

View on HN

How did not doing these things sink the company?

The problems stated are bad, but did something actually happen as a result, like a rogue script deleting things and the company no longer in business?

I think the implication of the sum of the things listed is that an old backup system overwrote/mangled newer backups because nobody turned it off and the two systems were targeting the same storage.

> but not implementing anti-tampering on local and remote backups i.e. protection from root.

How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

1st level of protection is by keeping your live production data & backup data in separate accounts.

(If 1 account gets compromised the other is still available)

Perhaps 2nd level is by using cloud specific features - such as AWS S3 Object locks (even owner can't delete the object till the retention period).

my 2cents

Backups sent to a remote system should be append only so that if a machine is compromised, a malicious actor cannot delete or corrupt previous backups.

An easy way is to stream the backups to tape, and have someone swap the tapes. Typically people who do this take the recorded tapes offsite, but to prevent against an electronic attack it’s adequate to have the removed tapes even in the same room, just not in the tape drive.

Separate storage of file hashes would make tampering detectable. Actually preventing root modifying a file probably has to be done at the hardware level.

Heard of a case where an admin was bribed by competition to delete everything.

Maybe something like the write protect switch on LTO tape cartridges, so that if compromised you never overwrite the tape when trying to restore.

Reply on news.ycombinator.com