VUW Accidentally Wipes Desktop Computers
critic.co.nz
critic.co.nz
Those who do backups and those who will...
Seriously, I can't imaging doing something like research or writing PhD thesis and not having at least one off-site backup.
Just get yourself couple of pendrives and rotate them somewhere else than where you live and work.
I personally have more data. For my backup needs I use 3 4TB drives, a USB HDD docking station and a script to detect when one of the drives is inserted and make backup of data from all places in my home that I need backed up.
I have one drive inserted in the station (once inserted, it will attempt one backup per day). One standing by or in transit and one with the family.
At any point in time at least one of those drives is safe with my family which I visit regularly. When I drive there I yank the backup drive from the station, immediately replace it with the standby drive and take the fresh backup with me. When I am going back home, I take the one drive that was previously kept by my family to use it as a standby.
It is not complicated, you can do the same with a pendrive.
There is no paid software involved, I wrote the script over the years but it is simple and has no extra features. It only detects when the drive is inserted (each has a file with info), makes full/incremental backups (full backup automatically after inserted, incremental every night) and removes old backups and linked increments as needed. Restore is manual as files are just tar.
That's a huge red flag when looking at the institution.
More generally: make sure you DR plan, even the simple "I have a backup" one that most people use, involves a second provider (which in some circumstances is your own local resources).
Even if the people keeping your data safe on a daily basis are entirely competent, bad things can happen and with centralised control they can happen to many people at once. Even if all the data can be restored you may not be a priority so if you need urgent access to the data having on off-provider copy you can take it from is a godsend.
Of course in a commercial setting or some university/research settings, maintaining an off-provider copy of any data may be a _massive_ policy breach. Or even a legal problem is it involves personal or other sensitive data.
They very likely do. On the M: and Z: drive.
As far as I can see this is just IT saying “if you stored your data somewhere you shouldn’t have, it’s now gone”.
But of course they can’t say that, so instead they have to help people recover from their mistake.
Or maybe the mistake was thinking people would actually do what you told them instead of whatever was more convenient.
You're assuming people are aware of where they — or their niche specialty software packages — store their data. I'm not even sure I'd know where e.g. some biotech-DNA-analysis-package stores its data.
Recently I've been given incredible abuse by my ex who accused me I took her external HDD when I moved out. I actually considered I did and scoured all my boxes and belongings and even invited her over to search herself. Meanwhile, I asked her if she had any backups.
Needless to say the abuse tripled. I never found the drive.
Which is actually not safe for cold storage in any way, shape, or form. Cheap flash storage devices are optimized for minimum cost per bit stored, not long-term data retention.
Why didn't IT use their powers to deploy a backup program before they deployed a deletion program?
Thought expermiment: what wuodt happen if a student accidently deleted school's files, instead of the reverse? Criminal penalties? Civil penalties?
Were these personal laptops, or university provided? The fact they have H: and M: drives mounted suggests the latter.
The trick is to look at the most productive research groups, in terms of funding and output, and see if they use the resources from the IT department or not.
I don't believe I am susceptible to a ransomware attack, though. All my machines including laptop are Linux and most of the software that contacts outside world (including browser, zoom, etc.) are running within dedicated containers or VMs.
Now, this does not make it safe against targeted attacks, but I did not plan it to stand a chance against really determined attacker. I just think no generic ransomware will expect to break out of container or VM.
Those are above and beyond what I would personally need to avoid the threat of ransomware. If you need improvements, visit your family more often - win win :)
Pendrive/USB ports may be locked. But you can always zip & email the very critical doc/xls files. This was you can also have 'versioning'.
Something that I assume people do, is journaling. So even if you DID lose 1-2-3 months worth of work (in academia) you can read your (handwritten) notes and will save a lot of time from remembering the key points, key resources, etc.
Either way, whoever wanted to wipe a couple of computers and managed to push the script to the full AD group.. I got no words. Article doesn't mention what OS they were sitting on, but I will go ahead and assume it's Windows, on an Active Directory, and the admin pushed the new 'think' to the wrong OU.
And yet people do this. Last year I read a newspaper article of a guy who worked 2 years on his PhD thesis but then forgot his laptop in a bus and now those 2 years of work are gone forever.
The real problem here is not just backups, though. It's a severe ignorance of technology and its limitations that leads to such carelessness. The old saying about advanced tech seeming like magic should be a warning. No sane person goes out into the rain without an umbrella thinking they just won't get wet because yesterday they didn't, as it's painfully obvious that water is wet. Yet when someone stores everything on a single point of failure they don't lose a second of sleep of what could go wrong, it's simply not in their perceived range of possibilities that e.g. a hard drive could fail.
In this case the University should still provide a good way of you backing everything up of course, and you can still back up anything which isn't confidential.
We seem to instinctively err on the side of avoiding any short term cost, no matter how potentially grave the consequences in the future if the probability of those becomes low or obfuscated.
You can see this not just in backups, but in lifestyle choices, business decisions, construction, natural resources, financial speculation ...
Any situation where you can say "it does not happen often, but if it does the results will be catastrophic", you can bet that the risks will be unhedged.
The workers are "required" to buy their own computers anyway, so it's not like the university has any say over how the computers are managed.
Everybody has a story about IT ruining their computer by installing some sort of unworkable security software. There's also a familiar story of IT coming in and wiping out a computer that runs some ancient scientific instrument that isn't supported by its vendor any more. And the vendor kindly sending them an unofficial copy of the ancient software because, well, because it's happened before.
To be fair, dealing with IT at a university must be a nightmare. My workplace takes the approach of providing all work computers, of a specific brand, and retiring the old ones before they turn into a support headache.
Always mount a scratch monkey.
I know someone who does this. The answer to this immovable mountain of a problem is an unstoppable force -- throw students at the problem.
It worked fairly well as the NAT router was presenting a WAN MAC address from a device that had already been "approved". Main reason was to get local 10Gbe available when the official LAN was still 100 Mbit (!!), and to enable new devices to be connected to the network without going through whatever official process was in place.
I'd never risk letting official IT people image a system - I've had to send people with laptops that were missing graphics drivers to "shadow IT support" to get their system properly installed by someone that knows how to clean install a system and install the necessary drivers.
Our "shadow IT" ended up so significant that we ended up able to directly request firewall changes and just send in the rule lines (in the correct format for the edge firewalls used) and they would be applied for us within the hour...
The end result is that nobody ever lets "main IT" near anything. Most people who have important work on the go used shadow IT. People who know what they're doing then run their own personal Shadow IT (but would use the real shadow IT if they had any issues like needing to reinstall a system, as that way it won't get connected to the main IT domain...)
You are absolutely correct - it's an absolute nightmare. Many university networks originate from the era of the flat LAN, where every system got a routable public IP, reachable via the internet. There are people running all kinds of servers and services and jobs across the network. None of them are documented, some might be relied upon by important internal or external things...
There's no standardisation of hardware generally. If you're lucky there's some managed desktops using standard configurations for computer labs used by students, and office administrative staff. Don't expect to be able to tell professors what they'll use - they're buying computers on their research grants for themselves and their teams. And their own NAS servers and everything. Heck the "shadow IT with a help-desk" was even offering storage and VMs on their shadow infrastructure.
I've seen people trying to use Windows XP on the network because some incredibly expensive piece of equipment needs it, and they would like to be able to browse the internet while they wait on it to do its work (!) Fortunately, shadow IT were on the lookout for that, and helped them set up a second computer on a supported OS, and get the XP box offline.
No "shadow IT," this was completely official and above board.
Everything that could be was NetBSD.
Compsci had its own network range within the university's /16.
I remember the ease of dealing with them, compared to the battle to get one network port livened so the computer club could expand its nascent WiFi network to the quad. Heck, within compsci land, the computer club managed to get a server with a static IP and even got it exposed to the internet.
Blaming the victims is not on. Assuming the victims should have done a better job than paid and trained professionals is not on. Assuming the victims are all CS majors is not on.
"Users should know better!" you scream, into a world where currently almost everything stored locally is already safely shuffled to the cloud to be retained even if your desktop dies (Windows 10 OneDrive, and Apple iCloud, and other solutions like Dropbox, Box). Except, obviously, at this university.
A large part of VUW's research focus covers law and politics [1], so this statement rings quite true. I would not expect that all researchers even in topics adjacent to IT such as physics and chemistry would understand prior to this event that IT could remote wipe their computer, where the limits of local and network storage lie, or what data storage locations are apparently their personal responsibility to back up.
I now apply this "What's the recovery from 95th percentile occurrence?" thinking to lots of new things.
Perhaps it should be culture in many departments to intentionally harm someone at small scale to insulate them from large scale. An inoculation of sorts.
I think if I have children, perhaps it will be worth having an intentional disk failure on some low cost thing to teach the importance of backups. Maybe have the hardware lose something like a save game in a game they've barely started. Hard not to make it some sort of pathological fear while still instilling the value.
One fictional, watching PCU and having a "haha that's funny oh crap" realization. One personal, when my HDD (in a circa 2000 PC) failed catastrophically (came to a grinding halt), fortunately I didn't have anything critically important on it at the time. One collective experience, a recreation of the PCU incident at a programming contest in a makeshift lab. Fortunately my computer was fine, and my team came out ok (3rd or 4th, memory fuzzy now) thanks to that and my then habitual use of C-s. My teammates' computers were also fine, but apparently they thought they only needed to hit C-s every 30+ minutes and lost a lot of work.
Of course, I also grew up with more unreliable systems (ever had someone put a magnet near your floppy disks?) so I'd already internalized a lot of the idea of being conscientious of my backups, but it still took a couple near catastrophic experiences to turn it into a true habit and routine.
For my part, I, too, grew up with unreliable hardware¹ and each escape actually enhanced deviance. Every time I got away with not losing data because I found it somewhere else or found it unnecessary made me act closer to the wire. I'd risk more each time.
One time I even saw a friend lose his sister's paper (causing her to have to rewrite a significant amount) and it did nothing. I even lost data due to power loss on XFS². Did nothing.
Then one day I just started getting nervous about data loss and it's hard for me to determine what it was. Now everything is backed up, but I wish I knew what it was.
It's the fact that I had this strange reaction to encountering data loss that makes me hesitant to recommend the strategy I espoused in GP.
¹ all the same floppies, low quality CD-RWs, old spinning disks, and to compound all this I had poor quality power that would intermittently fail
² the famous file truncation, if you were around back when Reiser was FS dev and not murderer
/s
I don't really know Windows enterprise distributed management, but from my experience at a former employer my impression is that some directories on a Windows computer end up "managed" and others are purely local.
We were always told to only put important things in the certain "managed" areas, because they would only be backed up from those directories (including for compliance purposes). The files in "managed" directories would also magically show up when you logged into your login on any managed Windows desktop, the idea was that the local HD was disposable.
My guess is that, ironically, those were exactly the directories from which you would have lost your files in this case.
One would think they would have backups of everything, but if they're telling people not to log in to the computer to potentially avoid overwriting deleted bytes, sounds like maybe not?
In fairness, it's madness not to make any backups of important data for an entire year. Hard drives die all the time.
Some students and staff likely use specialised (often very very expensive) software that locally process gigabytes of data, e.g. in the fields of biology and chemistry. These datasets or simulations will be in proprietary formats buried deep in some folder somewhere or in an even more inaccessible database hidden from the user. The PCs in question could also be physically tied to hardware systems in a lab. They're not going to expect that machine to be randomly wiped while working on the data, possibly for weeks at a time. They don't know where the data is. At best they might export the data (a feature which probably doesn't exist for most programs) and archive it at the end of their project. This IT screwup will hurt them a lot.
If film students can do this (and they do; I knew some in school), I think a scientist can be expected to.
Start with an elderly IT system where your network drive quota is about 5% the size of your laptop's hard drive.
Then imagine someone in (for example) a psychology department where they've got ethical obligations towards data security for things like survey results and medical data - so USB hard drives are not officially endorsed.
Then imagine the department is fragmented into labs and groups and people pursuing their own lines of research, so while some research groups in the department have lucked into having someone IT literate enough to get hold of a NAS or sort out backblaze accounts for everyone, others haven't.
During crunch weeks it's not uncommon for students to log 16+ hours a day on the computer, keeping up with heavy workloads.
Anyone who claims that it's okay for a specialized industry professional to be ignorant about (very) basic computer use is absolutely wrong; it applies to all fields either within the professional aspect, or at the very least during the learning process while slogging through college and the attached social networking.
We chastise the elderly for being unwilling or unable to learn to use and accept mobile phones, meanwhile current-generation students from non-cs fields are having trouble with the very basics of computer use, made all the worse by the fact that most modern colleges are by-and-large paperless and entirely reliant on the use of personal computers?
Great.
Ever since then I now backup and image the entire HDD and I have almost literally terrabytes of extra OS images "just in case" I forgot or lost something. Hasn't really happened yet. Except I still can't find my Bitcoin from 2010. Doh.
Yes, do backups. In the worst case, keep emailing your thesis to your gmail account (if it is not confidential).
I backup here: https://www.opendrive.com/
I have my data on two HHDs. On an LUKS encrypted ext4 FS, one a non ecrypted ext4 with a file based gocryptfs encryption. I mirrow the last one into opendrive. I tried several encrypted cloud backups that I can totally control. This was the only setup that worked reliable.
Its hard to imagine how they messed up so badly. Lovely that they gave themselves admin write access to user's devices but didn't implement a basic backup system.
I wish them luck recovering from this, and hope they learn to test better and have good backups.
I've seen "official IT" give people laptops without basic drivers installed, and the screen not running at native resolution! I've even seen shadow IT maintain their own install images to re-image every system with...
In my school, we could SSH from Internet using password. So one compromised password could provide entrance into network.
That's incredibly naive. Not everyone has a Comp-Sci degree. Not all specialised programs will allow users access to the data easily. Even if they do, the student/staff may not be knowledgeable enough to back it up. The data may be multiple gigabytes in size or greater... There are lots of reasons why local data exists like this.
If it wasn’t this issue it would have been something else, like getting up at the coffee shop and having your laptop stolen.
First you have to know that utility exists. Will most biology students know about Windows Backup? I doubt it. Even if they do, they're using a college administered PC. Backups are not their concern. They probably won't have permission to set it up even if they want to. In the article the staff member doesn't have permission to install the applications they need on their desktop PC.
I've had cloud providers nuke my disk intentionally, then demand I upgrade $$$ to get access to backup.
Backups do fail, get corrupted or are incomplete.
Lots of this data is not trivially copyable or even obvious to find.
Nevertheless, data inevitably gets erased. It's part of the data management lifecycle. If data is non-trivial to backup or obvious to find, it must not be very valuable to those who own the data. Think about it.
Not deleting data, you might even become criminally liable, according to privacy laws like GDPR.
So while this stuff isn't as clear-cut, stable and easy as it should be (we agree there), people must keep their data with them, if the data is valuable to them.
I don't see any way forward laying blame on others for the rest of one's life. That's no way to live.
This is why one implements data and IT governance.
Ownership is with the user.
However, it can be profitable to not undo the mistake, and the workers have no say what is negotiated in the contract.