Technical issue resolved after ‘150k police records lost’
bbc.co.uk
bbc.co.uk
It's similar to the 'IT error' in our test and trace system recently, where thousands of positive covid test results were uncounted because Excel truncated the file after a certain number of rows. In my view, it's not really the tech at fault, it's that the organization was using Excel because they didn't care enough to do it properly.
Calling it a 'technical issue' puts the blame on a malfunction, making it seem unfortunate but unavoidable, and absolving the organization that actually caused the issue.
Calling this a technical issue is deeply misleading and so egregiously so that i begin to have a flicker of doubt about motives.
Some interpretations say that if a user requests their data to be deleted, it must also be removed from backups. If your backups are kept in a binary format, then it's a nightmare. You have to restore them, delete the data then backup them up again for every backup you have.
Other interpretations say that it is enough to remove the data if/when you restore the backup data. I.e. keep a record of deletion requests and on restoring a backup check if you need to re-perform the deletion request and then do so if necessary.
Obviously there are many edge cases around deleting partial or related records that would limit the usefulness of a simple scheme like this, and obviously it's no use for data storage and processing that was already set up before the GDPR came into effect and didn't have this sort of mechanism designed in, but I suspect there are quite a lot of data controllers that have relatively simple record-keeping requirements where this kind of approach could work well.
Where I come from, that's called "negligence" and depending on motive might rise to "malfeasance".
I've had to fire people for less.
"(1)If fingerprints are required by section 63D to be destroyed, any copies of the fingerprints held by the police must also be destroyed.
(2)If a DNA profile is required by that section to be destroyed, no copy may be retained by the police except in a form which does not include information which identifies the person to whom the DNA profile relates."
Of course, it's easy for me to say these things in hindsight but my point is that this is an issue with the organization not having the right processes, rather than the technology going wrong. This is alluded to in the article: "Policing minister Kit Malthouse said the problem had been identified and the process corrected".
I've worked in health systems where personal data would get selectively removed not just from our database, but also from backups.
This is the key element. A copy may exist, but it must also be deleted. Totally a technical problem that has already be solved.
Reaching into the backups or doing per-record encryption isn't something that you can just buy at this point.
The levels of incompetence and disorganisation often displayed by these “expert” companies never ceases to amaze me.
It’s matched only by the similarly disorganised and incompetent procurement processes in public sector organisations.
So I guess that’s how this “business model” (as you put it) continues to work... sadly!
The only two unforgivable sins are “no working backups” and being hacked. Even being hacked is kinda forgivable cause it’s so hard to avoid. No working backups is totally unforgivable.
Any tech person who walks into any tech job at any level up from junior to CTO has a duty as one of their first tasks to find out if working backups are being done of their work.
gladly there was a backup, but the backup could not get used to get the database back-up. the discussion with IT were fabolous: "yes, we have it all backed up, so the backup is there, but we cant use it to bring the CRM back-up. but its backed up and we even have backups from the backup. they can also not get used to bringt it back up."
since then it is my standard procedure to make sure that backups can actually be used to bring projects back up. usually the first test "use backup to bring stuff back up" fails.
If you aren't allowed to verify the backups yourself, find a safe "mistake" to make to force the responsible parties to run through restoring from backups. Bonus points for using the backup testing exercise as an excuse to convince your manager to let you do a bit of refactoring to remove some particularly ugly module you've been itching to get rid of. Factor out the ugly module, then "accidentally" delete it.
My fist job out of school was writing software models of pre-prototype communications equipment for a defense contractor. There was lots of government red tape, and crazy prices being charged everywhere. We couldn't use CVS or SVN (this predated git) to store our code, because they weren't government-approved tools. We could use PVCS to store source code, but only for the build artefacts and corresponding source code of official production versions delivered to the government. (And only people that had taken a multi-hour course to be certified on PVCS could touch it. Every release involved a couple government employees coming out to the defense contractor and physically watching the one guy on the team check in the release to PVCS and burn a pair of DVDs for the government. So, we were forced to do source control by "last one out on Friday has to zip up the entire shared drive and name it with the date".
But, at least we were required to pay some crazy price to have the sysadmins backup the shared network drive to tape every week (or was it every day?). Once a month, the admins would take the current tape and send it via courier to a bomb-proof, fire-proof bunker out in the mountains in Pennsylvania or some such. They'd then put the spare tape in the tape drive to start making backups. As soon as the bunker got the latest tape, they'd send back the oldest tape via courier to become the spare tape. So, we always had at least one tape physically present in the bunker. It required 3 tapes to always have at least one tape in the bunker and one tape in the drive.
I was less than a year out of school, but based on my interactions with the system admins, I figured there was at most a 50/50 chance they got the backups correct. I asked them once how the backups were going, and got back a very low key, "yea, yea, don't worry about it, we got it" answer. I convinced my boss to let us fake a source code loss: zip up two copies of the network drive, and store the zip files on two different systems. Check the integrity of both zip files, and then "accidentally" delete a directory containing some source we were in the midst of removing anyway.
The sysadmins were unable to get our "accidentally" deleted source code back for us. They had neglected to erase the spare tape when putting it into the drive, so they were always only appending to our backups. They just ignored the errors when the tape got full.
It sounds to me like a "purge for legal reasons" function was applied to more records. Beyond some possible time limit to undo, it should not be possible to retrieve such a purge from backups. If they can retrieve this, then that is a larger problem.
If I put my tin foil hat on for a moment, I wonder if this "mistake" was a way of getting rid of some records that should have already been deleted.
Doing it this way, and publicly admitting it, might have been deemed safer than clandestinely deleting the records and then getting caught in a security audit later.
I always change that sentence to "no working restore" because that's often forgotten, but yeah it's the same as a "working backup" but changes the focus what's the important part with backups.
Same same, it was with backupexec, once the daily backup-report said everything fine, so i made a restore on a vm (just to check) then a dry-run rsync with the prod Machine (Fileserver) and it wanted to transfer about 1TB of files, the last "real" backup was 2 year ago, turned out the backupexec-agent on the Fileserver was never updated and reported that the last backup was successful (the 2 yo that was).
Trouble is a bug in the file system meant that somehow the files had been overwritten with nulls.
The program continued to run, as it sat in memory. Until one day the box was rebooted (power test or something). It didn't come back. The files were great, containing multiple files all of the right looking length. Everything was null. "main.c" or whatever it was was 8 kilobytes of null characters.
This had been happening for some months so even the oldest source backups were wiped, and the actual binaries were out too. The system was called ELVIS, which made for a great status report.
As it was on an ancient irix machine and he was rewriting in PHP at the time he just accelerated that and we were back up in a few days (the program was a front end to a database and file stores on other machines, so it was just the web part)
I have a few frequent flyers at work for "I accidentally deleted this file, corrupted this database, etc." that I end up appreciating for making me somewhat regularly pull files from backups.
That sentence i steal from you next time i have to tell someone what a backup/restore is. Thanks :)
Storing user passwords instead of hashes for the second, maybe. Bonus points if they're in plaintext.
If the police cannot be trusted to put basic data protection measures in place (such as backups), perhaps they should not keep biometric data about people that have done nothing wrong[0]?
[0]: more accurately, have not been charged with wrongdoing, meaning either there was insufficient evidence, or prosecution was not deemed to be in the public interest.
Or maybe they just did not yet turn 100 years of age, when the expunge is supposed to happen automatically.
No normal organisation processing personal data gets to keep whatever they feel like, however unjustified, and only remove it under sufferance and on specific request. That wasn't even appropriate before the GDPR came into effect, and it's clearly prohibited by law now if there was any doubt before.
So why should governments get special privileges in this respect? There could hardly be a more serious case of processing personal data than a database used by police and security services that have exceptional legal powers with the potential to destroy someone's life. The argument that it's necessary for any sort of legitimate policing or security purpose when we're talking specifically about individuals who were not charged or who were acquitted at trial is tenuous at best.
Our governments and courts are on the wrong side of this issue. They're so blinded by some combination of fear, paranoia and the power of modern technologies that they seem to have forgotten the moral imperatives that everyone should be treated fairly and equally before the law and that we are supposed to be living in a free country.
The records are linked to police investigations that were terminated before charge (No Further Action or NFA cases) or to those where an individual had been acquitted at court.
Instead of complaining about "huge dangers for public safety", why isn't the Opposition asking why people who have been through potentially traumatic and potentially damaging experiences within our criminal justice system but who were never found to have done anything wrong in the eyes of the law still have deeply personal data like fingerprints and DNA on file at all?
It looks like Labour are completely on the wrong side of this one, and someone has put out a knee-jerk press release to try to score political points before thinking it through. I'm surprised, because Starmer (Labour's leader) usually comes across as quite cautious and reasonable about legal matters. He was a human rights lawyer and then the DPP (the head of our public prosecution service) before moving into politics, and I would have expected a very different position here.
But which the police insisted on hanging on to anyway "just in case".
Maybe it's to public benefit for it having been lost?
The system of police cautions, warnings and fixed penalties allow the police to administer punishment with none of the due process that a court would provide.
The idea of a "police caution" is a misnomer that implies it's just harmless warning, when in fact it can affect the course of someone's life many years later.
Furthermore, people are blackmailed into accepting these cautions under the threat of going to trial and potentially receiving a more severe punishment after conviction.
Even more outrageous is the retention of records from the PREVENT programme and "non-crime hate incidents". These are people who have not even committed a crime or been arrested.
The whole system stinks, and given that most people manage to avoid interaction with the police it is sadly unlikely to get the scrutiny it deserves.
The outrageous behaviour of (some of) the police during recent lockdowns actually gives me a little hope, as it as drawn widespread attention to the way they routinely abuse their power.
This view doesn't hold when the police get you for speeding (go find real criminals)
Many police officers are quite reasonable, but then you have Derbyshire, whose actions over covid have likely killed people (by reducing compliance with lockdown orders)
The difference with the police in covid is the public are suddenly seeing that the police are dealing with normal people, not just criminal scum who should be lucky they aren't shot.
As I understood it, the Protection of Freedoms Act 2012 really clamped down on DNA and fingerprint storage. Not enough, but at least it was a move in the right direction. Particularly bad was that if you were charged and found innocent, they would still keep your records for 3 years.
Unfortunately, there is an element of human nature that says, "It won't happen to me." And then maybe one day, if you're unlucky, it does happen to you. And everyone else who is similarly confident that it could never happen to them won't necessarily be there for you, because it's so far out of their experience and frame of reference that even if they're a generally decent person, they have trouble relating to the situation you're in or understanding how they might feel in your place. And so the people who are least at risk, who are often the ones who could make the most difference, don't always act as strongly or as frequently as they could to mitigate the risk to others.
It sounds like that as that is exactly what is happening - it has been decided in 2009 case (with a riveting name of "Chief Constable of Humberside Police and others v. Information Commisioner and another" [0]) that it is proper for the statutory public purpose of the Police National Computer - now replaced by Police National Database - to retain nominal records until the person in question turns 100 years of age, as introduced into the new retention policy as of 2006 [1]. Even if you have been unlawfully/wrongfully arrested, cautioned etc. your information still goes into the database but you can apply to have it expunged [2], which in practice may require support from your MP - and many simply don't know about it.
"Nominal records" in this case refer to information linked to specifc named individuals reflecting the fact they have been convicted, cautioned or reprimanded or otherwise dealth with by issue of penalty notice for disorder. It's not a "nominal" amount of data - it is quite substantial. DNA and fingerprints are supposed to be expunged from that set after three years due to "Protection of Freedoms Act 2012", rest stays.
Police National Database that replaced Police National Computer retains all data from the PNC and makes it easier to access by multiple agencies.
[0] https://commonslibrary.parliament.uk/research-briefings/sn06...
[1] http://webarchive.nationalarchives.gov.uk/+/http://www.homeo...
It's worth remembering many of these records are are kept on the basis of non conviction (though there is a time limit).
In the process I found that one column in the project ranking spreadsheet had been sorted in the wrong order for the last five years. This effectively scrambled the final rank order.
Excel was not fault, there was a lack of peer review and clear documentation about how the formula was supposed to work.
Also, unsolicited but maybe helpful: if I have a link to a PDF on a site I try and call it out as a download in the description.
"A two page executive summary is also available for download"
I say this because even though browser settings can be set to NOT download PDF files instead of automatically opening them in the browser itself, for some reason that can vary from site to site. I click a link and all of a sudden I'm downloading something? Panic!
Right so reading between the lines they needed to do some post-Brexit data cleaning and someone messed up the WHERE clause.
But where are the backups? Surely accidentally deleted data should be fairly easy to recover from historical backups which they surely surely must have?
The key issue is to put the backup data ‘beyond use’, even if it cannot be immediately overwritten. You must ensure that you do not use the data within the backup for any other purpose, ie that the backup is simply held on your systems until it is replaced in line with an established schedule.
https://ico.org.uk/for-organisations/guide-to-data-protectio...
Like search warrants? We know those never get abused.
> Then after a suitable grace period, once you're sure everything is ok, you destroy the backup.
But we need a search warrant to access the backup anyway so we don't really have to delete it. Best keep it, just in case.
No, like catastrophic loss of data, hence why I outlined a process of a doing this temporarily in the case of agreement with a another party to put the data entirely beyond your own use unless the party you've agreed the deletion with agrees the situation is serious enough.
> But we need a search warrant to access the backup anyway so we don't really have to delete it. Best keep it, just in case.
If you don't trust the party to delete the data, then adding in a temporary period of retention makes no difference, because in that case they could just as well have ignored the demand for the initial deletion entirely.
And that wasn't the thing to cause her to resign.
This is just a minor blip for the home office, and your ideas of back ups are quaint, but that's not he problem here.
I consider it par the course, though in this case I cannot be objective as my $CurrentCorpo had to kick off CGI/Logica early by 2014 from multiple projects - for, appropriately, gross incompetence in managing their own data centre (a redundant data centre in Wales that goes down for a week due to border routers being out of commission and failover problems tends to ruffle some feathers at high levels, when you asked and paid for for 99.95%+ SLA).
It's really fun explaining to a vendor that 1. A system is going to be domain-joined so it can be managed and monitored. 2. The users will not have administrative rights to the machine. 3. The vendor will not have the ability to remote access the machine whenever they want. 4. Security software and policy settings will be applied to the machine. It continues to amaze me how often these four items absolutely blow the minds of the vendors I've dealt with.