Terry Gilliam nailed this well in Brazil. https://youtu.be/6OalIW1yL-k
"We don't make mistakes"
The problem is when these aren't verified and no support for bad data decisions.
When it is combined with authority and security it is even scarier in a dystopian way like Brazil or Idiocracy.
There will be mistakes, there needs to be a pathway for support and verifying this data. Almost like a right to trial but for data decisions made about you. And not the trial Not Sure got in Idiocracy.
We already have bad situations like this in credit tracking and identity theft, it gets worse when enforcement and oversight is favoring data over detective work.
Recourse on data decisions or algorithm decisions would be an excellent addition to the Bill of Rights in a much needed "Right to Data" amendment.
Edit: idiocracy is a great film. However watching these dystopian films as a software engineer is somewhat stressful :)
Mike Judge being an engineer went on to make lots of great satiric and really truthful comedic views of technology in Idiocracy, Office Space and Silicon Valley.
Gilliam and Judge have an excellent knack for nailing the biting areas of this and making fun of it fantastically with satire.
All you can do is laugh knowing how scary it can get, knowing how little say engineers actually have to be ethical and how unethical and incorrect some of these systems pushed can be.
We should take decisions by systems as we do foreign policy: "Trust, but verify". This is more difficult when verifying the decision process of an algorithm is unclear. At least giving someone a required speedy chance to deal with a data/system decision that affects their life.
I saw it again about a year later.
Which made me realise just how mind-warping the film was. Another three decades and some on, I realise, largely in a truthful way.