It's a little bit more complicated than that, it was "the rules said so" and then they just used humans to check if people used the rules. The government at the time was very hard on fraud after a couple of high-profile fraud cases by immigrants. Mostly Bulgarians falsely claiming welfare

Anyway, several ministers and other high officials pressured the welfare department to be hard on fraud, which caused the imposition of harsh fines for minor filing errors. The department also got into a bit of a witch hunt were anyone being investigated for fraud was almost always presumed to be guilty. This, combined with the fact that most of the welfare recipients were not very educated, didn't speak Dutch all that well and/or were under a lot of stress due to being generally economically disadvantaged, led to some big fines levied against people who didn't objectively do all that much wrong. Several of them lost their homes or got deep into debt due to the fines and legal costs.

Simply claiming it was "the algorithm said so" skips over all the history about why the rules were written the way they were, and the human errors made along the way.

Yes, there definitely was and it wasn't so much driven by the tech itself as it was by the need to be seen to be doing something against perceived fraud by minorities which led to mass profiling and huge fall-out (and too little, in my opinion). It was effectively racism under a thin veneer of plausible deniability provided by technology.

Tech by itself can be bad enough, tech wielded to achieve nebulous goals without any consideration for the human side of the effects is outright dangerous.

Indeed! https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...

Indeed, though that wasn't AI, that was direct discrimination and racism hidden behind a thin layer of policy.

https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...