It's not clear from your comment whether or not you're familiar with the book Weapons of Math Destruction by Cathy O'Neil, but you echo many of its main arguments about these types of systems. I can recommend the book if you haven't read it.