All the non-computer people just assumed that when the computer said something then it must also be true. Why wouldn’t it be, after all computers are magic and incorruptible truth-machines.

Yeah, you can trust a computer if you're know what you're doing and not. Societally, though? Fucking hell no.

Poland had some AI driven nonsense for their benefits claims for a while. Except of course it was always, in the end, a human who said "yeah do whatever". Except basically none of them ever did, it got so bad with it they just tossed it.

>As presented in last year’s report, in May 2014, the Ministry of Labor and Social Policy introduced a simple ADM system that profiles unemployed people and assigns them three categories that determine the type of assistance they can obtain from local labor office. Panoptykon Foundation, and other NGOs critical of the system, have been arguing that the questionnaire used to evaluate the situation of unemployed people, and the system that makes decisions based on it, is discriminatory, lacks transparency, and infringes data protection rights. Once the system makes a decision based on the data, the labor office employee can change the profile selection before approving the decision and ending the process. Yet according to official data, employees modify the system’s selection in less than 1% of cases. This shows that, even if ADM systems are only being used to offer suggestions to humans, they greatly influence the final decision.