A self-learning fraud-risk algorithm toppled a government
Dutch Tax Administration · 2013–2021
What they did
The Netherlands' tax administration used a self-learning risk-scoring algorithm to flag childcare-benefit applications for fraud review. The model treated dual nationality and non-Dutch surnames as risk signals, disproportionately flagging families with a migration background.
What happened
An estimated 26,000 families — many falsely — were ordered to repay €30,000 to €100,000 in benefits, plunging them into debt; parents lost homes and jobs, and at least 3,532 children were placed in foster care as a result. A parliamentary inquiry found the system violated fundamental rule-of-law principles, and the entire Rutte III cabinet resigned on 15 January 2021 over the scandal.
The so-what
Automated risk-scoring on already-marginalized populations doesn't need malicious intent to cause mass harm — it needs an unaudited feature (nationality) correlated with a protected characteristic, a punitive downstream process (full clawback, no graduated review), and years of nobody empowered to override the score.