In May 2016, two ProPublica journalists sat down to review the records of more than seven thousand people arrested in Broward County, Florida. They were after an answer to an uncomfortable question: the software that helped decide who waited for trial in jail and who waited at home, did it treat a Black defendant the same as a white one? The answer they found came from no laboratory and no controlled experiment. It came from real case files, real people, decisions that had already been made.
Algorithmic bias stops being an abstract topic when you look at it this way: not as a theoretical flaw, but as something that already happened, with a name and a face, in systems that were live in production. The three cases that follow are documented, audited and, in two of them, answered by the very companies that built them. Each left behind a distinct lesson about how a model learns to discriminate without anyone asking it to.
01 · the software judgeCOMPAS: when risk is predicted differently by color
COMPAS is a tool that calculates a "risk of recidivism" score for defendants in United States courts. A number from 1 to 10 that guides judges on whether someone will offend again. The system, from the company Northpointe (today Equivant), was in use in several states when ProPublica published its analysis in 2016.
The central finding was this: among the people who did not reoffend, Black defendants were nearly twice as likely as white ones to have been labeled high risk. And the reverse held too: among those who did offend again, white defendants were more likely to have been flagged as low risk [1]. In other words, the system's error did not fall evenly. It erred against some and in favor of others.
The system never asked about race. But it learned to discriminate by it all the same.
Here comes the nuance that makes this case a classic. Northpointe replied that its model was in fact fair: for every risk score, the actual recidivism rate was similar across groups. They were right by their definition. ProPublica was also right by theirs. The problem is that both notions of fairness (equalizing the correct predictions versus equalizing the errors) are mathematically incompatible when base rates differ between groups, something the researcher Jon Kleinberg and his colleagues proved formally soon after [2].
"Fair" is not one single thing. Before auditing a model you have to decide which definition of fairness matters for that context, because you cannot satisfy them all. In decisions that take away someone's freedom, equalizing the errors usually weighs more than equalizing the correct predictions. That choice must be explicit and human, not a silent byproduct of the code.
02 · the recruiter that hated women's résumésAmazon and the model that taught itself to discriminate
Between 2014 and 2017, Amazon secretly built an experimental engine to rank résumés. The idea was seductive: feed the system a hundred CVs and have it hand back the top five. In 2018, Reuters revealed that the project had been shut down for a blunt reason: the model discriminated against women [3].
No one programmed that discrimination. The model learned from the résumés Amazon had received over ten years, a period in which the tech sector hired mostly men. The system absorbed that pattern as if it were a signal of quality. It began to penalize CVs that contained the word "women's" (as in "women's chess club captain") and to lower the score of graduates from two all-women colleges, according to the sources cited by Reuters.
What is revealing is what happened when the engineers tried to fix it. They edited the model to ignore those explicit terms. But there was no guarantee that the system would not find other, subtler ways to infer gender from the language of a résumé. The bias did not live in a single word: it lived in the correlation between thousands of signals and a history of unequal hiring. Amazon ended up scrapping the tool.
The model did not invent the bias. It copied it from us, with uncomfortable fidelity.
| Case | Source of the bias | Documented by |
|---|---|---|
| COMPAS | Arrest data carrying historical inequality; the chosen definition of fairness | ProPublica, 2016 [1] |
| Amazon recruiter | Ten years of CVs skewed toward men in the sector | Reuters, 2018 [3] |
| Facial recognition | Training sets with few dark-skinned women | Buolamwini & Gebru, 2018 [4] |
Source: author's own work drawn from the cited research.
A model trained on data from the past inherits the injustices of the past. If you hired unequally for a decade, automating that record does not correct it: it accelerates it and lends it an appearance of objectivity. Removing the sensitive variable from the dataset is not enough, because the model reconstructs it from proxies. You have to audit the output, not just the inputs.
03 · the faces the system did not seeGender Shades: the facial recognition that failed on dark skin
In 2018, the researcher Joy Buolamwini, of the MIT Media Lab, and Timnit Gebru published Gender Shades, an audit of commercial systems that classify gender from a face, from companies such as IBM, Microsoft and Face++. Instead of accepting the global accuracy figures the vendors advertised, they broke down performance by skin tone and by gender.
The contrast was brutal. The systems classified light-skinned men with an error of less than 1%. For dark-skinned women, the error climbed to 34.7% in the worst case [4]. A gap of more than thirty points, hidden beneath a general average that sounded impeccable. A system can be "95% accurate" and still fail systematically with an entire group of people.
The cause was traceable: the datasets on which these systems were trained and evaluated were made up mostly of light-skinned faces. The model never practiced enough with faces that looked less like those in its data, and its performance gave it away. The good news is that the study had an effect: IBM and Microsoft revised their systems and reported notable improvements in the following months. The audit did not just flag the problem; it triggered the correction.
An accuracy average can lie. The only way to detect a bias like this is to break down performance by subgroup before putting the system into production. What is not measured separately is not seen. And what is not seen is not corrected.
04 · what ties the three togetherFrom the case to the method
Three systems, three distinct origins of bias. In COMPAS, the problem lay in the very definition of what it means to be fair. In Amazon's recruiter, in data that reflected an unequal history. In facial recognition, in a training set that was hardly representative. None was the work of a malicious programmer. All were the result of assuming that a system trained on data is, by default, neutral.
The common thread is simple to state and hard to practice: a model learns from what we show it, and what we show it carries our inequalities. Auditing by subgroup, explicitly choosing a definition of fairness and not trusting global averages are not ethical luxuries. They are basic engineering when the decision affects people. The three cases already paid the bill for not having done it in time. The advantage of whoever builds today is that they can read them as a manual of what is worth reviewing before, not after.
Sources
- Angwin, J., Larson, J., Mattu, S. & Kirchner, L. (2016). Machine Bias. ProPublica, May 23, 2016. propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
- Kleinberg, J., Mullainathan, S. & Raghavan, M. (2016). Inherent Trade-Offs in the Fair Determination of Risk Scores. arXiv:1609.05807.
- Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, October 10, 2018.
- Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of Machine Learning Research (Conference on Fairness, Accountability and Transparency), vol. 81, pp. 77-91. proceedings.mlr.press/v81/buolamwini18a.html.