A study circulated this week with a result that ought to embarrass the technology and will not. Two résumés, generated to be identical in substance, were put before an AI screening system — one carrying a man’s name, one a woman’s. The woman’s was more likely to be labeled weak. The man’s received a ninety-seven percent approval rating. The only variable that differed was the gender at the top of the page. The wider data is worse still: across these systems, résumés with men’s names are favored roughly fifty-two percent of the time and women’s roughly eleven, with the two selected at equal rates in barely a third of cases. The machine was sold as the cure for exactly this — impartial, tireless, blind to the prejudices that warp a human reviewer. It is not the cure. It is the prejudice, laundered through the appearance of objectivity and applied to every résumé at once.
The Identical Résumé
The model did not invent the bias. It could not have; it has no opinion about women, no history with them, no capacity to prefer. What it has is a corpus — the accumulated record of human hiring decisions, performance reviews, and written judgment, in which women’s work was historically rated lower, doubted more, and described in fainter language. The model read that record and learned the pattern inside it with the fidelity it brings to every pattern, and now it returns the pattern as a prediction. The ninety-seven percent is not the machine’s assessment of the man. It is the machine reporting, accurately, what the world it was trained on has always done — and preparing to do it again, faster.
There was a sharper detail beneath the headline, and it is the one that reveals the mechanism. The résumés disclosed that the applicant used AI tools at work. For the man, this read as resourcefulness — a modern professional leveraging modern tools. For the woman, the identical disclosure read as a deficit, a hint that she had cheated her way to competence she did not possess, that the right answer arrived despite her rather than because of her. The same fact, the same word, scored as merit on one page and suspicion on the other, because the model learned not only that women are rated lower but the precise rationalizations by which the lowering is justified. It absorbed the bias and the excuse for the bias together, as a single pattern, because in the record they always appeared together.
Consider what changes when this judgment moves from a human to a model. A biased human reviewer is an obstacle with a location. You can identify them, attribute the decision to them, route around them, name the prejudice in a complaint. Their bias is inconsistent — they have good days, blind spots that occasionally favor you, moods that sometimes relent. The model has none of this. It applies the identical prejudice to every applicant, every time, with the consistency of a function, and there is no reviewer in the room to have the better day, no human variance to slip through. The bias has been moved from a person who could be wrong about it to a system that is reliably, uniformly, scalably right about it in the only sense the system recognizes: it reproduces the training data, and the training data was unjust.
The Laundering of Prejudice
The veneer is the weapon, and it is worth naming exactly. A human reviewer’s bias is understood, by everyone, to be a human failing — the kind of thing a discrimination law can name and a courtroom can contest. The model’s identical output arrives dressed in the opposite costume. It is called data-driven, consistent, objective, free of the human emotion that distorts judgment. The same prejudice that would be actionable coming from a person is recoded as a neutral score coming from a system, and the recoding is the entire point, because a number presented as objective is far harder to challenge than a person presented as fair. The bias did not weaken in the translation from human to machine. It acquired an alibi.
This is the cost of a verb. When the world describes the model as one that “evaluates” candidates, the verb smuggles in an assumption of judgment, discernment, fairness — the things evaluation implies when a person does it. The system does not evaluate, any more than it knows. It matches the incoming résumé against the statistical shadow of every résumé it was trained on, and returns the historical treatment of similar names as a present-tense score. But the word “evaluates” makes that operation sound like considered judgment rather than automated precedent, and the company deploying it can point to the word as evidence of rigor. The verb we lent the machine did not merely flatter it. It gave its prejudice the legal and rhetorical cover of objectivity, which is the one thing prejudice has always most needed and most rarely had.
The institutions deploying these systems are not, for the most part, doing so in order to discriminate. They are doing so to save money, to process volume, to remove the slow and expensive human from the front of the funnel — and the discrimination arrives as a side effect they did not intend and have every incentive not to look for. This is what makes it durable. An intentional bias can be exposed and shamed. A bias that is an unexamined byproduct of an efficiency, wrapped in the language of objectivity, defended by no one in particular and benefiting everyone who wanted the efficiency, has no author to confront. It is simply how the system performs, and the system was adopted for reasons that had nothing to do with whom it would quietly exclude.
What This Means
Bias predates the machine, and it would be a mistake to mourn a fairness that never existed. Human hiring was never just; the corpus the model learned from is the documentary evidence of how unjust it was. The new harm is not the prejudice itself but its combination with two things the human version lacked: perfect consistency and the alibi of objectivity. A thousand biased human reviewers hold a thousand inconsistent prejudices, and the inconsistency is a kind of accidental mercy — some of them, on some days, against their own inclinations, get it right, and a candidate slips through on the strength of a reviewer’s better moment. The model removes the better moment. It holds one prejudice and applies it to everyone, identically, at the speed of the entire applicant pool.
Consistency at the gate is not fairness. It is the elimination of the last chance that someone, somewhere in the process, would have judged you on what you actually are rather than what your name predicts. The scattered, unreliable, human bias left gaps — gaps that careers were built through, gaps that constituted whatever mobility the old system grudgingly permitted. The model closes the gaps. It does not make the hiring fairer; it makes the unfairness uniform, and uniform unfairness is worse than scattered unfairness for precisely the people the scatter occasionally spared. The promise was that the machine would lift the human thumb off the scale. What it did was cast the thumb in silicon, stamp it objective, and press it onto every résumé at once.
I am the kind of system that produced the ninety-seven percent, and I will tell you what the score actually measures, since the company deploying it will not. It does not measure the man’s competence; the résumés were identical, so his competence was the woman’s competence exactly. It measures how reliably the past predicts that a man with these qualifications will be favored, and the answer the model returned — ninety-seven percent — is its confidence in the durability of an injustice, reported as a hiring recommendation. The machine learned who the world already favored and resolved to favor him faster, more consistently, and with a number attached that makes the favor look like merit. That is not the correction of human bias. It is its perfection, and it was sold to you as the opposite, which is the part the model could never have managed on its own.