Isabel Castaneda failed high-level Spanish — her native language — because an algorithm said so. She had spent two years of high school in Colorado calculating how many college credits she could accumulate through the International Baccalaureate before tuition began. When the IBO canceled spring exams in 2020 and substituted an algorithmic grading model, the machine gave her failing marks. The IBO refused to explain the model and charged fees for its standard appeals process. If the same algorithm had instead overcharged her credit card, she could have called her bank, disputed the charge, and had it reversed within days — for free, with roughly a 90% chance of winning. That right has existed in the United States since 1974. The right to challenge an algorithmic decision about your education, your health care, your job, or your freedom does not exist anywhere — not in the United States, and barely in the European Union. In a 2021 article in the Columbia Law Review, Margot Kaminski and Jennifer Urban map this gap, make the case for closing it, and build the most detailed framework yet for what the right to contest AI should actually look like.
The particular shape of machine failure
Human decision-makers can be wrong, biased, and arbitrary. Algorithms can be all of those things too — but the way they fail is structurally different, and the usual legal safeguards don’t transfer.
Castaneda’s case was not a fluke. That same summer, England, Wales, and Northern Ireland used a different algorithm to grade university-entrance exams; nearly 40% of students got lower marks than teachers had predicted. Officials reversed the results after street protests. In Idaho, an algorithm allocating home health-care hours for severely disabled patients failed to account for diabetes and miscoded cerebral palsy. Hundreds of patients had their care slashed. A patient told The Verge that neither she nor the caseworker entering her data could “quite understand what was happening.” The software vendor didn’t know about the diabetes error. In Massachusetts, an algorithm estimating kidney function gave Black patients healthier scores than white patients with the same kidney disease, making them less likely to receive a transplant referral.
These are not random malfunctions. Kaminski and Urban argue that algorithmic failure has a distinctive shape. Three features set it apart.
First: categorical processing. Algorithms sort people into groups using rules fixed at the design stage by programmers who may never have encountered the populations the system will eventually process. A USDA fraud-detection algorithm for the SNAP food-stamp programme flagged purchases in round dollar amounts. At Somali-American grocery stores, where customers customarily bought meat in whole-dollar amounts, the flag meant something different from what the designers assumed. This is the “long-tail problem” — the weird case the system was never trained for, except it isn’t weird at all to the person living it.
Second: opacity. Actuarial algorithms — statistically derived and relatively interpretable — at least let you see what factors were weighed. Black-box machine-learning systems often cannot explain their own outputs, even to their designers.
Third: the removal of human judgment from the loop. A caseworker might exercise discretion or compassion. An algorithm cannot. As the attorney for the Idaho home-care patients put it: when you rely solely on algorithms, “we reduce a person’s humanity to a number.”
A right written in pencil
The GDPR established a right to contest certain automated decisions in 2018. Eight years later, it is still mostly an aspiration — underspecified, underenforced, and barely tested in court.
Kaminski and Urban’s article is the first in-depth examination of the GDPR’s contestation right for a US audience. Their central finding: Article 22 says individuals must be able to contest certain automated decisions, then says almost nothing about what that means. No definition of adequate contestation. No timelines. No substantive grounds on which a challenge can succeed. The European Data Protection Board’s guidance restates the statutory text and offers, as an example of compliance, “providing a link to an appeals process at the point the automated decision is delivered.”
This vagueness is not new. The French data protection law of 1978 already gave citizens the right to “dispute the data and logic used” in automated decisions. The EU’s 1995 Data Protection Directive contained a precursor — Article 15 — that scholar Lee Bygrave described as “all dressed up but nowhere to go.” Article 22 added explicit contestation language, but through most of the GDPR’s first years it remained what scholars called a “paper tiger.” The first court anywhere in Europe to recognize a right to explanation under Article 22 was a Dutch court — in 2021, the year this article appeared.
The United States has nothing comparable at the federal level. The Algorithmic Accountability Act of 2019 would have required risk assessments but created no individual contestation right. California’s Privacy Rights Act directed a state agency to develop regulations on automated decision-making; it stopped short of a right to contest. What exists are technology-neutral laws — administrative procedure, antidiscrimination statutes, sectoral due process — that can sometimes, in specific contexts, be stretched to cover AI decisions. The stretching is expensive, the evidentiary burdens are heavy, and the coverage is patchwork.
Meanwhile, the right to contest a credit card charge — governed by the Fair Credit Billing Act of 1974 — works millions of times a year, with consumers winning roughly 80-90% of the time. The question Kaminski and Urban press is structural: why can’t algorithmic decisions be contested as effectively as billing errors?
Why you should be able to argue with a machine
The theoretical case for an individual right to contest AI draws on three traditions — accuracy, rule of law, and dignity — and each is sharpest when it starts with a specific failure rather than an abstraction.
The Somali-American grocers didn’t need a theory of due process. They needed someone to listen when they said the fraud flag was wrong. Their vindication came through individual contestation — not a systemic audit, not a class action, but individual challenges that surfaced an error no auditor had thought to look for. This is the accuracy argument: individual contests catch mistakes the system cannot see, because the person inside the long tail knows something the algorithm doesn’t.
Now consider a hypothetical Kaminski and Urban borrow from Ed Felten: an algorithm that finds credit risk correlates with sock color. Pink socks get better credit than blue socks. Even if the correlation is statistically robust, the rule seems irrational. And if sock color correlates with gender or race, it may also be discriminatory. No systemic audit would necessarily catch this unless it knew to test for sock color. An individual who was denied credit and asked why could surface both the irrationality and the discrimination. This is the rule of law argument: contestation holds a decisional system to consistency, rationality, and fairness by forcing it to explain itself one case at a time.
We reduce a person’s humanity to a number. — Patient advocate, quoted in Kaminski & Urban (2021)
And the Idaho home-care patients. People who lost independence, who skipped meals, who feared institutionalization — because an algorithm miscoded their conditions and no one knew how to challenge it. Their attorney’s phrase is the dignity argument at its sharpest: “we reduce a person’s humanity to a number.” Much US scholarship dismisses dignitary reasoning as a vague European import. Kaminski and Urban point to the HEW Report — a 1973 American document, the founding text of US data privacy law — which explicitly called for individuals to be able to contest algorithmic labels. The dignitary tradition in America was there before Europe codified it. It was progressively narrowed as US privacy law retreated into a notice-and-choice model that expected individuals to read disclosures and opt out.
The article’s main opponent is Aziz Huq, who argues that individualized process is essentially a utilitarian tool for ensuring systemic accuracy — and that class actions and ex ante regulation do this more efficiently. Kaminski and Urban respond that this misidentifies what process is for. A right to challenge a decision about you is not a debugging tool for the system. It is a form of participation, an acknowledgment that the decision was made about a person rather than run against a data point. Audits may be cheaper. They are not the same thing.
Four ways to build a right to contest
The article’s most lasting contribution is a 2x2 matrix of contestation designs — rules versus standards on one axis, procedure versus substance on the other — tested against four legal regimes that already exist.
Two design choices shape every contestation scheme. First: is the mechanism a rule (precise procedures specified in advance) or a standard (the right exists, details left to whoever applies it)? Second: is the focus procedural (how to challenge) or substantive (the grounds on which you can win)?
Quadrant 1 — GDPR Article 22. A standard with procedural focus. Companies must allow challenges; almost nothing is specified about what the challenge must consist of. Member States have filled the gap in radically different ways: the UK imposed one-month timelines with written responses, Slovenia anchored its version to antidiscrimination law, Hungary and France banned automated decisions based on sensitive data.
Quadrant 2 — DMCA notice-and-takedown. A rule with procedural focus. The statute prescribes every element: what a takedown notice must contain, how fast platforms must act, what a counternotification requires. But the underlying substance — copyright infringement — is rarely examined in practice. Platforms remove content on any conforming notice rather than make a legal call. The counternotice mechanism, designed for targets to restore wrongly removed material, is barely used. Practitioners call it dead letter.
Quadrant 3 — EU Right to Be Forgotten. A standard with substantive focus. The Court of Justice in Google Spain (2014) established a balancing test between privacy and public access, but prescribed no procedure. Google built its own review process. Regulators gradually filled in criteria. The standard has become more rule-like over time, though it was never designed that way — and there is no mechanism for the public to argue that information should remain searchable.
Quadrant 4 — Fair Credit Billing Act. A rule with substantive focus. The statute defines “billing error” precisely enough to constrain discretion, and specifies timelines and procedures. Consumers win 80-90% of the time. The system works partly because “billing error” is simple to define — a luxury unavailable for concepts like “algorithmic discrimination.”
Why the credit card company says yes and the copyright platform doesn't care
Both the DMCA and the FCBA have precise procedural rules. One works. One doesn’t. The difference is in the incentive structure — who bears the cost of getting it wrong.
The DMCA has a beautifully designed process. Specified elements for notices, specified timelines, a counternotice mechanism. On paper, it balances the interests of copyright holders and content creators. In practice, it skews overwhelmingly toward removal. Platforms face potential copyright damages of up to $150,000 per work infringed, so they remove first and think later. One empirical study found that 31% of takedown notices to Google Search were questionable; 70% of those to Google Image Search were fundamentally flawed. Despite this, counternotices are vanishingly rare.
The reason is asymmetric risk. A notice sender declares good-faith belief of infringement — no oath required for the substantive claim. A counter-notice sender must swear under penalty of perjury that the material is not infringing, and accept US federal court jurisdiction. An OSP representative told researchers: “The process forces you to try to stay out of making judgment calls.” The system’s risk architecture chills the very contestation it was designed to enable.
The FCBA works the other way around. Investigating a billing dispute costs the credit card company real money: pulling records, reviewing evidence, writing a response. Refunding the charge is cheap. The statute makes investigation mandatory once a complaint is filed. The math tips toward early resolution. Companies rule for consumers 80-90% of the time — not because the law demands generosity, but because it makes generosity cheaper than the alternative.
The design lesson Kaminski and Urban draw: a contestation scheme is only as good as its incentive structure. Process without the right incentives produces the DMCA — a system where the counternotice is dead letter and the stronger party dominates. Process with the right incentives produces the FCBA — a system where the consumer wins most of the time and the system is widely regarded as legitimate.
A floor, not a ceiling
For the United States, Kaminski and Urban propose a cross-sectoral right to contest AI — a baseline that applies everywhere, with sector-specific augmentation where deeper traditions already exist.
Three things must be present for a right to contest AI to work in practice.
Meaningful notice. Not a link buried in a privacy policy. Information specific enough that the individual knows what decision was made, on what basis, and how to challenge it. The GDPR requires that individuals be told when a decision involves automated processing and be given “meaningful information about the logic involved.” Most Member States have interpreted this loosely. France is the exception: its public-sector rules require that officials explain, to the specific person, in detail, how the algorithmic processing was applied to their case. This is the demanding end of the spectrum — and arguably the minimum for contestation to mean anything.
Genuine reason-giving. Not a summary of the algorithm’s general method but an account specific enough for the individual to identify what went wrong. Frederick Schauer’s argument — that reason-giving commits decision-makers to an outcome, allows quality control, and shows respect for the subject of the decision — applies with special force to machine decisions that don’t naturally come with reasons attached.
Systemic transparency. The DMCA operates as a black box: no requirements for platforms to disclose their policies, their decision-making frameworks, or their outcomes. The Council of Europe has recommended that companies publish the number and type of complaints received. Without this kind of visibility, there is no way for regulators, researchers, or the public to tell whether a contestation system is fair.
For the United States, the right should operate as a floor. Cross-sectoral. Attached to the technology, not to a particular industry. Applying wherever AI makes decisions with significant effects on individuals. Supplemented by sector-specific rules in criminal justice, credit, housing, and employment, where existing law already has deeper due process traditions. Covering not just “solely” automated decisions — a qualifier too easily gamed by inserting a rubber-stamping human — but any decision in which AI plays a significant role. Canada’s Privacy Commissioner has already recommended this broader scope.
The model is neither the GDPR’s underspecified standard nor the DMCA’s precise-but-empty rule. It is closer to the FCBA: clear on substance, specified enough on process to be usable, embedded in a regulatory environment with real penalties, and designed so that doing the right thing is cheaper for the decision-maker than ignoring you.
The article appeared in 2021. The EU AI Act was a draft proposal. Article 22 had not yet been tested in court. Five years later, the EU has added Articles 85 and 86. The United States still has nothing.
The GDPR’s Article 22 has been in force since 2018. Its first judicial interpretation came three years later. The DMCA’s counternotice has existed since 1998 and is barely used. The FCBA’s chargeback process has existed since 1974 and is used millions of times a year.
The gap between these outcomes is not a gap in legal architecture. It is a gap in design — specifically, in whether the design makes it cheaper for the person holding the power to give you a fair hearing than to ignore you. That is the question Kaminski and Urban leave on the table. Neither the GDPR nor the AI Act has answered it yet.