Automated Decision Making
The use of algorithmic systems to make or inform decisions affecting people — and the accountability, transparency and contestability questions this raises.
Overview
When algorithms make or inform decisions affecting people's lives — in employment, education, welfare, credit, criminal justice — questions arise about who is accountable for those decisions and how those affected can understand, challenge or correct them.
Automated decision making (ADM) concerns algorithmic systems that make or substantially inform decisions affecting people across employment, education, credit, welfare, criminal justice, and healthcare. The governance questions span accountability (who is responsible when an algorithm makes a bad decision?), transparency (can those affected understand the basis for the decision?), bias (does the system perpetuate or amplify historical inequalities?), and contestability (can those affected meaningfully challenge outcomes?).
Frank Pasquale's The Black Box Society (2015) was the first major public account of opacity in consequential algorithmic systems: financial scoring, search ranking, and health algorithms that determine access, opportunity and treatment, with no meaningful recourse for those affected. Pasquale's central argument is that the power asymmetry between the controllers of algorithmic systems and those they govern requires structural regulation, not merely consumer transparency measures. Virginia Eubanks' Automating Inequality (2018) provides the case studies: welfare eligibility systems, healthcare rationing algorithms, and predictive policing tools that embed historical inequalities and enforce them at scale, disproportionately affecting communities with the least ability to resist. Cathy O'Neil's Weapons of Math Destruction (2016) documents the feedback loops: algorithms trained on biased data produce biased outputs that produce biased data that train future algorithms.
Kate Crawford's Atlas of AI (2021) adds the material and political dimensions: AI systems are not neutral instruments but assemblages of labour, natural resources, and power relationships. The fantasy of algorithmic objectivity — that removing human judgment removes human bias — is precisely that: decisions embedded in model design, training data selection, and feature engineering reflect their makers' assumptions as surely as explicitly human decisions do.
The regulatory response has been substantial. GDPR Article 22 prohibits automated decisions with legal or similarly significant effects without meaningful human review, establishing a floor for human oversight. The Council of Europe's Recommendation on Automated Decision Making (CM/Rec(2020)1) provides a more detailed framework: the conditions under which ADM is permissible, the obligations of transparency and explanation, and the right to contest outcomes. The EU AI Act explicitly categorises AI systems used in employment, education, and credit as high-risk, requiring conformity assessment and human oversight mechanisms before deployment.
Key Texts
Foundational works in this research tradition.
Opacity in consequential algorithmic systems: financial scoring, search, health. The power asymmetry between algorithm controllers and those affected; the case for structural accountability requirements rather than voluntary transparency. The book that established ADM as a governance problem.
How poorly designed algorithms embed bias, create feedback loops, and harm at scale. Case studies across teacher evaluation, credit scoring, predictive policing, and university admissions. The accessible account that brought ADM problems to a general audience.
Automated welfare, healthcare rationing, and predictive policing systems and their disproportionate impact on low-income communities. Documents how algorithmic decision-making embeds and amplifies existing inequalities — and how affected communities resist.
The material, political, and labour dimensions of AI systems: mining, data centres, labour, and the assumption of objectivity. AI decisions embed their makers' assumptions and power relationships. A challenge to the fiction of neutral algorithmic judgment.
The right not to be subject to solely automated decisions with legal or similarly significant effects without human review. The legal floor for human oversight in consequential ADM — applicable to any AI system making inferences that inform employment, credit, educational, or other significant decisions affecting EU data subjects.
A detailed framework for the permissible conditions of ADM: transparency requirements, the right to explanation, human review obligations, and the right to contest outcomes. Grounds ADM governance in the European Convention on Human Rights.
Related Research
Connected areas of inquiry.