Executive Summary
Governments worldwide are deploying artificial intelligence systems in domains with profound consequences for individuals' lives: welfare eligibility determination, immigration processing, criminal risk assessment, tax fraud detection, and child protection. The OECD's AI Policy Observatory reports that 62% of OECD countries have deployed or piloted AI systems in at least one public sector domain, with government AI procurement spending reaching $18 billion in 2025. Yet the accountability frameworks governing these deployments remain woefully inadequate.
This report analyses public sector AI governance through the lens of principal-agent theory and social choice theory. Government AI systems create a novel accountability gap: decisions affecting citizens' fundamental rights are made by algorithmic systems that elected officials do not fully understand, procurement officers cannot adequately evaluate, and affected individuals cannot meaningfully challenge. Arrow's impossibility theorem — the foundational result in social choice theory demonstrating that no voting system can simultaneously satisfy all fairness criteria — has a direct analogue in algorithmic fairness: no classification system can simultaneously satisfy all plausible fairness definitions, forcing trade-offs that are inherently political rather than technical. We argue that effective public sector AI governance requires treating algorithmic deployment as a constitutional rather than merely technical challenge.
The Accountability Gap: Principal-Agent Problems in Algorithmic Governance
The principal-agent framework illuminates the governance failures endemic to public sector AI deployment. In the traditional model of public administration, the accountability chain runs from citizens (principals) through elected officials to civil servants (agents) who exercise delegated authority. Each link in this chain involves monitoring mechanisms: elections, legislative oversight, administrative law, and judicial review.
AI systems introduce a new agent into this chain — one that operates with a fundamentally different logic. Human decision-makers can be questioned about their reasoning, held accountable for errors, and retrained when policies change. AI systems optimise mathematical objective functions that may diverge from policy intent in ways that are technically opaque to the officials responsible for oversight. This opacity creates what scholars have termed an "accountability deficit" — decisions with life-altering consequences that no human actor can fully explain or defend.
The Netherlands' childcare benefits scandal (toeslagenaffaire) provides a devastating illustration. An automated risk-scoring system used by the Dutch Tax Authority flagged thousands of families — disproportionately those with dual nationality — for childcare benefit fraud investigations. The system's design incorporated features that correlated with ethnicity, but no individual official could be held responsible for what was effectively algorithmic discrimination at scale. The scandal brought down the Dutch government in 2021 and triggered a constitutional crisis that continues to reverberate through Dutch public administration.
Similar accountability failures have emerged across jurisdictions. Australia's "Robodebt" scheme used automated income averaging to issue 470,000 erroneous welfare debt notices, with several suicides attributed to the scheme's impact. The UK Home Office's visa-processing algorithm was found to discriminate on the basis of nationality. The US COMPAS recidivism prediction tool exhibited racial bias in criminal sentencing recommendations. In each case, the AI system operated within a governance vacuum where traditional accountability mechanisms could not function effectively.
The Impossibility of Algorithmic Fairness
A fundamental challenge in public sector AI is that "fairness" is not a single concept but a family of mutually incompatible definitions. Scholarship in algorithmic fairness has identified numerous formalised fairness criteria — including demographic parity, equalised odds, predictive parity, calibration, and individual fairness — and proven that, except in trivial cases, no classifier can simultaneously satisfy all of them.
This impossibility result mirrors Arrow's theorem in social choice theory. Just as no voting system can simultaneously satisfy unrestricted domain, non-dictatorship, Pareto efficiency, and independence of irrelevant alternatives, no algorithmic system can simultaneously achieve all fairness desiderata. The choice among fairness criteria is therefore a normative political decision, not a technical optimisation problem.
Consider a welfare fraud detection system. Demographic parity requires that the system flag the same proportion of applicants from each demographic group for investigation. Predictive parity requires that among flagged applicants, the proportion who are actually fraudulent is equal across groups. Equalised odds requires that the true positive rate and false positive rate are equal across groups. If base rates of fraud differ across groups — as they inevitably do due to structural socioeconomic factors — these criteria cannot be simultaneously satisfied. A system that achieves demographic parity will necessarily have different false positive rates across groups; a system that equalises false positive rates will necessarily flag different proportions from each group.
The implications are profound. When governments deploy AI systems that embed specific fairness trade-offs, they are making political choices about whose interests to prioritise — choices that should be made through democratic deliberation, not buried in technical specifications. Yet current procurement processes treat fairness as a technical requirement to be assessed by engineers rather than a normative choice to be debated by legislators and stakeholders.
Audit Mechanisms: From Post-Hoc to Continuous
The EU AI Act, which entered into force in 2025, mandates conformity assessments for "high-risk" AI systems — a category that includes most public sector deployments. However, the Act's conformity assessment framework is fundamentally a pre-deployment gate: systems are evaluated before deployment, with periodic reviews thereafter. This approach is inadequate for AI systems whose behaviour changes over time due to data drift, feedback loops, and environmental shifts.
We propose a transition from pre-deployment conformity assessment to continuous algorithmic auditing. Drawing on financial audit methodology — which combines ex-ante controls, continuous monitoring, and ex-post investigation — a comprehensive algorithmic audit framework would include three components:
1. Design-stage impact assessments. Before development begins, a structured assessment of the proposed system's impact on fundamental rights, with mandatory public consultation for high-stakes domains. This corresponds to the "environmental impact assessment" model used in infrastructure development and, increasingly, for data protection (DPIAs under GDPR).
2. Continuous monitoring dashboards. Real-time tracking of system outputs disaggregated by protected characteristics, with automated alerts when fairness metrics diverge beyond pre-defined thresholds. The UK's Alan Turing Institute has developed open-source tools for this purpose, and Canada's Algorithmic Impact Assessment framework provides a governance template.
3. Independent retrospective audits. Periodic (annual or biannual) independent audits by qualified external bodies, with findings published publicly and rights of response for affected communities. The audit methodology should include both statistical analysis of system outputs and qualitative investigation of individual cases, ensuring that aggregate fairness metrics do not obscure individual harms.
Institutional Design: Who Governs the Algorithms?
Effective algorithmic accountability requires dedicated institutional capacity. Existing regulatory bodies — data protection authorities, equality commissions, administrative law tribunals — lack the technical expertise and legal mandates to oversee AI systems effectively. Several jurisdictions are experimenting with new institutional models.
France's CNIL has established a dedicated AI team and published guidance on algorithmic accountability in public services. The Netherlands — responding directly to the toeslagenaffaire — created an Algorithm Authority (Algoritmewaakhond) with powers to investigate and halt government AI systems. The UK's AI Safety Institute, while focused on frontier models, has begun extending its evaluation capabilities to public sector systems. Brazil's proposed AI regulation includes provisions for a National AI Authority with oversight powers.
The institutional design challenge involves balancing technical expertise with democratic legitimacy. A purely technical body risks reproducing the accountability deficit it was created to address — substituting one set of unaccountable experts for another. A purely political body may lack the technical capacity to evaluate complex AI systems. The optimal design likely involves a hybrid structure: technically capable staff operating under governance boards that include civil society representatives, affected community members, and elected officials.
Remedies and Rights: Meaningful Contestation of Algorithmic Decisions
The right to contest automated decisions — enshrined in Article 22 of the GDPR and equivalent provisions in other data protection regimes — is largely theoretical in practice. Meaningful contestation requires that affected individuals: (a) know that an algorithmic system was involved in the decision; (b) understand the factors that influenced the outcome; (c) have access to an effective remedy; and (d) are able to challenge systemic rather than merely individual errors.
Each of these requirements faces practical obstacles. Notification is inconsistent: many government AI systems operate without informing affected individuals. Explanation is technically challenging: post-hoc interpretability tools provide approximations, not true causal explanations. Remedies are often limited to requesting human review — but human reviewers exhibit "automation bias," deferring to algorithmic outputs in 85–95% of cases according to a 2024 meta-analysis published in Nature Human Behaviour. Systemic challenge requires evidence of pattern discrimination that individual complainants rarely possess.
Addressing these obstacles requires legal and institutional innovation. Mandatory algorithmic impact registers — public databases disclosing all government AI systems, their purposes, and their evaluation results — would address the notification gap. The Netherlands, France, and the UK have piloted such registers. Collective redress mechanisms — enabling civil society organisations to bring challenges on behalf of affected groups — would address the systemic challenge gap. And shifting the burden of proof — requiring the deploying agency to demonstrate that its AI system complies with fairness and accuracy standards, rather than requiring the individual to prove it does not — would make the right to contest practically meaningful.
Implications for GDEF's Regulation & Policy Working Group
Public sector AI accountability sits at the intersection of technology governance, human rights, and democratic legitimacy. The governance frameworks needed are not merely technical standards but constitutional innovations that embed algorithmic systems within democratic accountability structures. GDEF's Regulation & Policy Working Group will advance recommendations on algorithmic accountability frameworks in its programme of work, with particular focus on the design of continuous audit mechanisms and the institutional architecture for algorithmic oversight.
References & Sources
- OECD, AI Policy Observatory: Government AI Readiness Index 2025. oecd.ai
- AI Now Institute, Annual Report 2025: AI in Government. ainowinstitute.org
- European Commission, AI Act Implementation: High-Risk AI Systems in Public Administration. ec.europa.eu/ai-act
- Arrow, K.J. (1951). Social Choice and Individual Values. New York: Wiley. doi.org/10.12987/9780300186987
- Chouldechova, A. (2017). "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." Big Data, 5(2), 153–163. doi.org/10.1089/big.2016.0047
- UK Alan Turing Institute, AI Standards Hub: Public Sector AI Governance. aistandardshub.org
- Parliamentary Committee on Digital Affairs (Netherlands), Report on the Toeslagenaffaire: Lessons for Algorithmic Governance, 2023. tweedekamer.nl
- Treasury Board of Canada, Algorithmic Impact Assessment Tool. canada.ca/responsible-ai
- Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). "Inherent Trade-Offs in the Fair Determination of Risk Scores." Proceedings of the 8th Conference on Innovations in Theoretical Computer Science. doi.org/10.4230/LIPIcs.ITCS.2017.43
- Royal Commission into the Robodebt Scheme (Australia), Final Report, 2023. robodebt.royalcommission.gov.au