Skip to Content
logologo
AI Incident Database
Open TwitterOpen RSS FeedOpen FacebookOpen LinkedInOpen GitHub
Open Menu
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse

Incident 611: UK Government AI Allegedly Targets Disproportionate Numbers of Certain Nationals for Fraud Review

Description: The UK's Department for Work and Pensions (DWP) faced scrutiny after many Bulgarian nationals reported unexplained suspensions of their Universal Credit benefits. The MP for Edmonton raised concerns about potential nationality-based targeting for benefit fraud investigations, leading to poverty and homelessness among affected individuals. The Home Office's own equality impact assessment found it was flagging a disproportionate number of marriages from Greece, Albania, Bulgaria and Romania.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History

Entities

View all entities
Alleged: Home Office , Department for Work and Pensions and British government developed an AI system deployed by Various British government offices , Home Office , Department for Work and Pensions and British government, which harmed Romanians in the United Kingdom , Greeks in the United Kingdom , Bulgarians in the United Kingdom , British public and Albanians in the United Kingdom.

Incident Stats

Incident ID
611
Report Count
13
Incident Date
2021-12-06
Editors
Applied Taxonomies
MIT

MIT Taxonomy Classifications

Machine-Classified
Taxonomy Details

Risk Subdomain

A further 23 subdomains create an accessible and understandable classification of hazards and harms associated with AI
 

1.1. Unfair discrimination and misrepresentation

Risk Domain

The Domain Taxonomy of AI Risks classifies risks into seven AI risk domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental harms, and (7) AI system safety, failures & limitations.
 
  1. Discrimination and Toxicity

Entity

Which, if any, entity is presented as the main cause of the risk
 

Human

Timing

The stage in the AI lifecycle at which the risk is presented as occurring
 

Post-deployment

Intent

Whether the risk is presented as occurring as an expected or unexpected outcome from pursuing a goal
 

Intentional

Incident Reports

Reports Timeline

+1
MP raises concern over Bulgarian nationals’ UK benefit suspensions
+3
DWP ramps up investment in AI models to detect fraud despite algorithmic bias concerns
UK warned over lack of transparency on use of AI to vet welfare claims+6
DWP’s fraud and error checking AI still displaying signs of bias
MP raises concern over Bulgarian nationals’ UK benefit suspensions

MP raises concern over Bulgarian nationals’ UK benefit suspensions

theguardian.com

DWP ramps up investment in AI models to detect fraud despite algorithmic bias concerns

DWP ramps up investment in AI models to detect fraud despite algorithmic bias concerns

techmonitor.ai

AI use widened to assess universal credit applications and tackle fraud

AI use widened to assess universal credit applications and tackle fraud

theguardian.com

Universal Credit: Warnings over AI use to risk-score benefit claims

Universal Credit: Warnings over AI use to risk-score benefit claims

bbc.com

DWP Widening Use of Anti-fraud Tech Amid Concerns From Campaign Groups

DWP Widening Use of Anti-fraud Tech Amid Concerns From Campaign Groups

digit.fyi

UK warned over lack of transparency on use of AI to vet welfare claims

UK warned over lack of transparency on use of AI to vet welfare claims

theguardian.com

DWP’s fraud and error checking AI still displaying signs of bias

DWP’s fraud and error checking AI still displaying signs of bias

techmonitor.ai

UK risks scandal over ‘bias’ in AI tools in use across public sector

UK risks scandal over ‘bias’ in AI tools in use across public sector

theguardian.com

UK government using AI to decide benefit payments and spot sham marriages, investigation reveals

UK government using AI to decide benefit payments and spot sham marriages, investigation reveals

uk.news.yahoo.com

AI employed for crucial decisions in over 8 Whitehall departments

AI employed for crucial decisions in over 8 Whitehall departments

interestingengineering.com

Decisions on benefits and migrants ‘being made by AI’

Decisions on benefits and migrants ‘being made by AI’

thetimes.co.uk

UK govt using AI to decide benefits for people, marriage licenses: Report

UK govt using AI to decide benefits for people, marriage licenses: Report

hindustantimes.com

Revealed: Government using AI to decide on benefits and driving licences

Revealed: Government using AI to decide on benefits and driving licences

independent.co.uk

MP raises concern over Bulgarian nationals’ UK benefit suspensions
theguardian.com · 2021

Dozens of Bulgarian nationals have been tipped into poverty after unexplained benefit suspensions following the creation of a “risk review” unit in the Department for Work and Pensions, an MP has said.

Kate Osamor, the Labour MP for Edmonto…

DWP ramps up investment in AI models to detect fraud despite algorithmic bias concerns
techmonitor.ai · 2023

The Department for Work and Pensions (DWP) is ploughing £70m into its digital transformation projects over the next three years. However, the National Audit Office (NAO) has raised concerns about investments the department is making in arti…

AI use widened to assess universal credit applications and tackle fraud
theguardian.com · 2023

The government has widened its deployment of artificial intelligence to uncover welfare fraud, despite warnings of algorithmic bias against groups of vulnerable claimants.

In a £70m investment applying “advanced analytics” to requests for u…

Universal Credit: Warnings over AI use to risk-score benefit claims
bbc.com · 2023

The Department of Work and Pensions (DWP) has revealed plans to widen its use of the technology to tackle fraud.

Campaigners say more information is needed to ensure the system does not make biased referrals for benefit investigations.

The …

DWP Widening Use of Anti-fraud Tech Amid Concerns From Campaign Groups
digit.fyi · 2023

In a recent National Audit Office (NAO) report concerning the Department for Work and Pensions (DWP) financial accounts, it has come to light that the DWP is expanding its use of machine learning for identifying potential benefits fraud.

Si…

UK warned over lack of transparency on use of AI to vet welfare claims
theguardian.com · 2023

The UK government risks contempt of court unless it improves its response to requests for transparency over the use of artificial intelligence (AI) to vet welfare claims, the information commissioner has said.

Over the past two years, the D…

DWP’s fraud and error checking AI still displaying signs of bias
techmonitor.ai · 2023

Senior civil servants told a committee of MPs that an AI being developed by the Department for Work and Pensions (DWP) to detect fraud and error is still in the learning stages of its development and is exhibiting bias against some claimant…

UK risks scandal over ‘bias’ in AI tools in use across public sector
theguardian.com · 2023

Kate Osamor, the Labour MP for Edmonton, recently received an email from a charity about a constituent of hers who had had her benefits suspended apparently without reason.

“For well over a year now she has been trying to contact DWP [the D…

UK government using AI to decide benefit payments and spot sham marriages, investigation reveals
uk.news.yahoo.com · 2023

The government's use of artificial intelligence (AI) risks producing discriminatory results against benefit claimants and ethnic minorities, an investigation has found.

A total of eight Whitehall departments and some police forces are using…

AI employed for crucial decisions in over 8 Whitehall departments
interestingengineering.com · 2023

In the past, artificial intelligence (AI) use in public services has caused uproars. For instance, in the Netherlands, tax authorities used technology to find fraud but made many mistakes. This resulted in a hefty fine of €3.7m, pushing tho…

Decisions on benefits and migrants ‘being made by AI’
thetimes.co.uk · 2023

Civil servants in at least eight government departments and the police are using artificial intelligence to make decisions on welfare, immigration and criminal justice, it has been claimed.

The Guardian reported that the systems were being …

UK govt using AI to decide benefits for people, marriage licenses: Report
hindustantimes.com · 2023

The report made findings of some potentially discriminatory results made by the artificial intelligence.

The UK government officials are using artificial intelligence (AI) and complex algorithms to help make various decisions including thos…

Revealed: Government using AI to decide on benefits and driving licences
independent.co.uk · 2023

Government officials are using artificial intelligence (AI) to help decide on everything from approving marriage licences to who should be given benefits, it has been revealed.

Civil servants in at least eight Whitehall departments, and sev…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

Selected by our editors
Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud

Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud

Jun 2024 · 5 reports
By textual similarity

Did our AI mess up? Flag the unrelated incidents

Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities

Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities

Oct 2019 · 6 reports
Australian Automated Debt Assessment System Issued False Notices to Thousands

Australian Automated Debt Assessment System Issued False Notices to Thousands

Jul 2015 · 39 reports
Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Jul 2017 · 6 reports
Previous IncidentNext Incident

Similar Incidents

Selected by our editors
Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud

Department for Work and Pensions (DWP) Algorithm Wrongly Flags 200,000 for Housing Benefit Fraud

Jun 2024 · 5 reports
By textual similarity

Did our AI mess up? Flag the unrelated incidents

Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities

Opaque Fraud Detection Algorithm by the UK’s Department of Work and Pensions Allegedly Discriminated against People with Disabilities

Oct 2019 · 6 reports
Australian Automated Debt Assessment System Issued False Notices to Thousands

Australian Automated Debt Assessment System Issued False Notices to Thousands

Jul 2015 · 39 reports
Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Airbnb's Trustworthiness Algorithm Allegedly Banned Users without Explanation, and Discriminated against Sex Workers

Jul 2017 · 6 reports

Research

  • Defining an “AI Incident”
  • Defining an “AI Incident Response”
  • Database Roadmap
  • Related Work
  • Download Complete Database

Project and Community

  • About
  • Contact and Follow
  • Apps and Summaries
  • Editor’s Guide

Incidents

  • All Incidents in List Form
  • Flagged Incidents
  • Submission Queue
  • Classifications View
  • Taxonomies

2023 - AI Incident Database

  • Terms of use
  • Privacy Policy
  • Open twitterOpen githubOpen rssOpen facebookOpen linkedin
  • 8b8f151