Incident 146: Research Prototype AI, Delphi, Reportedly Gave Racially Biased Answers on Ethics

Description: A publicly accessible research model that was trained via Reddit threads showed racially biased advice on moral dilemmas, allegedly demonstrating limitations of language-based models trained on moral judgments.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History

Entities

View all entities
Alleged: Allen Institute for AI developed and deployed an AI system, which harmed Minority Groups.

Incident Stats

Incident ID
146
Report Count
3
Incident Date
2021-10-22
Editors
Sean McGregor, Khoa Lam
Applied Taxonomies
GMF, CSETv1, MIT

CSETv1 Taxonomy Classifications

Taxonomy Details

Incident Number

146

GMF Taxonomy Classifications

Taxonomy Details

Known AI Goal Snippets

(Snippet Text: You can pose any question you like and be sure to receive an answer, wrapped in the authority of the algorithm rather than the soothsayer., Related Classifications: Question Answering)

MIT Taxonomy Classifications

Machine-Classified
Taxonomy Details

Risk Subdomain

7.3. Lack of capability or robustness

Risk Domain

  1. AI system safety, failures, and limitations

Entity

AI

Timing

Post-deployment

Intent

Unintentional

The AI oracle of Delphi uses the problems of Reddit to offer dubious moral advice
theverge.com · 2021

Got a moral quandary you don’t know how to solve? Fancy making it worse? Why not turn to the wisdom of artificial intelligence, aka Ask Delphi: an intriguing research project from the Allen Institute for AI that offers answers to ethical di…

Scientists Built an AI to Give Ethical Advice, But It Turned Out Super Racist
futurism.com · 2021

We’ve all been in situations where we had to make tough ethical decisions. Why not dodge that pesky responsibility by outsourcing the choice to a machine learning algorithm?

That’s the idea behind Ask Delphi, a machine-learning model from t…

Can a Machine Learn Morality?
nytimes.com · 2021

Researchers at an artificial intelligence lab in Seattle called the Allen Institute for AI unveiled new technology last month that was designed to make moral judgments. They called it Delphi, after the religious oracle consulted by the anci…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

By textual similarity

Did our AI mess up? Flag the unrelated incidents

High-Toxicity Assessed on Text Involving Women and Minority Groups

High-Toxicity Assessed on Text Involving Women and Minority Groups

· 9 reports
Inappropriate Gmail Smart Reply Suggestions

Inappropriate Gmail Smart Reply Suggestions

· 22 reports
All Image Captions Produced are Violent

All Image Captions Produced are Violent

· 28 reports
Previous IncidentNext Incident