Skip to Content
logologo
AI Incident Database
Open TwitterOpen RSS FeedOpen FacebookOpen LinkedInOpen GitHub
Open Menu
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse
Discover
Submit
  • Welcome to the AIID
  • Discover Incidents
  • Spatial View
  • Table View
  • List view
  • Entities
  • Taxonomies
  • Submit Incident Reports
  • Submission Leaderboard
  • Blog
  • AI News Digest
  • Risk Checklists
  • Random Incident
  • Sign Up
Collapse

Incident 810: TikTok Network Amplifies AI-Generated Nazi Propaganda and Hate Speech

Description: A coordinated neo-Nazi network on TikTok used AI-generated media, including Hitler speeches, to spread Nazi propaganda and extremist content, violating TikTok’s hate speech policies. The network evaded platform moderation through coded language, imagery, and music, with some accounts accumulating millions of views. TikTok’s algorithm further amplified the reach of this content, despite community guidelines.
Editor Notes: Reconstructing the timeline of events: On February 6, 2023, an account with violent antisemitic content calling for armed revolution began posting on TikTok. In May 2024, the Institute for Strategic Dialogue (ISD) created two dummy accounts to track how TikTok's algorithm recommended pro-Nazi content. By June 3, 2024, ISD had identified a Nazi account that had accumulated over 87,000 views before being banned. Despite this, on June 6, 2024, a heavily extremist pro-Nazi account remained active, recruiting members to off-platform groups. Finally, on July 29, 2024, ISD published its full report, revealing a coordinated network of over 200 accounts promoting Nazism and extremist content, some of which used AI-generated media to bypass TikTok’s moderation efforts. For this incident ID, I am taking the ISD's date of publication for their findings as the incident date. Note on "NazTok": The ISD report uses the term "NazTok" in their title for the report as a shorthand for "Nazi TikTok" while not explicitly defining their use of this term beyond that inferred meaning. I am replicating "NazTok" in the Deployers field to tie likely new incidents that will emerge together in this particular genre of incident. I have retroactively applied "NazTok" to Incident 809's Deployer field too.

Tools

New ReportNew ReportNew ResponseNew ResponseDiscoverDiscoverView HistoryView History

Entities

View all entities
Alleged: Pro-Nazi TikTok accounts developed an AI system deployed by Unknown AI developers, which harmed TikTok users.

Incident Stats

Incident ID
810
Report Count
1
Incident Date
2024-07-29
Editors
Applied Taxonomies
MIT

MIT Taxonomy Classifications

Machine-Classified
Taxonomy Details

Risk Subdomain

A further 23 subdomains create an accessible and understandable classification of hazards and harms associated with AI
 

1.2. Exposure to toxic content

Risk Domain

The Domain Taxonomy of AI Risks classifies risks into seven AI risk domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental harms, and (7) AI system safety, failures & limitations.
 
  1. Discrimination and Toxicity

Entity

Which, if any, entity is presented as the main cause of the risk
 

AI

Timing

The stage in the AI lifecycle at which the risk is presented as occurring
 

Post-deployment

Intent

Whether the risk is presented as occurring as an expected or unexpected outcome from pursuing a goal
 

Unintentional

Incident Reports

Reports Timeline

+1
NazTok: An organized neo-Nazi TikTok network is getting millions of views
NazTok: An organized neo-Nazi TikTok network is getting millions of views

NazTok: An organized neo-Nazi TikTok network is getting millions of views

isdglobal.org

NazTok: An organized neo-Nazi TikTok network is getting millions of views
isdglobal.org · 2024

Self-identified Nazis are openly promoting hate speech and real-world recruitment on TikTok. Not only is the platform failing to remove these videos and accounts, but its algorithm is amplifying their reach.

Content warning: This report inc…

Variants

A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.

Similar Incidents

Selected by our editors
TikTok Hosts AI-Generated English-Language Hitler Speeches with Millions of Views

TikTok Hosts AI-Generated English-Language Hitler Speeches with Millions of Views

Apr 2024 · 1 report
By textual similarity

Did our AI mess up? Flag the unrelated incidents

Google’s YouTube Kids App Presents Inappropriate Content

Google’s YouTube Kids App Presents Inappropriate Content

May 2015 · 14 reports
DALL-E 2 Reported for Gender and Racially Biased Outputs

DALL-E 2 Reported for Gender and Racially Biased Outputs

Apr 2022 · 3 reports
Amazon Censors Gay Books

Amazon Censors Gay Books

May 2008 · 24 reports
Previous IncidentNext Incident

Similar Incidents

Selected by our editors
TikTok Hosts AI-Generated English-Language Hitler Speeches with Millions of Views

TikTok Hosts AI-Generated English-Language Hitler Speeches with Millions of Views

Apr 2024 · 1 report
By textual similarity

Did our AI mess up? Flag the unrelated incidents

Google’s YouTube Kids App Presents Inappropriate Content

Google’s YouTube Kids App Presents Inappropriate Content

May 2015 · 14 reports
DALL-E 2 Reported for Gender and Racially Biased Outputs

DALL-E 2 Reported for Gender and Racially Biased Outputs

Apr 2022 · 3 reports
Amazon Censors Gay Books

Amazon Censors Gay Books

May 2008 · 24 reports

Research

  • Defining an “AI Incident”
  • Defining an “AI Incident Response”
  • Database Roadmap
  • Related Work
  • Download Complete Database

Project and Community

  • About
  • Contact and Follow
  • Apps and Summaries
  • Editor’s Guide

Incidents

  • All Incidents in List Form
  • Flagged Incidents
  • Submission Queue
  • Classifications View
  • Taxonomies

2023 - AI Incident Database

  • Terms of use
  • Privacy Policy
  • Open twitterOpen githubOpen rssOpen facebookOpen linkedin
  • 30ebe76