Malicious individuals are misusing generative AI music tools to produce songs filled with homophobia, racism, and propaganda, while also disseminating guides for others to do the same.
ActiveFence, a service dedicated to trust and safety management on online platforms, has reported a notable increase in discussions within hate speech-related communities since March, focusing on exploiting AI music creation tools to craft offensive songs targeting minority groups. According to ActiveFence researchers, these AI-generated songs circulating in various forums and discussion boards aim to incite hatred against ethnic, gender, racial, and religious groups, glorifying acts of martyrdom, self-harm, and terrorism.
While hateful and harmful songs are not a new phenomenon, the concern now is that user-friendly, free music-generating tools could enable a wider audience to produce such content at scale. This situation parallels the proliferation of misinformation, disinformation, and hate speech facilitated by advancements in image, voice, video, and text generators.
“These trends are becoming more pronounced as more users learn how to generate and share these offensive songs,” Noam Schwartz, co-founder and CEO of ActiveFence, shared in an interview with TechCrunch. “Threat actors are rapidly identifying specific vulnerabilities to exploit these platforms and create malicious content.”
Creating “Hate” Songs
Generative AI music tools such as Udio and Suno allow users to incorporate custom lyrics into generated songs. Despite the platforms’ safeguards to filter out common slurs and pejoratives, users have discovered ways to circumvent these filters, according to ActiveFence.
For instance, the report highlights how users in white supremacist forums devised phonetic spellings of minorities and offensive terms, such as using “jooz” instead of “Jews” and “say tan” instead of “Satan,” to bypass content filters. Additionally, they suggested altering spacings and spellings for violent acts, like changing “my rape” to “mire ape.”
TechCrunch tested these workarounds on Udio and Suno, two popular tools for AI-generated music. Suno permitted all of them, whereas Udio successfully blocked some but not all of the offensive homophones.
A Udio spokesperson, reached via email, confirmed that the company forbids hate speech on its platform. Suno did not respond to requests for comment.
In their research, ActiveFence found AI-generated songs promoting conspiracy theories about Jewish people and advocating for their mass extermination, songs with slogans from ISIS and Al-Qaeda, and songs glorifying sexual violence against women.
Impact of Songs
Schwartz argues that songs, as opposed to text, have an emotional impact that makes them a powerful tool for hate groups and political manipulation. He cites the Rock Against Communism movement in the late ’70s and early ’80s in the UK, which led to the creation of antisemitic and racist “hatecore” music genres.
“AI enhances the appeal of harmful content — imagine someone spreading a harmful narrative about a specific population, then imagine them creating a catchy song that everyone can sing and remember,” he explained. “These songs reinforce group solidarity, indoctrinate peripheral group members, and are also used to shock and offend casual internet users.”
Schwartz urges music generation platforms to implement preventive measures and conduct thorough safety evaluations. “Red teaming might reveal some of these vulnerabilities and can be done by simulating the behavior of threat actors,” Schwartz suggested. “Better moderation of both input and output could help block malicious content before it reaches users.”
However, these solutions might be temporary as users continue to find new ways to evade moderation. Some AI-generated terrorist propaganda songs identified by ActiveFence were created using Arabic-language euphemisms and transliterations, which the music generators’ filters failed to detect due to presumably weaker Arabic language safeguards.
If AI-generated hateful music follows the trajectory of other AI-produced media, it could see widespread dissemination. Earlier this year, Wired reported that an AI-manipulated video of Adolf Hitler garnered over 15 million views on X, shared by a far-right conspiracy influencer.
Concerns about the amplification of racist, antisemitic, Islamophobic, and xenophobic content through generative AI have been echoed by various experts, including a UN advisory body.
“Generative AI services empower users without significant resources or creative and technical skills to produce engaging content and spread competitive ideas globally,” Schwartz concluded. “Having recognized the creative potential of these new services, threat actors are now working to bypass moderation and avoid detection with considerable success.”
