Techdirt's think tank, the Copia Institute, is working with the Trust & Safety Professional Association and its sister organization, the Trust & Safety Foundation, to produce an ongoing series of case studies about content moderation decisions. These case studies are presented in a neutral fashion, not aiming to criticize or applaud any particular decision, but to highlight the many different challenges that content moderators face and the tradeoffs they result in. Find more case studies here on Techdirt and on the TSF website.

Content Moderation Case Studies: Snapchat Disables GIPHY Integration After Racist 'Sticker' Is Discovered (2018)

from the sticker-moderation dept

Summary: Snapchat debuted to immediate success a decade ago, drawing in millions of users with its playful take on instant messaging that combined photos and short videos with a large selection of filters and “stickers.” Stickers are graphics that can be applied to messages, allowing users to punch up their presentations (so to speak).

Snapchat’s innovations in the messaging space proved incredibly popular, moving Snapchat from upstart to major player in a few short years. It also created more headaches for moderators as sent messages soared past millions per day to billions.

Continuing its expansion of user options, Snapchat announced its integration with Giphy, a large online repository of GIFs, in February 2018. This gave users access to Giphy’s library of images to use as stickers in messages.

But the addition of thousands of images to billions of messages quickly resulted in an unforeseen problem. In early March of 2018, Snapchat users reported a search of the GIPHY image database for the word “crime” surfaced a racist sticker, as reported by Josh Constine for TechCrunch:

“We first reported Instagram was building a GIPHY integration back in January before it launched a week later, with Snapchat adding a similar feature in February. But it wasn’t long before things went wrong. First spotted by a user in the U.K. around March 8th, the GIF included a racial slur.” — Josh Constine, TechCrunch

Both platforms immediately pulled the plug on the integration while they sorted things out with GIPHY.

Company Considerations:

  • What measures can be put in place to prevent moderation problems from moving from one platform to another during cross-platform integration?
  • What steps should be taken prior to launch to integrate moderation efforts between platforms? 
  • What can “upline” content providers do to ensure content moving from their platforms to others meets the content standards of the “downline” platforms? 

Issue Considerations:

  • What procedures aid in facilitating cross-platform moderation? 
  • Which party should have final say on moderation efforts, the content provider or the content user?

ResolutionInstagram was the first to reinstate its connection with GIPHY, promising to use more moderators to examine incoming content from the image site:

“We’ve been in close contact with GIPHY throughout this process and we’re confident that they have put measures in place to ensure that Instagram users have a good experience” an Instagram spokesperson told TechCrunch.

GIPHY offered its own apology for the racist image, blaming the slipup on a bug in its filters. Here’s what GIPHY’s spokesperson told Gizmodo:

After investigation of the incident, this sticker was available due to a bug in our content moderation filters specifically affecting GIF stickers.

We have fixed the bug and have re-moderated all of the GIF stickers in our library.

The GIPHY staff is also further reviewing every GIF sticker by hand and should be finished shortly.

Snapchat was the last to reinstate its connection to GIPHY, stating it was working directly with the site to revamp both moderation systems to ensure offensive content would be prevented from being uploaded to GIPHY and/or making the leap to connected social media services.

Originally published to the Trust & Safety Foundation website.

Filed Under: , , , ,
Companies: giphy, snapchat

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Content Moderation Case Studies: Snapchat Disables GIPHY Integration After Racist 'Sticker' Is Discovered (2018)”

Subscribe: RSS Leave a comment
18 Comments
Anonymoussays:

Re:

Emojis are even more limited, even those which are not part of unicode. You only get what is implemented.

GIFs and stickers run in the millions. If you can’t find one that says what you want it to say, use your own bloody words or choose your image to do so.

If you like being a dick, you might get moderated by the service which is someone else’s private property. They aren’t going to be told by you what is approved for expression, either.

Scary Devil Monasterysays:

Well, it doesn't exactly come as a surprise.

Since there’s always that one dick or douche around who gets their rocks off by being an asshole no platform can really afford to allow too open personalization.

This is why we can’t have nice things.

In more cynical moments I’m inclined to the belief that we have, as a society, become too permissive when people whose only hobby is to undermine others are never sanctioned beyond, at most, a "You aren’t welcome here". It’s shit like in the OP, writ large, which makes me wonder if it’ll eventually turn out that China is on to something with their "social rating scores".

Because when you look at the sheer amount of incentivized assholery among those 70-90 million americans still subscribing to Qanon and Dear Leader it sure does make you wonder if humanity is able to handle such a high ideal as "freedom". It always seems to result mainly in large amounts of people clamoring for their "rights" to be shitty unto other people.

Surely there has to be something society can do to rid itself of or push out persistent douchebags from their midst without invoking government power? How come when angry assholes migrate in herds led by a single voice liberal society just meanders around without any form of plan?

Rockysays:

Re: Re: Well, it doesn't exactly come as a surprise.

"Push out" in this context is comparable to kicking out an odious asshole from your own apartment. If you somehow think that’s not liberal, congratulations, it isn’t, it’s just people taking action removing assholes from their presence. If you find the whole concept disturbing, you are free to associate with assholes all you want.

PaulTsays:

Re: Re: Well, it doesn't exactly come as a surprise.

"I don’t know what kind of society you mean that would "push out" (to where?) people you don’t like, but it isn’t a liberal one"

What’s worse – people who are violently opposed to "liberals" being told to go elsewhere among people who they do agree with, or them remaining and disrupting the communities everyone else wants to be part of?

The fact that a society composed of only people like them would be a violent cesspool of hate and loathing should not be a factor in the rest of us deciding if we want to welcome them into the doors of our own society.

Scary Devil Monasterysays:

Re: Re: Well, it doesn't exactly come as a surprise.

"I don’t know what kind of society you mean that would "push out" (to where?) people you don’t like, but it isn’t a liberal one."

Go read up on Karl Popper’s "Paradox of Tolerance". If you have a liberal society and allow those who aren’t liberal to be part of it, essentially your society just drank poison. Because intolerant people will always end up with disproportionate power simply because they have the incentive to beat down and kick out the other whereas liberals are far more inclined to, sadly, tolerate those who are intolerant.

I think one Niemöller is enough.

"I find that thought deeply disturbing."

And I find it disturbing that more people aren’t having those thoughts. The US was founded by people who would not tolerate oppression and authoritarians. To the point of mounting a bloody revolution.

Yet today when an alt-right horde marches up the street flying the banners of nazi germany and the old slaver states, most liberals just stay inside and shrug. They don’t go out, meet that march halfway, and block its path, showing those benighted assholes unambiguously that their hatred is a minority opinion for which contemporary society has no place.

Because you people failed to forcefully and persistently tell racists and bigots to shove it, today 30% of the US belongs to that camp. Hitler had to make do with 12% when he took power and the last days of Trump-in-office came close to being a rinse-repeat of history with the one saving grace probably being that unlike Hitler Trump was blatantly obvious in his contempt visavi the military.

If you want to have and keep a liberal society then you are going to have to fight for it. And that struggle never ends, because there will always be those who want to use discontent, envy and bigotry to propel themselves into high office.

Jeffersson and Washington knew this. You all have forgotten it.

I invite you – cordially – to tell us all how it can be "liberal" to encourage those who would dismantle liberalism to take a seat at the table? How can it be liberal to build a society which has a core belief in base values and then invite those who reject not only that society but every value it was founded on?

Normally I have to read some passage from the text I’m about to quote to republicans who, despite always hollering about it, appear to never have read it;

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.

The opposite also follows. Those who are incapable of aligning with this very simple sentence are in the wrong country.

Those people whining about "censorship" because they’re not allowed to use the N-word in open forums? The very fact that they feel compelled to use that word means that they reject the most basic rules of the society they live in.

And if you stand to defend them or their "right" to be assholes in the mistaken assumption that this is the liberal thing to do…then you’ve abandoned liberal ideology because what you really are doing is to defend their privilege of being assholes to someone else.

Malcolm X was not a decent person but he did have a few things correct and he was dead right when he made the claim that he feared the white liberal more than the nakedly aggressive conservative. For in the end the american liberal always ends up defending the right of the conservative to undermine the worth and value of the other – in the sacred name of "personal opinion" and the mistaken belief that every opinion is worthy of protection.

It’s not, and both your founding fathers and any nation with a national charter ensconcing free speech knew this.

PaulTsays:

Re:

Yeah, pretty much. If you allow people to post things via a 3rd party and the 3rd party’s promised filters aren’t working as required, it’s better to remove that feature than put up with the PR fallout from people blaming you directly.

Talking of overreactions, we can point to your comment, though – "everything" was not shut down, only the feature that used a 3rd party as a bonus feature whose promised filtering wasn’t working properly. People using Snapchat without also using giphy would not have noticed any change.

Anonymoussays:

Re:

Emojis are even more limited, even those which are not part of unicode. You only get what is implemented.

GIFs and stickers run in the millions. If you can’t find one that says what you want it to say, use your own bloody words or choose your image to do so.

If you like being a dick, you might get moderated by the service which is someone else’s private property. They aren’t going to be told by you what is approved for expression, either.

Rockysays:

Re: Re: Well, it doesn't exactly come as a surprise.

"Push out" in this context is comparable to kicking out an odious asshole from your own apartment. If you somehow think that’s not liberal, congratulations, it isn’t, it’s just people taking action removing assholes from their presence. If you find the whole concept disturbing, you are free to associate with assholes all you want.

PaulTsays:

Re: Re: Well, it doesn't exactly come as a surprise.

"I don’t know what kind of society you mean that would "push out" (to where?) people you don’t like, but it isn’t a liberal one"

What’s worse – people who are violently opposed to "liberals" being told to go elsewhere among people who they do agree with, or them remaining and disrupting the communities everyone else wants to be part of?

The fact that a society composed of only people like them would be a violent cesspool of hate and loathing should not be a factor in the rest of us deciding if we want to welcome them into the doors of our own society.

Scary Devil Monasterysays:

Re: Re: Well, it doesn't exactly come as a surprise.

"I don’t know what kind of society you mean that would "push out" (to where?) people you don’t like, but it isn’t a liberal one."

Go read up on Karl Popper’s "Paradox of Tolerance". If you have a liberal society and allow those who aren’t liberal to be part of it, essentially your society just drank poison. Because intolerant people will always end up with disproportionate power simply because they have the incentive to beat down and kick out the other whereas liberals are far more inclined to, sadly, tolerate those who are intolerant.

I think one Niemöller is enough.

"I find that thought deeply disturbing."

And I find it disturbing that more people aren’t having those thoughts. The US was founded by people who would not tolerate oppression and authoritarians. To the point of mounting a bloody revolution.

Yet today when an alt-right horde marches up the street flying the banners of nazi germany and the old slaver states, most liberals just stay inside and shrug. They don’t go out, meet that march halfway, and block its path, showing those benighted assholes unambiguously that their hatred is a minority opinion for which contemporary society has no place.

Because you people failed to forcefully and persistently tell racists and bigots to shove it, today 30% of the US belongs to that camp. Hitler had to make do with 12% when he took power and the last days of Trump-in-office came close to being a rinse-repeat of history with the one saving grace probably being that unlike Hitler Trump was blatantly obvious in his contempt visavi the military.

If you want to have and keep a liberal society then you are going to have to fight for it. And that struggle never ends, because there will always be those who want to use discontent, envy and bigotry to propel themselves into high office.

Jeffersson and Washington knew this. You all have forgotten it.

I invite you – cordially – to tell us all how it can be "liberal" to encourage those who would dismantle liberalism to take a seat at the table? How can it be liberal to build a society which has a core belief in base values and then invite those who reject not only that society but every value it was founded on?

Normally I have to read some passage from the text I’m about to quote to republicans who, despite always hollering about it, appear to never have read it;

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness.

The opposite also follows. Those who are incapable of aligning with this very simple sentence are in the wrong country.

Those people whining about "censorship" because they’re not allowed to use the N-word in open forums? The very fact that they feel compelled to use that word means that they reject the most basic rules of the society they live in.

And if you stand to defend them or their "right" to be assholes in the mistaken assumption that this is the liberal thing to do…then you’ve abandoned liberal ideology because what you really are doing is to defend their privilege of being assholes to someone else.

Malcolm X was not a decent person but he did have a few things correct and he was dead right when he made the claim that he feared the white liberal more than the nakedly aggressive conservative. For in the end the american liberal always ends up defending the right of the conservative to undermine the worth and value of the other – in the sacred name of "personal opinion" and the mistaken belief that every opinion is worthy of protection.

It’s not, and both your founding fathers and any nation with a national charter ensconcing free speech knew this.

PaulTsays:

Re:

Yeah, pretty much. If you allow people to post things via a 3rd party and the 3rd party’s promised filters aren’t working as required, it’s better to remove that feature than put up with the PR fallout from people blaming you directly.

Talking of overreactions, we can point to your comment, though – "everything" was not shut down, only the feature that used a 3rd party as a bonus feature whose promised filtering wasn’t working properly. People using Snapchat without also using giphy would not have noticed any change.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...
Older Stuff
15:30 Content Moderation Case Study: Tumblr's Approach To Adult Content (2013) (8)
15:41 Content Moderation Case Study: Twitter's Self-Deleting Tweets Feature Creates New Moderation Problems (2)
15:47 Content Moderation Case Studies: Coca Cola Realizes Custom Bottle Labels Involve Moderation Issues (2021) (16)
15:28 Content Moderation Case Study: Bing Search Results Erases Images Of 'Tank Man' On Anniversary Of Tiananmen Square Crackdown (2021) (33)
15:32 Content Moderation Case Study: Twitter Removes 'Verified' Badge In Response To Policy Violations (2017) (8)
15:36 Content Moderation Case Study: Spam "Hacks" in Among Us (2020) (8)
15:37 Content Moderation Case Study: YouTube Deals With Disturbing Content Disguised As Videos For Kids (2017) (11)
15:48 Content Moderation Case Study: Twitter Temporarily Locks Account Of Indian Technology Minister For Copyright Violations (2021) (8)
15:45 Content Moderation Case Study: Spotify Comes Under Fire For Hosting Joe Rogan's Podcast (2020) (64)
15:48 Content Moderation Case Study: Twitter Experiences Problems Moderating Audio Tweets (2020) (6)
15:48 Content Moderation Case Study: Dealing With 'Cheap Fake' Modified Political Videos (2020) (9)
15:35 Content Moderation Case Study: Facebook Removes Image Of Two Men Kissing (2011) (13)
15:23 Content Moderation Case Study: Instagram Takes Down Instagram Account Of Book About Instagram (2020) (90)
15:49 Content Moderation Case Study: YouTube Relocates Video Accused Of Inflated Views (2014) (2)
15:34 Content Moderation Case Study: Pretty Much Every Platform Overreacts To Content Removal Stimuli (2015) (23)
16:03 Content Moderation Case Study: Roblox Tries To Deal With Adult Content On A Platform Used By Many Kids (2020) (0)
15:43 Content Moderation Case Study: Twitter Suspends Users Who Tweet The Word 'Memphis' (2021) (10)
15:35 Content Moderation Case Study: Time Warner Cable Doesn't Want Anyone To See Critical Parody (2013) (14)
15:38 Content Moderation Case Studies: Twitter Clarifies Hacked Material Policy After Hunter Biden Controversy (2020) (9)
15:42 Content Moderation Case Study: Kik Tries To Get Abuse Under Control (2017) (1)
15:31 Content Moderation Case Study: Newsletter Platform Substack Lets Users Make Most Of The Moderation Calls (2020) (8)
15:40 Content Moderation Case Study: Knitting Community Ravelry Bans All Talk Supporting President Trump (2019) (29)
15:50 Content Moderation Case Study: YouTube's New Policy On Nazi Content Results In Removal Of Historical And Education Videos (2019) (5)
15:36 Content Moderation Case Study: Google Removes Popular App That Removed Chinese Apps From Users' Phones (2020) (28)
15:42 Content Moderation Case Studies: How To Moderate World Leaders Justifying Violence (2020) (5)
15:47 Content Moderation Case Study: Apple Blocks WordPress Updates In Dispute Over Non-Existent In-app Purchase (2020) (16)
15:47 Content Moderation Case Study: Google Refuses To Honor Questionable Requests For Removal Of 'Defamatory' Content (2019) (25)
15:32 Content Moderation Case Study: Vimeo Sued For Encouraging Infringement Via 'Lipdubs' (2009) (4)
15:38 Content Moderation Case Study: Friendster Battles Fakesters (2003) (4)
15:33 Content Moderation Case Study: Lyft Blocks Users From Using Their Real Names To Sign Up (2019) (6)
More arrow
This site, like most other sites on the web, uses cookies. For more information, see our privacy policy. Got it