Content Moderation At Scale Especially Doesn't Work When You Hide All The Rules

(Mis)Uses of Technology

from the getting-twitchy dept

For quite a while now, we’ve pointed out that doing any serious content moderation on major internet sites is laughably difficult, if not impossible. Whether done in a purely automated format, or with real human oversight, everything ends up boiling down to just how much collateral damage are we all willing to accept when sites attempt to enforce moderation rules. Even when sites attempt to communicate the rules to the public in a somewhat transparent fashion, such as Facebook, it all inevitably goes to the kind of hell that includes nixing accounts for sharing what is purely art.

But when sites don’t bother to tell their users what the rules are, even after exacting punishment for violation of those rules? Well, that’s when you have a bunch of Twitch streamers wondering what the hell is going on.

Earlier this week, Twitch suspended a streamer named Quqco for wearing a cosplay of Street Fighter heroine Chun-Li on stream, deeming her outfit “sexually suggestive.” This took Quqco—and many others—by surprise, given that the outfit was not overtly risqué. But this was not an isolated incident. In the past few days, Twitch has been cracking down on so-called “sexually suggestive” content more aggressively than usual, and picking some questionable targets for its crusade.

The whole post is long and very detailed and should absolutely be read in full. The article lists numerous examples of Twitch streamers being handed out warnings and suspensions under any number of circumstances that range from cosplaying as Street Fighter characters, to wearing a sports bra and baggy shorts on a stream, to drawing clothed cartoon characters. All those interviewed for the post that were on the receiving end of these warnings or suspensions were completely flummoxed.

Especially when their own inquiries into their punishment were being responded to like this.

For example, Fareeha’s warning specifically accused her of wearing “underwear or lingerie,” when she was, in reality, wearing baggy gym clothes in a setting where you’d expect to see them. Twitch’s guidelines around what streamers can and cannot wear are vague and contextual; the sort of attire Fareeha was wearing might not have cut it if a streamer was broadcasting from their bedroom, but streamers regularly wear gym clothes in the gym. Some men even go shirtless. It’s not clear why Twitch singled out Fareeha.

Fareeha specifically believes she may be the victim of another aspect of the internet that makes content moderation at scale impossible: trolls. It turns out that there are dedicated Discord channels out there where people get off on doing mass-reports of Twitch streamers they don’t like for any number of reasons. Obviously, those many numbers of reasons often fall into categories that include “they’re women doing things we don’t like” and “they’re people that don’t look like us doing things we don’t like.” The theory goes that whatever automation Twitch has built into its moderation system — and there surely must be something of that sort of thing — it’s likely to be sensitive to mass reporting of ToS violations. This is being gamed by trolls to get the system to punish Twitch streamers who otherwise would never have been punished.

Is that what’s happening? Nobody knows, mostly because Twitch is being frustratingly opaque on the matter.

Saruei, who was suspended for drawing “nude” characters, declined to speak to Kotaku out of concern that she could face further repercussions from Twitch. However, prior to her recent suspension, she spoke out against what she feels is “hypocrisy” on Twitch’s part. Suggestive poses apparently aren’t allowed in her drawings, she said, but it’s fine when some people do them IRL.

“These are suggestive poses, right?” she said of her own art while discussing her Twitch warning during a recent (now-unavailable) stream. “We agree with that, right? Why I can’t draw waifus like this when there is fucking Twitch girls that can do it?” She went on to express frustration about the lack of clarity that she, like others, has had to deal with. “I hope it won’t happen again, because I asked them ‘What is the problem with these drawings?’” she said. “Is it the clothing or the pose? What is against Twitch guidelines? I need to know.”

Those last two sentences are all you need to hear to recognize the massive problem that is Twitch attempting to do content moderation at a scale that is largely impossible while not being transparent as to how to stay on the right side of that moderation. There is entirely too much gray area, too much room for gaming the moderation, and too much stupid collateral damage to make any of this worth it.

Meanwhile, the Twitch streamers who make Twitch’s product attractive are stuck trying to figure out what the rules are. Until they decide to go somewhere else, that is.

Filed Under: content moderation, content moderation at scale, quqco, rules, transparency
Companies: twitch