Creating Family Friendly Chat More Difficult Than Imagined (1996)

Content Moderation

from the the-kids-will-find-a-way dept

Summary: Creating family friendly environments on the internet presents some interesting challenges that highlight the trade-offs in content moderation. One of the founders of Electric Communities, a pioneer in early online communities, gave a detailed overview of the difficulties in trying to build such a virtual world for Disney that included chat functionality. He described being brought in by Disney alongside someone from a kids? software company, Knowledge Adventure, who had built an online community in the mid-90s called ?KA-Worlds.? Disney wanted to build a virtual community space, HercWorld, to go along with the movie Hercules. After reviewing Disney?s requirements for an online community, they realized chat would be next to impossible:

Even in 1996, we knew that text-filters are no good at solving this kind of problem, so I asked for a clarification: “I?m confused. What standard should we use to decide if a message would be a problem for Disney?”

The response was one I will never forget: “Disney?s standard is quite clear:

No kid will be harassed, even if they don?t know they are being harassed.”…

“OK. That means Chat Is Out of HercWorld, there is absolutely no way to meet your standard without exorbitantly high moderation costs,” we replied.

One of their guys piped up: “Couldn?t we do some kind of sentence constructor, with a limited vocabulary of safe words?”

Before we could give it any serious thought, their own project manager interrupted, “That won?t work. We tried it for KA-Worlds.”

“We spent several weeks building a UI that used pop-downs to construct sentences, and only had completely harmless words ? the standard parts of grammar and safe nouns like cars, animals, and objects in the world.”

“We thought it was the perfect solution, until we set our first 14-year old boy down in front of it. Within minutes he?d created the following sentence:

I want to stick my long-necked Giraffe up your fluffy white bunny.

In that initial 1996 project, chat was abandoned, but as they continued to develop HercWorld, they quickly realized that they still had to worry about chat, even without a chat feature:

It was standard fare: Collect stuff, ride stuff, shoot at stuff, build stuff? Oops, what was that last thing again?

“?kids can push around Roman columns and blocks to solve puzzles, make custom shapes, and buildings.”, one of the designers said.

I couldn?t resist, “Umm. Doesn?t that violate the Disney standard? In this chat-free world, people will push the stones around until they spell Hi! or F-U-C-K or their phone number or whatever. You?ve just invented Block-Chat^TM. If you can put down objects, you?ve got chat. We learned this in Habitat and WorldsAway, where people would turn 100 Afro-Heads into a waterbed.” We all laughed, but it was that kind of awkward laugh that you know means that we?re all probably just wasting our time.

Decisions for family-friendly community designers:

• Is there a way to build a chat that will not be abused by clever kids to reference forbidden content (e.g., swearing, innuendo, harassment, abuse)?
• Can you build a chat that does not require universal moderation and pre-approval of everything that users will say?
• Are there ways in which kids will still be to communicate with others even without an actual chat feature?
• How much of a ?community? do you have with no chat or extremely limited chat?

Questions and policy implications to consider:

• Is it possible to create an online family friendly environment that will work?
  • If so how do you prevent abuse?
  • If not, how do you handle the fact that kids will get online whether they are allowed to or not?
• How do you incentivize companies to create spaces that actually remain as child-friendly as possible?
• If ?the kids will always find a way? to get around limitations, does it make sense to hold the companies themselves responsible?
• Should family friendly environments require full-time monitoring, or pre-vetting of any usage?

Resolution: Disney eventually abandoned the idea of HercWorld due to all of the issues raised. However, the interview highlights the fact that they tried again a couple of years later, with an online chat where users could only pull from a pre-selected list of sentences, but it did not have much success:

“The Disney Standard” (now a legend amongst our employees) still held. No harassment, detectable or not, and no heavy moderation overhead.

Brian had an idea though: Fully pre-constructed sentences ? dozens of them, easy to access. Specialize them for the activities available in the world. Vaz Douglas, our project manager working with Zoog, liked to call this feature “Chatless Chat.” So, we built and launched it for them. Disney was still very tentative about the genre, so they only ran it for about six months; I doubt it was ever very popular.

The same interview notes that Disney tried once again in 2002 with a new world called ?ToonTown?, with pulldown menus that allowed you to construct very narrowly tailored speech within the chat to try to avoid anything that violated the rules.

As the story goes, Disney still had problems with this. To make sure people were only communicating with people they knew in real life, one of the restrictions in this new world was that you had to have a secret code from any user you wished to chat with. The thinking was that parents would print these out for kids who could then share them with their friends in real life, and they could link up and ?chat? in the online world.

And yet, once again, people figured out how to get around the restrictions:

Sure enough, chatters figured out a few simple protocols to pass their secret code, several variants are of this general form:

User A:”Please be my friend.”
User A:”Come to my house?”
User B:”Okay.”
A:[Move the picture frames on your wall, or move your furniture on the floor to make the number 4.]
A:”Okay”
B:[Writes down 4 on a piece of paper and says] “Okay.”
A:[Move objects to make the next letter/number in the code] “Okay”
B:[Writes?] “Okay”
A:[Remove objects to represent a “space” in the code] “Okay”
[Repeat steps as needed, until?]
A:”Okay”
B:[Enters secret code into Toontown software.]
B:”There, that worked. Hi! I?m Jim 15/M/CA, what?s your A/S/L?”

Incredibly, there was an entire Wiki page on the Disney Online Worlds domain that included a variety of other descriptions on how to exchange your secret number within the game, even as users were not supposed to be doing so:

For example, let’s say you have a secret code (1hh 5rj) which you would like to give to a toon named Bob.

First, you should make clear that you want to become their SF.
You: Please be my friend!
You: (random SF chat)
You: I can’t understand you
You: Let’s work on that
Bob: Yes
Now, start the secret.
You: (Jump 1 time and say OK. Jump 1 time because that is the first thing in your code. Say OK to confirm that was part of your secret.)
Bob: OK (Wait for this, as this means he has written down or otherwise recorded the 1)
You: Hello! OK (Say hello because the first letter of hello is h, which is the second part of your secret.)
Bob: OK (again, wait for confirmation)
Repeat above step, as you have the same letter for the third part of your secret.
Bob: OK (by now you should know to wait for this)
You: (Jump 5 times and say OK. Jump 5 times as this is the 4th part of your secret)
Bob: OK
You: Run! OK (The 5th part of your secret is r, and “Run!” starts with r)
Bob: OK
You: Jump! OK (Say this because j is the last part of your secret.)
Bob: OK
At this point, you have successfully transmitted the code to Bob.
Most likely, Bob will understand, and within seconds, you will be Secret Friends!

So even though Disney eventually did enable a very limited chat, with strict rules to keep people safe, it still left open many challenges for early trust & safety work.

Images from HabitChronicles

Filed Under: chat, content moderation, family friend
Companies: disney