How Toy Story 2 Almost Got Deleted… Except That One Person Made A Home Backup
from the did-it-break-the-rules? dept
Here’s a random story, found via Kottke, highlighting how Pixar came very close to losing a very large portion of Toy Story 2, because someone did an rm * (non geek: “remove all” command). And that’s when they realized that their backups hadn’t been working for a month. Then, the technical director of the film noted that, because she wanted to see her family and kids, she had been making copies of the entire film and transferring it to her home computer. After a careful trip from the Pixar offices to her home and back, they discovered that, indeed, most of the film was saved:
Now, mostly, this is just an amusing little anecdote, but two things struck me:
- How in the world do they not have more “official” backups of something as major as Toy Story 2. In the clip they admit that it was potentially 20 to 30 man-years of work that may have been lost. It makes no sense to me that this would include a single backup system.
- I wonder if the copy, made by technical director Galyn Susman, was outside of corporate policy. You would have to imagine that at a place like Pixar, there were significant concerns about things “getting out,” and so the policy likely wouldn’t have looked all that kindly on copies being used on home computers.
The Mythbusters folks wonder if this story was a little over-dramatized, and others have wondered how the technical director would have “multiple terabytes of source material” on her home computer back in 1999. That resulted in an explanation from someone who was there that what was deleted was actually the database containing the master copies of the characters, sets, animation, etc. rather than the movie itself. Of course, once again, that makes you wonder how it is that no one else had a simple backup. You’d think such a thing would be backed up in dozens of places around the globe for safe keeping…
Filed Under: backups, mythbusters, toy story
Companies: pixar
Comments on “How Toy Story 2 Almost Got Deleted… Except That One Person Made A Home Backup”
It’s called Godzilla-proofing. Know it. Live it.
Re: Human Trumps Radioactive Dinosaur
Godzilla-proofing is far easier than Idiot-proofing.
And this is why the Canadian/American governments should stop trying to make the act of backing up videos illegal.
You could say it’s a different thing all together, but it really isn’t.
Re: Re:
No, it certainly is. Without her, we’d have never seen the dreck from after the first one. SHE TOTALLY RUINED IT FOR ME!!!!
There's a trope for that!
http://tvtropes.org/pmwiki/pmwiki.php/Main/KeepCirculatingTheTapes
Re: There's a trope for that!
And there went my morning. I can’t resist that place….
Re: Re: There's a trope for that!
Tere’s a trope for that too
Shame on this horrible woman for making an illegal backup of this movie. By doing so, she cost so many workers over a year of salary for restoring it, all for her own selfish greed. Pathetic. When will these freetards think of the workers and the customers?
This story gave me a woody
Re: Re:
…and a buzz?
/lameness
Re: Re: Re:
/lameness
I doubt it 🙂
Re: Re: Re:
And a zerg!
Re: Re:
You are a sad, strange little man, and you have my pity.
Re: Re:
Your such a Ham
so, no back ups allowed, even when your copy of the movie is deleted means all the work for the whole thing is lost. good strategy, guys! i suppose that the one ‘illegal’ copy, regardless of who had it and that it saved some serious arse kicking from happening and a hell of a lot of money is irrelevant? as it applies to any download by anyone for any reason, all i can say is ‘but but but piracy’!
Reminds me of ....
Some decades ago I was called to a customer site, a bank, to diagnose a computer problem. On my arrival early in the morning I noted a certain panic in the air. On querying my hosts I was told that there had been an “issue” the previous night and that they were trying, unsuccessfully, to recover data from backup tapes. The process was failing and panic ensued.
Though this was not the problem I had been called on to investigate, I asked some probing questions, made a short phone call, and provided the answer, much to the customer’s relief.
What I found was that for months if not years the customer had been performing backups of indexed sequential files, that is data files with associated index files, without once verifying that the backed-up data could be recovered. On the first occasion of a problem requiring such a recovery they discovered that they just did not work.
The answer? Simply recreate the index files from the data. For efficiency reasons (this was a LONG time ago) the index files referenced the data files by physical disk adresses. When the backup tapes were restored the data was of course no longer at the original place on the disk and the index files were useless. A simple procedure to recreate the index files solved the problem.
Clearly whoever had designed that system had never tested a recovery, nor read the documentation which clearly stated the issue and its simple solution.
So here is a case of making backups, but then finding them flawed when needed.
Re: Reminds me of ....
That’s why, in the IT world, you ALWAYS do a “dry run” when you want to deploy something, and you monitor the heck out of critical systems.
Yeah good point, you think that they would be more secure, whats to stop a employee from “saving” a film on their home computer then selling it to someone who would release it online or something? That must have been a wake up call.
Two notes on backups
1. Everyone who has worked in computing for any period of time has their own backup horror story. I’ll spare you mine, but note that as a general observation, large organizations/corporations tend to opt for incredibly expensive, incredibly complex, incredibly overblown backup “solutions” sold to them by vendors rather than using the stock, well-tested, reliable tools that they already have. (e.g., “why should we use dump, which is open-source/reliable/portable/tested/proven/efficient/etc., when we could drop $40K on closed-source/proprietary/non-portable/slow/bulky software from a vendor?”
Okay, okay, one comment: in over 30 years of working in the field, the second-worst product I have ever had the misfortune to deal with is Legato (now EMC) NetWorker.
2. Hollywood has a massive backup and archiving problem. How do we know? Because they keep telling us about it. There are a series of self-promoting commercials that they run in theaters before movies, in which they talk about all of the old films that are slowly decaying in their canisters in vast warehouses, and how terrible this is, and how badly they need charitable contributions from the public to save these treasures of cinema before they erode into dust, etc.
Let’s skip the irony of Hollywood begging for money while they’re paying professional liar Chris Dodd millions and get to the technical point: the easiest and cheapest way to preserve all of these would be to back them up to the Internet. Yes, there’s a one-time expense of cleaning up the analog versions and then digitizing them at high resolution, but once that’s done, all the copies are free. There’s no need for a data center or elaborate IT infrastructure: put ’em on BitTorrent and let the world do the work. Or give copies to the Internet Archive. Whatever — the point is that once we get past the analog issues, the only reason that this is a problem is that they made it a problem by refusing to surrender control.
Re: Two notes on backups
“Real Men don’t make backups. They upload it via ftp and let the world mirror it.” – Linus Torvalds
Re: Re: Two notes on backups
I should make stickers for hard drives/USB drives/disks:
Proud to be a backup
Let people print ’em for free and let them go wild on their media collection of choice (and I’ll never lose the sticker image either). Go ahead, add some character to that aging collection of DVDs, they’ll all mold over eventually!
What I suspect is that she was copying the rendered footage. If the footage was rendered at a resolution and rate fitting to DVD spec, that’d put the raw footage at around 3GB to 4GB for a full 90min, which just might fit on the 10GB HDD that were available back then on a laptop computer (remember how small OSes were back then). Even losing just the rendered raw footage (or even processed footage), would be a massive setback. It takes a long time across a lot of very powerful computers to render film quality footage. If it was processed footage then it’s even more valuable as that takes a lot of man hours of post fx to make raw footage presentable to a consumer audience.
Re: Re:
Nah, she says she made the copies because she needed to “work” from home. You don’t work on rendered footage, it’s basically done. Also, the other guy doesn’t say that the movie was disappearing from the drive, he says characters or parts of characters were vanishing.
I’m guessing she’s a texture artist who was making backups of the character/scene models and she was copying them home so she could see final result with her artwork included.
Though, when I think about more, does Toy story even have complicated textures? I have a strong memory of mostly solid colors.
Re: Re: Re:
Toy Story 2 had very complicated shaders for it’s time. They don’t just use solid-color textures. Even shaders that appear solid in texture to you have very subtle variations that insinuate reality. This is part of what made and makes Pixar so special.
Re: Re: Re:
Not so much complicated as detailed. In Toy Story 2, Woody’s arm has athread pattern, and the texture works with this. The stuffing in his arm is also very detailed. Just a few examples
Backups
Everyone knows to make backups, but not everyone thinks to validate and test restoring them.
Re: Backups
Indeed. I’m reminded of the guy in my Intro to OS class accidentally removing everything from the Lecturer’s computer.
Human complacency and dependance on computers is scarey sometimes…
a retelling by Oren Jacob
Oren Jacob, the Pixar director featured in the animation, has made a comment on the Quora post that explains things in much more detail. The narration and animation was telling a story, as in storytelling. Despite the 99% true caption at the end, a lot of details were left out which misrepresented what had happened. Still, it was a fun tale for anyone who had dealt with backup problems. Oren Jacob’s retelling in the comment makes it much more realistic and believable.
The terabytes level of data came from whoever posted the video on Quora. The video itself never mentions the actual amount of data lost or the total amount the raw files represent. Oren says, vaguely, that it was much less than a terabyte. There were backups! The last one was from two days previous to the delete event. The backup was flawed in that it produced files that when tested, by rendering,
exhibited errors. They ended up patching a two-month old backup together with the home computer version (two weeks old). This was labor intensive as some 30k files had to be individually checked.
The moral of the story. Firstly, always test a restore at some point when implementing a backup system. Secondly, don’t panic! Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer. Thirdly, don’t panic! Despite, somehow, deleting a large set of files these can be recovered apart from a backup system. Deleting files, under Linux as well as just about any OS, only involves deleting the directory entries. There is software which can recover those files as long as further use of the computer system doesn’t end up overwriting what is now free space.
Re: a retelling by Oren Jacob
What’s worse? Corrupting some files or deleting all files?
Re: Re: a retelling by Oren Jacob
In this case they were not dealing with unknown malware that was steadily erasing the system as they watched. There was, apparently, a delete event at a single point in time that had repercussions that made things disappear while people worked on the movie. I’ll bet things disappeared when whatever editing was being done required a file to be refreshed. A refresh operation would make the related object disappear when the underlying file was no longer available. Apart from the set of files that had already been deleted, more files could have been corrupted when the computer was unplugged. Having said that, this occurred in 1999 when they were probably using the Ext2 filesystem under Linux. These days most everyone uses a filesystem that includes journaling which protects against corruption that may occur when a computer loses power. Ext3 is a journaling filesystem and was introduced in 2001.
In 1998 I had to rebuild my entire home computer system. A power glitch introduced corruption in a Windows 95 system file and use of a Norton recovery tool rendered the entire disk into a handful of unusable files. It took me ten hours to rebuild the OS and re-install all the added hardware, software, and copy personal files from backup floppies. The next day I went out and bought a UPS. Nowadays, sometimes the UPS for one of my computers will fail during one of the three dozen power outages a year I get here. I no longer have problems with that because of journaling.
who the hell punched in RM* to begin with?
Re: Re:
Obviously Disney did.
Re: Re: Re:
Well, it was pretty well-known that Disney wanted the partnership to fail at the time.
I've gotta story like this too
Ive posted in athe past on Techdirt that I used to work for Ticketmaster. The is an interesting TM story that I don’t think ever made it into the public, so I will do it now.
Back in the 1980s each TM city was on an independent computer system (PDP unibus systems with RM05 or CDC9766 disk drives. The drives were fixed removable boxes about the size of a washing machine, the removable disk platters about the size of the proverbial breadbox. Each platter held 256mb formatted.
Each city had itts own operations policies, but generally, the systems ran with mirrored drives, the database was backed up every night, archival copies were made monthly. In Chicago, where I worked, we did not have offsite backup in the 1980s. The Bay Area had the most interesting system for offsite backup.
The Bay Area BASS operation, bought by TM in the mid 1980s, had a deal with a taxi driver. They would make their nightly backup copies in house, and make an extra copy on a spare disk platter. Tis cabbie would come by the office about 2am each morning, and they’d put the spare disk platter in his trunk, swapping it for the previous day’s copy that had been his truck for 24 hours. So, for the cost of about two platters ($700 at the time) and whatever cash they’d pay the cabbie, they had a mobile offsite copy of their database circulating the Bay Area at all times.
When the World Series earthquake hit in October 1988, the TM office in downtown Oakland was badly damaged. The only copy of the database that survived was the copy in the taxi cab.
That incident led TM corporate to establish much more sophisticated and redundant data redundancy policies.
Re: I've gotta story like this too
I like that story. Not that it matters anymore, but taxi cab storage was probably a bad idea. The disks were undoubtedly the “Winchester” type and when powered down the head would be parked on a “landing strip”. Still, subjecting these drives to jolts from a taxi riding over bumps in the road could damage the head or cause it to be misaligned. You would have known though it that actually turned out to be a problem. Also, I wouldn’t trust a taxi driver with the company database. Although, that is probably due to an unreasonable bias towards cab drivers. I won’t mention the numerous arguments with them (not in the U.S.) over fares and the one physical fight with a driver who nearly ran me down while I was walking.
Re: Re: I've gotta story like this too
RM05s are removable pack drives. The heads stay in the washing machine size unit – all you remove are the platters.
Re: I've gotta story like this too
Very interesting story, Danny…
& I Don’t mean to nit-pick but the quake was Oct. ’89. After reading the sloppy coding entries, I’ve been paying attention 2 details way more than I’d like to. I’ve been seeing 0’s & 1’s in my sleep. Damn nerds
So, making copies of an entire copyrighted film isn’t Piracy?
This is why limiting copying of a work isn’t “protecting” it.
It may protect to company’s monetary interest in it, but once the product is out there it hinders the growth of the work and in some cases cause it to be lost forever if it isn’t “stolen” by those damn dirty pirates
What I want to know is this...
She copied bits of a movie to her home system… how hard did they have to pull in the leashes to keep Disney’s lawyers from suing her to infinity and beyond after she admitted she’d done so(never mind the fact that he doing so saved them apparently years of work…)?
Mike, ‘deleting’ a file does not delete it from HDD unless it’s overwritten.
BTW once again “Linux” is mentioned.
Fake
I really don belive this, cant be 🙂
Not nearly as exciting but i saved my boss’s job one year when he accidentally wiped our database. I had been making copies to save my work in case I quit. I hated this boss more than anything but I couldn’t find it in me to allow him to be fired, so I lied and said I had been making regular backups.
http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-and-again-for-its-own-good/
Evidently, the film data only took up 10 GB in those days. Nowhere near TB…
TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
DELETE TOY STORY 3 MOVIE
YES
NOW
NO TOY STORY 3 MOVIE DELETE NOW
YES
AND ANDY BOY SMALL YES
ANDY NO MAN BIG DELETE
NOW YES
GO NOW
TOY STORY 1 YES AND TOY STORY 2 YES
NOW
ANDY BOY SMALL YES TOYS
TOYS ANDY TOY STORY 1 YES
TOYS ANDY TOY STORY 2 YES
NOW YES
GO NOW
DELETE TOY STORY 3 ANDY MAN
ANDY NO MAN BIG DELETE
ANDY BOY SMALL YES TOYS
GO NOW
DELETE TOY STORY 3 MOVIE
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
NOW
NETWORK DELETE TOY STORY 3 MOVIE
TV CHANNEL DELETE TOY STORY 3 MOVIE
ANDY BOY SMALL YES TOYS
ANDY LOVE MUMMY WOMAN
ANDY BOY YES
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
GO NOW
YES
OK
They're no better at looking after stuff nowadays
Whether it’s accidental deletion or hacking or an inside job – or simply losing millions of dollars of bitcoins! I guess the thing is no one thinks it will happen to them. Till it does.