Ooops! Forgot to Back Up Horror Stories

[Kelley <kwalker2 -at- gte -dot- net>]

Hi,

We're doing an in-house newsletter about all the little ways the firm can 
lose data through mistakes and "cascading failures". For the feature on 
remembering to back up your documents as you work, we're hoping to collect 
some "Ooops, I forgot to backup horror stories!!". I've appended a sample 
of what I wrote for last year's issue to give you an idea of the range of 
stories we included.

I'm not writing the newsletter this year; I've tasked it out. It's her 
first time writing content like this--she's a online help techwriter I 
hired because she needed a job and we need a writer! So, to ease her 
transition from one type of writing to another, I thought I would do the 
work of collecting stories. I'm (and most of the folks I would ask) are 
fresh out of new ideas/stories, so I figured that TECHWRirlers might have 
some memorable stories. I want to ask for "humorous" stories, but I was 
scolded by a sysadmin friend who told me that backup failure stories are 
NEVER funny. :)

Anyway, if you would be so kind, I'd love to hear your stories offlist I 
presume, since it's not exactly "on topic". (Although, I'm sure some of us 
could angle it into an "on topic" discussion... :p )  Identifying 
details--names and orgs--will be anonymized.

Thanks!

Kelley

Sample Stories:

Backup is so crucial and so often overlooked that Geek Culture has a "law" 
known as Huber's law: The chances of losing your data are directly 
proportional to the length of time since your last backup.

Once upon a time, there was a company called Kaypro computers. They ran 
their entire business on 10MB hard disk computers, which, despite 
management's errant beliefs, were terribly prone to failure. And so, the 
day before an IRS filing was due, the key accounting computer's hard disk 
crashed. It cost the company almost $400,000 in fines and penalties because 
they had no backups and had to recreate the records from hard copy files. 
At least they had hard copy, but they could have bought a lot of backup 
devices for $400K.

One of XXX's consultants does a lot of writing, and occasionally writes in 
bed late at night. He laid the computer on the floor next to the bed one 
night and his cat spilled a water glass onto the computer. The computer was 
fried and the hard disk was trashed, too. He did have a backup made a week 
earlier, but all of the articles he had written had to be rewritten. 
Several days of work gone for triple stupidity: Computer on floor, water 
near the computer, and no back up.

When a fire swept through Oakland and Berkeley Hills in Northern California 
in 1991, Maxine Hong-Kingston lost her home as well as a half-finished 
manuscript for her book, which is now being published as Fifth Book of 
Peace.  Asked about the experience she said, "It's horrifying how long it 
takes."  It was a struggle to try to write about the loss to the fires. In 
her subsequent re-writing of the 1800 page book, her thoughts continually 
turned to water because it seemed the only thing that might have saved her 
work, her home, her neighborhood, and the 25 people who died in the fires.

Another XXX consultant recalls why she backs up five different ways.  A 
colleague, a Ph.D. candidate at Cornell University, had finished her 
masterpiece: a 500 page tome that her five member committee had ostensibly 
read in various stages throughout the five year long project.  The week 
prior to her dissertation defense, the apartment building she lived in 
burned to the ground.  She stood outside, dressed in a nightgown, holding 
her cat, screaming about her dissertation.  Five firefighters had to hold 
her back.  She had kept all three copies of her work in her home.  Although 
her committee had read her work, none of them had kept the e-mailed 
attachments she'd sent them.  As far as they were concerned, it was her job 
to have foreseen the possibility of such a mishap.  She had to reproduce 
all her work.  They refused to confer the Ph.D. without a hard copy of it.

A proud papa, a systems administrator, decided to take his 2-year-old son 
to visit his workplace.  It was the Thanksgiving holiday, most people were 
heading toward the company festivities-everyone except our proud papa and 
forty employees busily wrapping up projects before heading to the party 
themselves. "Being the brilliant 2-year- old he was at the time," according 
to his father, the young geek reached out as he toddled past the Novell 
server and pressed the shiny button on the Compaq fileserver.  The result: 
a crashed network, lost files since the backup tape was from the night 
before, an hour of work to get all the services back on-line, and 40 irate 
employees who lost all the work they'd been doing.

The Home Shopping Network is amazing. Located in St. Petersburg, FL, they 
are subject to more lightening strikes than anywhere on earth. So they 
built in backup and redundancy everywhere. They have a bank of batteries 
that fill an entire building for the regular blackouts. They have two 
mainframe computers for sales; the second is there just in case the first 
dies. The TV studio you see is only one of two. There is an exact duplicate 
of it that can be turned on in minutes in case the first one dies. They 
spend a lot of money on backup, but have you ever seen them off the air?

Back Biting Y2k+1 Bug: The chain of convenience stores, 7-Eleven, felt the 
sting the first day of the New Year, 01/01/01.  A glitch that read the date 
as January 01, 1901 halted all computer mediated credit-card sales at 5,200 
stories, despite having spent nearly $9 million on a Y2K 
upgrade.   Computer glitch halts credit card sales at stores across the 
U.S.  It took nearly two days to get all the systems back on-line because 
the credit-card sales system does more than simply record sales 
information.  It is also integrated with a tracking system that examines 
weather forecasts, inventory, traffic conditions, holidays and other data 
in order to keep an adequate supply of goods on hand for changing weather 
patterns, road conditions, holiday and seasonal needs.

A few days after the New Year, the owner of a PDA discovered that he'd been 
bitten by the Y2K+1 bug.  The electronic organizer displayed an error 
message the first time he turned it on after the New Year.  He lost four 
years worth of notes and phone numbers.  He could have avoided the problem 
with a backup kit, which costs the same as the PDA. Still, it was well 
worth the money since the costs of restoring four years worth of data 
certainly exceeded the cost of the equipment needed to backup in the first 
place.  What would happen to you if you lost your PDA?  How much 
information do you store there? Is any of it backed up?  Backing up is not 
so hard to do!

Perhaps the experiences of two systems administrators, neophytes back in 
the early days of campus computing, are illustrative.  Joe was hired as a 
systems administrator for an engineering department.  He was given a 
choice.  His boss said, "We have a budget; I can give you a higher salary 
and we can buy some new disk drives. Or, we can buy a tape drive and 
provide backup service for the graduate students. What's your 
preference?"  Many years later, Joe says sardonically, "I can't tell you 
how fun it was to get mail asking for help restoring dissertations."  Since 
then Joe has never chosen disk drives over backup upgrades.

Paul, on the other hand, feels pretty strongly about the topic, insisting 
that one simply cannot describe an "I failed to plan for the worst" story 
as anything short of a nightmare.  His worst experience came one day when 
he unmirrored a drive in the morning with the intent of replacing it later 
in the day.  In the meantime, however, a second drive crashed.  Half a 
day's work was lost for everyone.

Jack Carrol, a network security pro, characterized the most common backup 
disasters as "cascading failures": a well-intentioned but poorly-planned 
effort to fix one problem results in the problem getting worse, possibly 
drastically worse. She insists that its not that the systems administrator 
or average user made a stupid mistake per se.  Rather, it's just that the 
situation is generally high-pressure and the decision-making is muddled as 
a result.  They panic, in other words. The best thing to do in that 
situation is try to count to ten and take several-many-deep breaths.  If 
possible, walk away from the situation and try to clear your head.  Get 
help from another person or a third party.  Even if they can't help, 
sometimes sharing the burden with a trusted colleague can help. Then go 
back to the task and try to look at the situation with fresh eyes and a 
clear head.

Copyright Interpact, Inc. 2001



^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Be a published author!  iUniverse gives you: a high-quality paperback, a
custom cover design, and distribution to 25,000 retailers.  And it's
affordable.  Join our almost 10,000 published authors today.
http://www.iuniverse.com/media/techwr

 Your monthly sponsorship message here reaches more than
 5000 technical writers, providing 2,500,000+ monthly impressions.
 Contact Eric (ejray -at- raycomm -dot- com) for details and availability.

---
You are currently subscribed to techwr-l as: archive -at- raycomm -dot- com
To unsubscribe send a blank email to leave-techwr-l-obscured -at- lists -dot- raycomm -dot- com
Send administrative questions to ejray -at- raycomm -dot- com -dot-  Visit 
http://www.raycomm.com/techwhirl/ for more resources and info.