[Repost] UIETips #2: CD-ROM Tutorial Observations

["Jared M. Spool" <jspool -at- UIE -dot- COM>]

The following is a repost of a message sent to UIETips, a discussion
list on usability issues.  I thought it might be of interest to
members of this list.  Information about signing up for UIETips can
be found at the end of this message.

Jared

____________________________________________________
UIETips #2  February 6, 1996  Carolyn Snyder
<For more info on UIETips, see the end of this message>

We've recently conducted some studies on a CD-ROM based tutorial.
This tutorial was chartered with introducing potential customers to a
very complicated product.  The expectation of the tutorial was that
they would, at the end, be interested enough in the product to
continue an extensive evaluation, eventually purchasing the product.

The tests were done with a combination of the paper prototype and the
real system.  During the tutorial, the users would get the chance to
"try what they learned."  Since the product actually existed, we had
them use it at this point.  (In the actual tutorial, their is a
technical restriction preventing the user from seeing both the product
and the tutorial on the screen at the same time.)

Here are some of the things we learned during the testing:

______________________________
Controls and Audio

When the tutorial is auto-paced and has an audio track (like a slide
show), users may not pause it when they have questions.  When it is
manually paced (Next and Back buttons) without audio, users spend more
time but comprehend more. Interestingly, the audio seemed to have more
effect on their pace than the manual controls (this may have been an
artifact of paper prototype testing - the "audio track" was a human
being, and users may have been less willing to interrupt it). Maybe
there is some subconscious notion that once it's done talking, you've
spent enough time here (pure speculation).

Users said they would probably disable the audio.  This would be a
problem for any interface that is providing essential information only
through the audio channel.

______________________________
Chunking Of Information

The tutorial we tested presented some steps and then had users
switch over and try it. This seemed to work pretty well - they
weren't frustrated by too-frequent switching or didn't have "stack
overflows" from too-infrequent switching.

______________________________
Remembering Inputs

When users just had to pick a specific file, this wasn't a problem,
but when they had to remember a name or numbers to type in, they
either had to refer back to the instructions or they got it wrong
(pretty much without exception).

______________________________
Coordination With A Manual

The current version of the tutorial we tested started life as a
Getting Started manual. What we found is that a good manual doesn't
translate well into tutorial format -- users can't easily skim
material that is introductory or too basic. Also, the tutorial format
offers capability for animation that a manual can't: to explain a
concept that is dynamic, (such as how a database is automatically
connected to a set of fields on a form,) a moving picture may replace
many words.

Users tended not to follow along with the manual, particularly when
the tutorial was auto-paced. When users needed to refer to the manual,
it was usually to find the input they were supposed to type in.

The manual had many pages of similar-looking illustrations; hard to
find the "you are here." One solution we explored was to number the
tutorial screens and have the manual use these same numbers in the
margins. We also discussed having the tutorial contain page numbers,
but this is harder for translation, which causes repagination.

______________________________
Shortcomings Of Tutorial Format

Users don't necessarily traverse the sections in order, even when you
tell them to (now there's a surprise). In this case, they didn't want
to see a lot of overview material - they wanted to get some hands-on
with the product they were evaluating.

The first time a complex diagram or illustration appears, users in a
self-paced tutorial may spend a lot of time trying to digest it. This
was a problem because the next several tutorial screens (which the
user didn't realize were coming) were devoted to the explanation they
needed. In the electronic version, the tutorial developers plan to use
highlighting of the area under discussion which might alleviate some
of this.

______________________________
Material Appropriate For Audience

Users were really put off by material that was too basic (like a
discussion of the Help system - they'd all used Windows-based Help).
For a sales-oriented thing, this seemed to be a worse sin than
material over users' heads, since it affected their willingness to
continue using the tutorial.

______________________________
Reduction Of Work

The testing we did actually allowed us to prune the schedule
tremendously.  Even though we were testing with a paper prototype, the
actual code was being written at exactly the same time.  Our testing
showed that several of the topics that were planned were unnecessary,
therefore reducing the amount of development required.

______________________________
Your Experiences

Have you created a tutorial with audio and animation?  Have you had a
chance to evaluate its effectiveness?  What results did you see?

========================================

Permission is granted to non-profit organizations to reproduce this
article as long as this message is reproduced in its entirety, with
the following copyright statement intact.

(c) 1996, User Interface Engineering, All Rights Reserved
North Andover, MA 01845  USA
(508) 975-4343   uie -at- uie -dot- com

========================================

UIETips is about designing user interfaces -- what works
and what doesn't.  For more information, send a message to
UIETips-Request -at- uie -dot- com with the word INFO in the body of the
message.

********************** DISCLAIMER ***********************

The information presented here is based on observations we've made of
hundreds of users working with dozens of products.  However, we have
never tested *your* users working with *your* product.  Our experience
is that every user uses every product differently, your mileage will
vary.

User Interface Engineering is a consulting firm specializing in
Product Usability issues.  Our mission is to empower development teams
by providing key information for making strategic design decisions.
Our services include training, consulting and research.

==========================================================
 Jared M. Spool                User Interface Engineering
 jspool -at- uie -dot- com            800 Turnpike Street, Suite 101
 (508) 975-4343                   North Andover, MA 01845
 fax: (508) 975-5353                                  USA

      If you send me your postal address, you'll get
     the next issue of our newsletter, Eye For Design.
==========================================================