Re: Wish list for academic research

["Eric J. Ray" <ejray -at- raycomm -dot- com>]

Janet Valade wrote:
>         <<Jason Sayz:
> > So, if the majority of the users think a manual sucks, but we perform a
> > study that says it doesn't, can we tell the users that they are morons?>>
> >
>         The point is-- We have no idea what the users think. My crystal ball
> is seriously out-of-order. And the point is--to define *suck*. Too hard? Too
> easy? Too long? Too expensive? No pictures? Too heavy? Unclear? Too much
> info? Too little info? Wrong info? No jokes? Too many jokes? Stupid jokes?
> Can't find info? Too many syllables per word? Dangling participles? None of
> the above?
> 
>         Perhaps you are fortunate enough to have spoken to every one of the
> users of your manual. And they have all told you that the manual sucks. And
> have explained clearly what "sucks" means. And have told you in a clear and
> informed manner exactly how to make the manual not suck. Most of us are not
> so fortunate.

And what's worse, what users _tell_ you and what the facts are may or may
not really be related. Starting from the converse...if a user tells you 
that the manual is good, there are a number of issues (methodologically
speaking) to address before you can really assume it's good. That is, 
does the reader mean that it didn't fall on his toes? That it's
legible? That it has good production values? That the one tidbit that
she looked up was there and accurate? That it keeps the door open?
What is good? Does the user tell you it's good to not hurt your feelings?
To keep the interview to a minimum length? To avoid probing questions?
What's the motivation here?

These issues, not surprisingly, were addressed at great length in
sex studies (Masters and Johnson, etc) in which people would either
deliberately mislead researchers or "shade" their recollection or
simply _believe and report_ something that's patently untrue. 
Additionally, these problems with the accuracy of self-reporting
have been verified in weight=loss studies and other studies in which
the accuracy of self-reported information can be verified. 

Granted, if your users report that your manual sucks, it probably does,
or at least doesn't meet the objectives. But, even if your readers
love it, that doesn't mean that the manual does them any good, or
that they get what they need out of it. For example, there are some
lousy Dummies books out there that are just fun to read. They're funny,
entertaining, and great until you start really looking into the facts. 

That's why we need objective, empirical, thorough, unbiased studies--just
_asking_ isn't enough. 

Eric