TechWhirl (TECHWR-L) is a resource for technical writing and technical communications professionals of all experience levels and in all industries to share their experiences and acquire information.
For two decades, technical communicators have turned to TechWhirl to ask and answer questions about the always-changing world of technical communications, such as tools, skills, career paths, methodologies, and emerging industries. The TechWhirl Archives and magazine, created for, by and about technical writers, offer a wealth of knowledge to everyone with an interest in any aspect of technical communications.
Gwen Barnes wrote to express her pain at receiving an "F" in one of
the "evaluate this manual" cards returned by a reader. Cheer up, Gwen,
and anyone else who has been burned this way! Most of these cards
represent a poor excuse for usability testing, and statistically,
they're meaningless in most cases too.
Case in point: we include these cards in our technical notes series,
but they were designed by a manager who knows nothing about survey
design or usability testing. The results of the cards are analyzed
each year to check on our performance. Last year, our
"illustration/graphics" category slipped a bit, with a few more
"average" ratings than the previous year. When we went back to the
reports covered by the survey, the reports that scored the worst were
the ones with the clearest and best-produced graphics (judged by
objective and subjective standards); conversely, the reports with the
highest ratings had graphics that we produced under heavy work loads
and tight deadlines, and that we were a bit embarrassed to publish in
retrospect. The cards are poorly designed, and we didn't get a large
number of replies, yet the manager forced a review of the whole
graphics process, which provided few concrete results. More
importantly, my suggestion that we actually test the graphics on our
audience was ignored. So in effect, we conducted an ivory tower
exercise for no good reason and missed the point of usability
analysis. Go figure!
Yes, you can get useful information from these cards, but unless you
get hundreds of cards returned, or follow up with each respondent, you
probably won't get what you want to get.
--Geoff Hart #8^{)}
geoff-h -at- mtl -dot- feric -dot- ca
Disclaimer: These comments are my own and don't represent the opinions
of the Forest Engineering Research Institute of Canada.
