Re: Designing a survey

[Peter Collins <peter -dot- collins -at- BIGFOOT -dot- COM>]

You asked for "words of wisdom regarding surveys, sample questions ... etc"

1. Decide what you want to USE the results for - what problem are you trying to solve? How will the information help you reach a
solution. If you have a decision to make between alternatives that will provide different features and benefits for your public,
that is fairly easy to address (list all the competing features and ask for a score for each). If you propose to upgrade your
product, however, and are looking for suggestions on what to add or alter, that will be harder to achieve, for you are asking your
public to be inventive for you, not just tick boxes. Hardest of all might be if you are losing ground to your competitors and don't
know why (how do you ask - are you leaving us because of a)... b)... c)... d) other - please specify? - and this is hard to analyse
because it mixes 'closed' and 'open' (see below) responses in one question, and also because the closed options rather lead the
witnesses and bias the response to the open alternative.)

2. If you only need a few replies, but need to know in depth all the underlying reasons ('open' quesetions eg "why do you feel that
way about our service?") then an interview survey is best, though they can be made self-administering. Tricky to design, needs
trained interviewers (but I am sure there will be professionals out there who can guide and train you, or do the whole thing for
you. If you do not use this approach but still want to know the background reasoning you will have to make some clever guesses about
possible reasons, and go for the closed question conditional approach ("if you answered 'no' to Q7, go to Q9 otherwise go to Q8 ...
Why were you dissatisfied? Because a)... b)... c)... " etc). This puts less load on the responder than the 'give reasons' open
approach but still isn't entirely painless. It works best where you get fined for failing to respond (like your tax return) or where
it is somehow fun or rewarding to fill in (as when the first x replies get a new car, free). Of course you can only do statistical
analysis on closed questions unless you are prepared to do a detailed analysis of all the open responses and hope you have so many
of them that the same suggestions turn up enough to give you statistically meaningful cell numbers. But often, just having some open
responses is better than nothing, when you are casting around for a new direction and need some outside input to kcik-start the
momentum.

3. If you want validly to compare your results with some other survey it is almost mandatory that you use exactly the same wording,
methods and formal population sampling as the one you will be compared with.

4. Formal Population Sampling is only required if you want to extrapolate from the results and accuracy is important. (But if it
isn't - then why are you dong the survey?) Rule of thumb: "If your marketplace is teen-age males, don't seek responses from
geriatric females." If you have to estimate demand ("40% of all hardware retailers will buy our system" - because 40% of your sample
said they would) even assuming your questions elicited the truth (doubtful) you better know whether your sample was sufficiently
representative (most unlikely). Did you only sample the ones near your plant? Did you only sample the small shops? Did you only
sample the ones in malls? On average, were your 40% acceptances identical to the 60% who rejected you, or were you accepted by small
shops and rejected by the chains? Look up 'stratification' in a sampling survey techniques text. For the right sort of question,
none of the math is hard, but the power it adds to your reliability is immense.

5. The math. Unless you are measuring cardinals ("how tall are you?", "how many cars do you own?") ordinary averages are useless,
and you MUST tabulate the replies ("of the 43 who replied, 10 want blue, 20 want red and 13 don't know). When you are dealing with
ordinals ("blue is my number one choice, followed by red, green and lastly yellow") we are into the grey, ghostly land of
non-parametric statistics, and you WILL need professional help to ensure a valid interpretation of the full set of results. You can
get a half-assed result by tabulating only the first choice (and throwing away the other useful information) but if you want to use
all you have gathered, more complex math is needed. Australian elections are conducted that way - we all put down EVERY candidate in
order ("I vote for A, but if he doesn't get a simple majority, then I vote for B instead, and she doesn't make it, then I would
rather have C - and so on. Only three people in the country understand the system fully, and two of them are liars. I jest, but not
much.

6. Pay a consultant statistician (try university maths or business department,perhaps, for a name) a couple of hours fee to help you
sort out the above and related that I have not mentioned. It will increase the bang you get for your business out of the time and
bucks you put into your survey..
========================================================
Peter Collins, VIVID Management Pty Ltd,
26 Bradleys Head Road, MOSMAN 2088, Australia
+61 2 9968 3308, fax +61 2 9968 3026, mobile +61 (0)18 419 571
Management Consultants and Technical Writers
email: peter -dot- collins -at- bigfoot -dot- com                ICQ#: 10981283
web pages: http://www.angelfire.com/pe/pcollins/
========================================================


 From ??? -at- ??? Sun Jan 00 00:00:00 0000=