The Problem With Polls: “Don’t ask me”


A Tangible Manifesto for Better Research


The biggest problems predicting consumer behaviour arise when you ask people to do the predicting about themselves.

Sometimes the very purpose of the exercise is to help answer the questions “will they?”, “when” and “how often?”. However, we know that “would you?”, “will you?” and “when might you?” are all dangerous questions in isolation and a researcher cannot rely on the answers to these as safe indicator of outcomes.

The problems with polls in particular has been well documented over recent days and has raised questions of confidence about polls and surveys, whether they are online, by telephone or conducted face to face. Not least of the issues is coming to terms with three potential sources of bias:

Avidity bias – when people are not particularly engaged so can fail to explain the true reason why they respond a certain way to a question, even to the point of whether their response is “true”. The reality is that they don’t always spontaneously know why they think, feel and do what they say they do.

Self selection bias is often pitched at online panels which require people to sign up, sign in and actively participate. The argument is that research becomes a study only amongst people who want to participate in research. At worst there is a challenge that respondents are self defining and manipulate their profiles to conveniently fit the screener for the survey in question.

Social desirability bias has, like it or not, had a lot of press. The theory is that being asked questions can make people shy about what they say if they think their response is likely to be unpopular or a minority view. In other words they modify their response to take account of where they think the herd is going so they are not left exposed. People like this love to know what everyone else has said or voted before it’s their turn.

Of course, good research takes account of these biases and calibrates models and adjusts to improve accuracy. But in spite of the best efforts of the best experts, errors, misjudgements and unexpected outcomes can all occur. So, where does this leave research, can it be trusted; is it worth bothering if it can produce results that end up to be wrong?

Our answer to this is a question: Can you trust yourself to work out what is going to work best for your customers without involving them in your idea creation, product and service development and activity executions?

The problem is people doing bad research (in terms of motivation, design or execution), not research itself. Tangible’s Manifesto for better research is not carved in stone, but it is based on tried and tested principles we have been following for the last 15 years:

    1. Good research is a good thing. Actually, it’s invaluable. It is the means of understanding what people want, think and feel and can help inform better decisions about what to do to make customers happy
    2. Mixed methodology approaches mitigate research bias and improve the potential to see more clearly by taking different perspectives
    3. People are notoriously bad at explaining their own behaviour – that’s the job of the researcher
    4. Make the research process as representative as possible and simulate the buying environment, the decision process and go through the action – of identifying, assessing, choosing and buying. (Take a tip from the exit poll methodology here – where rather than asking how people voted, electors are given a duplicate ballot paper to complete and place in a box)
    5. Observe and interpret rather than ask. But engage and discuss to confirm and diagnose once a hypothesis is built
    6. Don’t cut corners – use random sampling if you seek the wisdom of crowds, and laser sharp recruitment of you want specific target market understanding
    7. Remember, quantitative research has a margin of error. This is real and can impact marginal results
    8. Test, pilot and then scale up. It is a sure way to get better research.

To find out more, get in touch and judge the results for yourself.

Related Articles