Speaking Engagements & Private Workshops - Get Dean Bubley to present or chair your event

Need an experienced, provocative & influential telecoms keynote speaker, moderator/chair or workshop facilitator?
To discuss Dean Bubley's appearance at a specific event, contact information AT disruptive-analysis DOT com

Wednesday, June 07, 2006

Nonsense Surveys

I've worked as an technology analyst & consultant for a long time. One of the things I've learnt is that large-scale surveys are exceedingly difficult to do well, and generally produce either self-evident truths, self-serving sponsored propaganda or just utter nonsense. The general rule-of-thumb is that:

"95% of respondents didn't know what they were talking about. And the other 5% lied."

Surveys of consumers (especially the "would you use this function/service? and how much would you pay for it?" type) are invariably suspect. They're often poorly-worded, with a poor selection mechanism for people, and poorly-interpreted. They tend to be over-represented by people who enjoy doing surveys (ie too much time, probably a bit geeky, and liking the self-validation from someone actually caring about their preferences).

Surveys of businesses have to contend with the fact that the people who actually know stuff or buy stuff (CIOs, architects, IT managers etc) are bombarded by survey requests - in some cases more than 50 per week. They might do 1 or 2 at most, if they have a good reason (something decent in return, or more likely being part of panels where they get to network with peers). Survey companies therefore have to make up numbers by speaking to people who aren't saturated with interview requests. I'll let you guess how much those people know, and whether their supposed job titles are actually real.

And then lastly, quite often nobody ever bothers to sanity-check the results. "Hmm, now why is it that 78% of people have claimed they're happy to sell their grandmother to pay for a bundle of 200 MMS?"..... "well, that's the survey result, so I guess that's true then. I'm sure the question was well-worded & asked to the right people. And we spent lots of money on the survey. Let's put it in a press release." And you can absolutely bet that the PR will never mention any findings that are "inconvenient" and that don't support the sponsor's view of the world.

With all that in mind, the survey-of-the week award goes to Nokia , with a survey sponsored by its high-end N-series division called "Multifunctional Mobiles Make the World Go Round"

Some claimed highlights:
- 68% of Indians use their mobile phones as their "primary camera".
- over two thirds predict a music-enabled mobile will replace their MP3 player
- over a third (36%) of respondents are browsing on their mobile devices at least once a month
- more than one in two (58%) of those questioned would like to be able to control all their household appliances via their mobile device. This is especially true in India (85%)

Actually, some of the stuff in the press release was actually genuinely interesting, and I was sufficiently intrigued to contact the PR company for the more detailed findings. I haven't had a chance to go through everything, but some alarm bells started ringing with about 20 seconds.

One chart caught my eye in particular. "% who regularly use MP3 function on their mobile phone". It lists two options "Yes" and "Function not available".... presumably the rest being "No" or "Don't know". For India, Yes = 88% and Not Available = 4%.

Yes, that's right - 96% of all mobile phones in India have MP3 players built in. And 88% use the capability on a regular basis. Allegedly.

So I skipped to the end, to the important bit:

"ICM Research interviewed 5,500 respondents aged 18 – 35 years old across eleven countries between 27 February and 18 March 2006. There were 500 interviews conducted in each of the following countries: Brazil, China, France, Germany, India, Italy, Japan, Spain, Saudi Arabia, UK and USA. All interviews were conducted online with the exception of Saudi Arabia where interviews were conducted via telephone."

So they've selected an audience that has a PC and Internet access. And that is solely focused on the more "sophisticated" age group. In this article, it suggests that India, a country with a population about a billion, has fewer than 5m PC households. (It also has the fascinating line "Cyber café’s are the primary mode of access. 60-70% of internet users access the net at cyber cafés"... but do you seriously expect people who pay-per-hour for online time to spend it filling in surveys?).

And how about this line "Almost half (44%) of respondents are already using their mobile phone as their primary camera." What exactly does "primary" mean? The one they take most photos on? The one they take their most important photos on? The one they carry the most? The one they'd pick out of their pocket if they were carrying both a phone & camera at the same time? The one they show other people pictures on most?

It may well be that there's some good, accurate, meaningful stuff in the survey. But unfortunately, by including what appear to be spurious or ambiguous results, it throws the credibility of the whole thing into question.

Bottom line - if you're a vendor thinking about using a survey... make sure you have someone appropriately cynical check the questions, survey methodology, and results, before putting them out into the public domain - or, worse still, use them as the basis for business plans & investments.

2 comments:

Michael Mace said...

I agree with you about surveys, Dean. It's always critical to check the methodology (usually in small type at the bottom of the press release, if it's given at all), and in order to really evaluate a survey you need to see the questionnaire and know how the respondents were selected.

Often the scariest thing about surveys like this is that the company sponsoring the survey actually believes the results. This inability to distinguish between PR and reality would be enough to get you hospitalized if you were an individual, but it's standard at many companies.

Dale Vile said...

Hi Dean - as someone who designs and executes surveys for a living, that's our core business, I actually applaud this analysis of the research we often see quoted in press releases. Designing a survey that is objective and useful is perfectly possible, even with a self selecting audience like a Web based study, but it takes skill, experience, industry knowledge and a responsible attitude. We frequently turn down "opportunities" to conduct sponsored research for vendors who clearly just want a market research firm to substantiate pre-defined headlines. Trouble is, there are a lot of firms out there who will interview 100 CIOs for you for £2k (????) and just give you the answer your want – we know this because we have to justify the more responsible and objective approach on a regular basis when vendors are comparing the primary research services of an industry analyst firm like ours with the tick-and-bash questionnaire jockeys out there.

Good research is important though. To put the other side of the argument, I constantly hear in industry analysts and other commentators make ludicrous claims about how advanced a market is, what they see people “doing” out there, and what the market wants or needs – when we have solid research showing otherwise. I think industry analysts “guessing” or extrapolating based own interests, the influence of the vendor shilling, or theories they develop in their own head or based on a few vendor conversations or a handful of exceptional customer encounters are even more dangerous, especially as people often regard them as “authorities”. I am surprised more people do not question the Emperor’s new clothes with many such characters and ask them to explain and justify the foundation for the flawed opinions they put forward as facts.