Nonsense Surveys

I've worked as an technology analyst & consultant for a long time. One of the things I've learnt is that large-scale surveys are exceedingly difficult to do well, and generally produce either self-evident truths, self-serving sponsored propaganda or just utter nonsense. The general rule-of-thumb is that:

"95% of respondents didn't know what they were talking about. And the other 5% lied."

Surveys of consumers (especially the "would you use this function/service? and how much would you pay for it?" type) are invariably suspect. They're often poorly-worded, with a poor selection mechanism for people, and poorly-interpreted. They tend to be over-represented by people who enjoy doing surveys (ie too much time, probably a bit geeky, and liking the self-validation from someone actually caring about their preferences).

Surveys of businesses have to contend with the fact that the people who actually know stuff or buy stuff (CIOs, architects, IT managers etc) are bombarded by survey requests - in some cases more than 50 per week. They might do 1 or 2 at most, if they have a good reason (something decent in return, or more likely being part of panels where they get to network with peers). Survey companies therefore have to make up numbers by speaking to people who aren't saturated with interview requests. I'll let you guess how much those people know, and whether their supposed job titles are actually real.

And then lastly, quite often nobody ever bothers to sanity-check the results. "Hmm, now why is it that 78% of people have claimed they're happy to sell their grandmother to pay for a bundle of 200 MMS?"..... "well, that's the survey result, so I guess that's true then. I'm sure the question was well-worded & asked to the right people. And we spent lots of money on the survey. Let's put it in a press release." And you can absolutely bet that the PR will never mention any findings that are "inconvenient" and that don't support the sponsor's view of the world.

With all that in mind, the survey-of-the week award goes to Nokia , with a survey sponsored by its high-end N-series division called "Multifunctional Mobiles Make the World Go Round"

Some claimed highlights:
- 68% of Indians use their mobile phones as their "primary camera".
- over two thirds predict a music-enabled mobile will replace their MP3 player
- over a third (36%) of respondents are browsing on their mobile devices at least once a month
- more than one in two (58%) of those questioned would like to be able to control all their household appliances via their mobile device. This is especially true in India (85%)

Actually, some of the stuff in the press release was actually genuinely interesting, and I was sufficiently intrigued to contact the PR company for the more detailed findings. I haven't had a chance to go through everything, but some alarm bells started ringing with about 20 seconds.

One chart caught my eye in particular. "% who regularly use MP3 function on their mobile phone". It lists two options "Yes" and "Function not available".... presumably the rest being "No" or "Don't know". For India, Yes = 88% and Not Available = 4%.

Yes, that's right - 96% of all mobile phones in India have MP3 players built in. And 88% use the capability on a regular basis. Allegedly.

So I skipped to the end, to the important bit:

"ICM Research interviewed 5,500 respondents aged 18 – 35 years old across eleven countries between 27 February and 18 March 2006. There were 500 interviews conducted in each of the following countries: Brazil, China, France, Germany, India, Italy, Japan, Spain, Saudi Arabia, UK and USA. All interviews were conducted online with the exception of Saudi Arabia where interviews were conducted via telephone."

So they've selected an audience that has a PC and Internet access. And that is solely focused on the more "sophisticated" age group. In this article, it suggests that India, a country with a population about a billion, has fewer than 5m PC households. (It also has the fascinating line "Cyber café’s are the primary mode of access. 60-70% of internet users access the net at cyber cafés"... but do you seriously expect people who pay-per-hour for online time to spend it filling in surveys?).

And how about this line "Almost half (44%) of respondents are already using their mobile phone as their primary camera." What exactly does "primary" mean? The one they take most photos on? The one they take their most important photos on? The one they carry the most? The one they'd pick out of their pocket if they were carrying both a phone & camera at the same time? The one they show other people pictures on most?

It may well be that there's some good, accurate, meaningful stuff in the survey. But unfortunately, by including what appear to be spurious or ambiguous results, it throws the credibility of the whole thing into question.

Bottom line - if you're a vendor thinking about using a survey... make sure you have someone appropriately cynical check the questions, survey methodology, and results, before putting them out into the public domain - or, worse still, use them as the basis for business plans & investments.