The risks of using someone else's numbers & forecasts

I've been at the Avren Femtocell conference in London today, and the Femto Forum awards dinner last night.

One thing that struck me is that I've seen the same set of Cisco VNI data traffic forecasts used by almost every presenter. I'm sure you've seen them too - 59x mobile data traffic growth 2009-2014, lots of it video. Exabytes, petabytes etc. About 5% of all Internet traffic going via mobile access and so forth.

Yes, it's good stuff. Really detailed, lots of interesting insight - there's a lot of work that's gone into it - I was on a analyst conference call with the VNI team recently and was impressed. But what rather worries me is that lots of people seem to be using it as a central part of their marketing, without appearing to understand what's in it, or how the forecasts have been constructed.

They certainly don't appear to have spoken to any of the VNI project's authors, understood the methodology or assumptions, or even to have bothered reading the fine detail. I upbraided one speaker today for citing the 66% of mobile data traffic which is video, as being a justification for needing more QoS, referencing things like videotelephony.

Yet the table in Cisco's data which describes this splits out the 2.3m TB/month of mobile video about 3/4 of the way down this page. 1.4m TB/mo is PC and tablet-based Internet video - basically YouTube, iPlayer and the like being watched over a USB modem or 3G module. The sort of application that can be expected to be buffered, rate-adaptive and so forth. The proportion likely to be QoS-based is tiny. Conversely, video communications (Apple FaceTime, Skype Video, the roughly 17 people using IMS video-sharing and so on) accounts for much less.

I sincerely hope that strategy and product planning folk at vendors and operators are being a bit more rigorous about finding *relevant* and specific addressable statistics for their own products instead of convenient data "soundbites" available for free on the web.

But given the number of pitches I hear using the same data deluge/wave/tsunami/apocalypse language to justify their femto or wifi offload, or policy-management products' value proposition, I'm starting to get worried.

Especially as none of them also bother putting up a chart of signalling traffic, which if anything is the cause of more problems to operator networks than the bulk of data downloaded. (NSN is the main exception I can remember seeing to show signalling growth trends - and I think they use their own data forecasts rather than relying on Cisco's too).

Another sobering example of mistakenly trusting web numbers is Amdocs, which is obsessed with the magical word "trillion" as a means of drawing attention to its (worthy) solutions for service providers - personalisation, billing, operational efficiency, business models and so on.

In last October they were saying "By 2015, if not sooner, experts project that the world will include a trillion networked devices providing consumers with a dizzying array of applications, services, connections, and experiences......... The business benefits could be enormous: managing (and charging for) customer access across a trillion interconnected devices, services, and locations; generating sizable new revenue streams from infinitely larger personal and community networks; and becoming the long-term trusted partner of millions of customers around the globe."

At the time, I raised an eyebrow and challenged their executives to a spread-bet with me, at say 0.000001 cent per networked device at 31st December 2015. I'm a seller at 500 billion.

Now, they've upped their game in a recent white paper. "Within the next decade, the convergence of connected devices, predicted to reach 7 trillion by 2017 (World Wireless Research Forum, Technologies for the Wireless Future: Wireless World Research Forum,
Vol.3, November 2008), and customer demand for ubiquitous connectivity, and pervasive digital content and applications will give rise to the connected world."

7 trillion by 2017. 1000 devices per person. 3000 devices per person living in developed economies.

Time for a common-sense check. I can envisage a long, long term future with lots of RFID tags on products. Maybe massive distributed sensor networks, like the military's "smart dust" concepts. Even if it occurs, I'm pretty sure only the tiniest fraction will be of any relevance to Amdocs or its telecoms customers. When my car's indicators are controlled via wireless to the dashboard, to save on the weight of a wiring loom, I'm pretty sure I won't be being billed per corner by my mobile operator.

Not only that, but the heritage of that number is fascinating. It seems to have first surfaced in a book published by Wiley, and edited by Prof Tafazolli of the Wireless World Research Forum in 2004. It's mentioned in a 2005 presentation here on slide 26. It may well have been a worthy piece of work (unfortunately I don't have the underlying documentation) - and it seems to have been re-reported in the 2nd and 3rd editions of the same book.

But 10 minutes trying to track it down in detail yielded this other document from 2009 by WWRF, which contains the footnote "Originally, the vision was developed for the year 2017. However, work within WWRF has extended the time period to 2020". In other words, even the forecast's original authors now no longer believe it's true and have pushed it out three years.

Personally, I'm pretty skeptical of 2020 as a target for 7 trillion too, but I don't have sight of their full methodology, definitions or assumptions so I'll temper my criticism.

Come on Amdocs, drop the trillion nonsense. It's not helping your message. If anything, it does the opposite, wrapping something important and opportunity-laden, with a veneer of irrelevant and non-credible big numbers just for the sake of it.

Or of course, you can put your money where your mouth is, and take me on at my bet.

Now to be fair, this is all the "science" of marketing. But to me, it illustrates the dangerous ways that poorly-understood numbers get recycled, without people sanity-checking them, or thinking through the real implications if they're wrong.

I've got similar doubts about the oft-repeated line "the next billion Internet users will first get access on a mobile phone". I think that's fabricated rubbish - perhaps the largest myth in the industry. I'm not utterly convinced by assertions that 85%+ of mobile data will be used indoors in the home/office by 2013 either - although the vagueness of whether "mobile data" includes WiFi access to fixed broadband would probably let it scrape past. A lot of hysteria about mobile apps is based on irrelevant numbers too, as Tomi Ahonen dissects with forensic detail here.

Bottom line - be very, very, wary of people citing statistics when they don't know the original source and haven't made efforts to understand the definitions, methodology and assumptions.

Sales and marketing messages are *much* more credible when they're based on real, specifically-addressable figures - ideally developed on a custom basis to illustrate the point they're being used to make. Yes, it's always easy to poke holes in forecasts - including my own - but that's no excuse for skimping on validation of the source (or even just smell-checking for plausibility) wherever possible.