One of the things you learn early on in any intro-level statistics course is that the mathematics behind data analysis is, by itself, fairly sanguine about how it is used. Formulae are not terribly picky about the ends to which they’re put to work, and so any professional statistician takes care to apply those formulae, and present results, with discretion and care.
Data are easy to manipulate, precisely because there’s usually more than one way to perform an analysis. And because, “no significant effect could be determined” rarely makes for interesting reading, there’s a great temptation to dig deeper, and see if another analytic approach can’t “tease out” the truth.
When I read a headline like, “Half of all time spent on mobile internet is spent on social networking sites,” one big question comes to mind: what was the sample?
GroundTruth’s release claims that the sample size is about 3 million mobile phone subscribers. Which sounds appreciably huge, until you realize that there are about 285 million mobile phones in the US (see CTIA Semi-Annual Wireless Survey, PDF.) So their sample covers about 1% of US Mobile handsets.
That’s certainly still a reasonable sample size, if the sample is representative. And to determine whether or not it is, we need to know at least something about the networks on which these handsets operate.
Why? According to Ground Truth, “…Ground Truth™ captures usage directly from network data provided by mobile operators and other data partners.” That’s a rich and reliable source of data, to be sure: but it says nothing about the representativeness of the handset users on those networks, as compared to the US population at large.
Ground Truth isn’t being very forthcoming about these data sources. In response to a question of mine, they tweeted: “We have a large sample of mobile phone users from a diverse group of mobile operators. Confidentiality limits disclosure.”
Well. Confidentiality is fine and understandable, but if you can’t cite your sources, don’t publish your conclusions.
The press release regarding this “study” is fairly vague in terms of data, but perhaps offers one or two clues. They go out of their way to mention that “mobile-centric social networking sites such as MocoSpace and AirG are better at engaging consumer than are with PC heavyweights like Facebook and MySpace.”
MocoSpace is an “off deck” social networking website, which means that access is not restricted to or sponsored by any particular network operator. They have a mobile-ready site, which renders well on an iPhone (although they do not have an iPhone app.)
AirG is a somewhat different story. It’s a mobile-based chat service, more or less an instant messenger client for your phone. It is marketed under about two dozen different brand names, including Virgin Vibe, Boost Hookt, Amp’d Chat, TELUS Chat Central, Amp’d Lounge….
Hey, wait a minute! These all have something in common. Most of these seem to be affiliated with pre-paid or pay-as-you-go mobile phone providers. That might tell us something about the audiences using these social networking sites, as well as whether or not these users’ behavior is representative of the mobile internet audience as a whole.
It might also be important to know whether or not either of these two site operators are clients of Ground Truth.
Social networking usage on mobile is important, there is no doubt about that. But publishers and advertisers need the whole story if they are going to make an informed decision about where to commit resources. To say that pre-paid mobile subscribers spend 60% of their time on social networking sites is important, relevant, and useful. But it is not the same thing as saying that “Half of all time spent on the mobile internet is spent on social networking sites.”
For a company with “Truth” in their name, Ground Truth ought to be going out of their way to tell the whole story, if for no other reason that to ensure that their data are taken seriously.