Question the Metrics
I love this post. Originally titled, Misleading Types of Graphs For The Media, it not only tears down the repeated claims and promoted metrics of video dominance, traditional media reach and more, it also questions info we are presented and often take for granted. Much like Nate Silverman discusses in The Signal and the Noise, be sure you are discerning the correct application of metrics and context. Long, but so worth your time. -pw
The Importance of Skepticism
My friend Avinash Kaushik posted a wonderful article the other day about the importance of analysts to have a skeptical nature. I absolutely agree with him. Skepticism, along with fact-checking, and a strong urge to take a step back to look at things from the larger perspective, is the key trait of anyone working with metrics and media strategies.
But I want to expand on his article because there are several types of graphs that I see all the time. Each painting a completely misleading picture. And each one of these is dominating the media landscape. And constantly used in presentations at pretty much all of the big media conferences.
So let’s talk about this.
The curse of the market share graphs
The first truly misleading graph metrics is the one most people use to indicate market share. Either of their own business, their audiences, or the things that we use to get our publications to the market.
For instance, take a look at this graph:
Here you see two different things/products/whatever and how they changed from 2011 to 2016. So, what conclusions do you get from this?
The conclusion most people come to is that the yellow market has experienced catastrophic decline. While the red market is dominating more and more.
Right? Eh… no. What if I told you that the yellow market hadn’t declined at all? And to prove this, here is the exact same graph but using the raw numbers instead of a percentage.
What’s actually happening here is that the market overall has expanded. The yellow market has experience only a minor growth. While the new red market has created an entirely new market on top of the old one.
You see how bad this is?
The first graph forced you to come to entirely wrong conclusions.
So, one of many places where this is happening is when people talk about the rise of mobile. For instance, people keep talking about how laptop computers are dead, and the graph they use is the one below.
What you see here is the same as before. As a percentage, mobile has been growing rapidly over the past five years at what looks like the expense of laptops. And if you see this graph (or those similar that are widely circulated by media executives) you may indeed think that laptops are dead.
What makes this graph even worse is when it is backed up by graphs showing volume of sales, where, again, mobile is dominating. This should not come as a surprise to anyone. We buy a new mobile at a much higher frequency than laptops. Especially today where we have so many devices to play with.
So, are laptops dying?
Is it game over for desktop computing? Nope… not even close.
Because here are the exact same numbers, but this time drawn using their real data instead of as a percentage. And what you see here is that laptop use per person is the same today as it was five years ago.
The growth of personal laptop usage may have peaked, at slightly less than 3 hours per day. But it shows no sign of decline. What has happened instead is that we now have a new mobile market on top of the old one. It’s not killing the laptops. It’s extending it.
Think about how many times people have told you that mobile is killing laptops over the past few years. Both at media conferences and on Twitter. It’s just insane how many have been fooled by looking at percentages instead of the actual numbers.
Don’t be one of those people. Always insist on seeing the real metrics.
There is, however, an even worse example than this. And that’s when we see studies from newspaper associations. Almost all of them are using completely misleading graphs by default. Either because they don’t know any better. Or, worse, because they are trying to hide the decline that we all know is happening.
The main reason why these graphs are so bad is because not only are they based on total percentages, they are also leaving out critical data.
Think about it like this. Imagine your market was defined by three types of audiences. Print, digital and not reached. Not reached are the people who have stopped reading newspapers as we know them.
Now imagine that we map this change over the past five years, we might see something like this:
What we see here is that the market overall is going up (due to the growth in population). But the share of people who don’t subscribe to newspapers is increasing. For newspapers, their market is in heavy decline. And even though digital is growing, it in no way makes up for the decline.
Sound familiar?
But then the newspaper associations do a study. And instead of looking at the market as a whole, they decide to only look at their remaining market. In other words, they decide to simply ignore all the people who have been lost, and then map the rest as a percentage.
The result is a graph that looks like this:
Now we have the same problem as before. This graph is not only completely useless, but also completely misleading. Now, you no longer see any sign that the market is in trouble. And while digital is growing, it looks like print is still going strong.
This is terrible.
One example of this was when the Canadian newspaper measurement agency Vividata published this study:
Just look at this. This graph actually makes it look like newspapers are winning. They are up from 77% reach to 81% reach. And even the magazines are doing fantastically. Sure they are down by a bit, but it’s nowhere close to anything that could threaten their future.
This is great. The Canadian newspaper industry has apparently found a way to keep winning with print. No problem here. Right?
Eh… no. The reality is, of course, that the media industry in Canada is in just as much trouble as newspapers anywhere else in the western world. And that their circulation and advertising revenue is in a terrible state… like what we see here:
I’m not saying that Vividata’s data is wrong. It’s probably an accurate reflection of what they measured. But what I am saying is that they designed the study to only look at things that had no real use for the challenges newspapers are faced with.
As they say:
Seventy percent of newspaper readers still read a printed edition daily. That’s down from 90% five years ago. While print remains the leading source for most newspaper readers in Canada today, digital and cross-platform continues to grow.
No. Just no. This is an idiotic way to look at the data.
The future of the newspaper industry is challenged by external factors. So it makes absolutely no sense to do a study that only looks at the internal factors. This is stupidity at its worst, and it has a serious consequence.
When the Spiegel’s Innovation report was leaked last week, one of the key problems they found was that they ‘lacked a sense of urgency’. And of course they did. If you are constantly being told print is still leading and digital is growing, you don’t feel any need to change. Things sound like they are going fine.
But this is not the reality.
So, be skeptical about newspaper studies. Almost every one of them is disastrously misleading. Especially so if they are based on percentages. Whenever I see a newspaper study that has percentages in it, my red warning lights begin flashing.
Not understanding what is being studied
Another huge problem that we see with studies and metrics about the media industry is that we often see that studies that are measuring one thing are used to prove something completely different.
One example is this slide from BBC’s Esra Doğramacı at the News:Rewired conference.
I have lost count of how many times I have seen media people use this study in relation to video. And people were absolutely lapping it up at the conference. Several people tweeted that video was the only thing to focus on in the future.
But this is not what that study is saying. The Cisco study has nothing to do with video consumption. It doesn’t tell us anything about whether people actually watch video more than before in relation to the media.
There are a number of reasons for this.
Firstly, the Cisco study is looking at the volume of network traffic going through their routers for all internet traffic as a whole. It’s not looking at video consumption specifically.
This alone is highly misleading. You think that an increase in video use means people also watch more video. But that’s not necessarily true.
Consider this.
Imagine that it’s 2010, back with poor wifi and relatively slow mobile devices. You look at the consumption patterns of a single person. So, you have a person reading 10 articles, at 2 MB, and spending 2 minutes reading each. Then this person also watches one video, at 360p (again because of slow connections), at 16 MB, spending 6 minutes.
What you get is this:
As you can see, 44% of the data usage metrics and 23% of the time spent goes to video, the rest goes to the articles.
Now fast forward to 2016. We now have 4K and amazingly fast wifi connections, so now we have got this:
You see what’s happened here?
In this example, the consumption pattern stayed the same. This person is still only spending 23% of his time watching video. But look at the data usage metrics. It’s now 96% going to video.
And this is what the Cisco study is predicting. It is looking at the size of the video files and it’s projecting how much data that will require in 2019. Cisco is not talking about time spent.
Read the rest of this post HERE
Written by Thomas Baekdal
