Web Browser Stats: Problems With Data Gaps

Last week in Quick Analysis of Web Browser Stats I provided a line chart (“timeline”) to augment Ed Bott’s discussion of Google Chrome’s inroads in the browser market (from Chrome takes a bite out of IE and Firefox). Ed occasionally revisits the relative market share of the different browsers, but it’s not a regular monthly feature. In his article, he compared three time points, September 2008, September 2009, and March 2010. I suffer from a unique symptom of OCD called Data Loss Aversion, so I noticed right away that there were eleven missing months between the first two timepoints, and another five in the gap between the last two.

I don’t have access to Ed’s web site statistics, but I can use my own data to illustrate the issues with gaps in a timeline. This chart shows browser share on my site for the same months that Ed reported. Ed and I have different demographics among our followers, so IE makes up a larger share of my visitors. But I’ve seen the same steady rise in users of Chrome.

Because Safari and Opera represent such a small and relatively constant percentage of browser usage, I’ll leave them out of the rest of this analysis.

Interpolated Data

As soon as I noticed the gaps in the data, I wondered what the browser shares looked like during the intervening months. The first inclination is to assume that the data follows the lines between the known points. This is not a good assumption, since the lines themselves provide no information about how the data behaves between known points.

Note that in a line chart, we put distinct markers to try to indicate where we have actual measurements, and we use straight segments to join points to help the reader trace measured points for the same series. Using curved or smoothed lines between actual points would imply incorrectly that the intervening data followed particular trajectories.

Actual Data

The actual data is shown here for all months between September 2008 and March 2010.

Here I’ve plotted the three widely spaced points as solid markers and the intervening months as unfilled markers. The assumption that the data follows the drawn lines isn’t accurate, though it’s not too far off.

If we didn’t have the intervening data points, it is a matter of faith to believe that the actual behavior followed the connecting lines.

What If? (Random Data)

What if the data was random, and our three “known” points came from sampling from normally distributed data sets? To illustrate this, I used the same means as my real data, and slightly elevated standard deviations, and generated normally distributed random points based on these statistics.

The randomly generated points no longer appear to be following the connecting lines.

In fact we can construct run charts for our three main browsers, as I’ve done non-rigorously below. The three dotted lines around each set of points denote the mean, the mean plus three standard deviations, and the mean minus three standard deviations. (See Introducing Control Charts (Run Charts) and Statistical Process Control.)

All three data sets appear to represent processes controlled by normal random fluctuation. There is no systematic variation in any of the data. No points in the three data sets fall outside the ±3 SD control limits, or two out of three beyond ±2 SD (not drawn, but trust me). In no case do we find more than six points in a row all above or all below the mean (the rule usually states that nine points indicates out of control). We never see more than half a dozen points in an alternating up and down pattern (rule is usually 15 points), or more than half a dozen points all increasing or all decreasing (usual rule is 7 points).

If we plot the actual data as run charts, we can conclude that  there is a systematic variation in the Chrome data. Too many consecutive values for Chrome are increasing, for example, which indicates nonrandom variation.

While the full data set does confirm that the trends seen by the first three time points are real, without studying the intervening points, we cannot tell the difference between systematic trends and randomly sampled points.

Related Posts:

• SPC Approach to Browser Stats
• Quick Analysis of Web Browser Stats
• Bar Chart Value Axis Scale Must Include Zero
• Introducing Control Charts (Run Charts)
• Smooth Talking Lies
• Types of Control Charts