Should I Believe Chartbeat’s Data?

chartbeat-analytics-1

I’m a research geek and I like to run 3-4 different analytics platforms at the same time to see how they measure up to each other. It’s a good system of checks and balances because when you need to give good or bad news about a website, you want to be sure the data is accurate.

I’ve been using Google Analytics for years and I also use Mint. Mint cost $30 and it was well worth it. Before Google Analytics did real-time data, Mint had it covered. I’d use Mint for the minute by minute updates and Google Analytics would be the muscle behind it to give historical data. **I also use ComScore and Quantcast, but since they don’t display hourly traffic, the info isn’t relevant for the purposes of this post.**

I finally got around to testing out Chartbeat about a month ago. It was really basic info, but presented in a very clean, easy to read dashboard. Since I started using it, I’ve had tons of obsessive fun watching the leader board of posts in real-time. It didn’t add much info that I didn’t already have from the other platforms, but the dashboard is great to keep handy. For $10/month after a free trial, you really can’t go wrong testing it out.

Today I noticed something strange on one of my sites. There was a relatively large traffic bump at around noon. I noticed this in my Mint dashboard and headed over to Chartbeat to see how it was doing. Huge surprise: Chartbeat didn’t notice the spike and thought traffic was lower WoW (week over week.)

Here’s Chartbeat’s data:

chartbeat-analytics-1

Chartbeat Data

The blue line is today, the gray was last Tuesday.

Now here’s the data from Google Analytics. Same date/time range:

Google Analytics Data

Google Analytics Data

The blue line is today, the orange was last Tuesday. Notice anything different? How ’bout a huge spike that Chartbeat didn’t catch?

For the triple check, I pulled the data from Mint and made a graph in Excel. Mint doesn’t let you compare data ranges, so this is just from today.

Mint Data

Mint Data

Hmmm… That spike looks similar to what Google Analytics measured, but not Chartbeat. Let’s mash it all together.

Analytics Mashup

Analytics Mashup

This graph is not 100% scientific. The ranges don’t match up, but the trends should at least match. The most interesting part is that the grey line is Chartbeat’s data for last week and the orange line is Google Analytics’. Both report that the two hours prior to and post-spike were higher the previous week.

So the main question: today’s traffic spike was real. Why didn’t Chartbeat know about it? I know the traffic to this website is virtually non-existant compared to their other clients, but I have to question when a free platform and a piece of $30 software seems to be more accurate than a SaaS solution.

4 thoughts on “Should I Believe Chartbeat’s Data?

  1. Hi David,

    I get how annoying it must be to see such a difference, but there’s actually a simple explanation. Chartbeat didn’t actually miss it, the reason it’s not such an extreme bump is that you’re comparing pageviews to concurrents, which are two fundamentally different measures of traffic. Pageviews just count pageloads; concurrents are a combination of two things: the number of people coming to your page and the amount of time they spend on your page (http://chartbeat.com/infographics/measure-different). It’s a better measure of not just the amount of traffic but the quality of that traffic.

    That spike was kicked off by twitter to your Bates Motel story and your concurrents immediately broke your 30 day max (see screenshot). However, the engaged time on that story was only 13 seconds, which suggests that while a lot of people were coming to your site they weren’t staying for very long and thus while you had a huge spike in pageviews, the number of people concurrently on your site didn’t spike to the same extent (you had more people visiting but that was counteracted by the fact that they stayed for less time). So yes, Google Analytics and Mint which only capture pageviews and nothing in between will show a huge spike, whereas if your visits are higher but time on site is lower your concurrents won’t show such a large spike. Make sense?

    Feel free to reach out to me at tony at the aforementioned company.com if you want me to talk this through.

  2. Hi Tony,

    Thanks for this information. It helps clear things up. In the past I’ve usually seen correlations between server performance data (concurrent connections, CPU, memory etc.) and pageviews during traffic spikes. It’s interesting to see a measuring method that doesn’t match up as close as the others have. Sometimes it’s hard to take off the old sysadmin hat and realize that “undetected” sudden spikes aren’t the end of the world!

    If 13 seconds of engagement causes such a difference, I’d be curious to see what Vine.co would look like!

Leave a Reply