Jun 04, 2012

Determining Statistically Significant Differences Between Display Advertising Click-Through Rates

Have you ever run a display marketing campaign through multiple vendors and afterward tried to determine which vendor performed the best?  Often times you end up comparing the click-through rates (CTR) between the vendors and concluding that the vendor with the higher CTR performed better than the vendor with the lower CTR.  But, can you really be sure of this conclusion?  What if the vendor with the lower CTR is really not worse than the other vendor?  You need to make sure your digital marketing dollars are well spent.

Thankfully, there is a revolutionary solution available to help make these numbers meaningful…good old fashioned statistics!  Before you stop reading this post let me assure you that it is not as boring and as difficult as you think it is.  In fact, you can easily find calculators or spreadsheets that do the heavy math lifting for you.  The important thing is that you know how to interpret the results and that is what we will examine now.

Let’s say we ran a display ad campaign and we used two vendors for our ads: Facebook and Pandora.  Let’s also assume that for Facebook there were 2,600 impressions and 35 clicks, which computes to a CTR of 1.35%.  And for Pandora there were 2,500 impressions and 45 clicks, which computes to a CTR of 1.80%.  Which vendor performed better?

On the surface it appears that Pandora outperformed Facebook because even though there were 100 fewer impressions on Pandora compared to Facebook, there were 10 more clicks, giving Pandora a higher CTR.  However, when we apply what is known in statistics as a two tailed t-test we actually discover that the difference in CTRs between the two vendors is not statistically significant, or in other words, we cannot say based on this data that Pandora performed better than Facebook. 

Knowing this is incredibly helpful when making digital marketing spend decisions.  In this example, it would be a mistake to assume you will get more return on your investment with Pandora digital ads instead of Facebook.  Had the results been statistically significant after doing the two tailed t-test then we could conclude that Pandora display ads performed better than the Facebook ads, but in our example that was not the case.


[Contributed by Ben Hill]