The truth is in the Test: How to Definitively Prove the Incrementality of Your Programmatic Program

Thursday, December 01, 2016 Listrak 0 Comments


It’s been more than a century since merchant and advertising pioneer John Wanamaker famously said, “Half the money I spend on advertising is wasted; the trouble is, I don’t which half,” and when it comes to digital marketing attribution, the situation has sadly not improved. Marketers have long known that cost per click reporting and last click attribution models are broken, but with no alternative, they continue to be used to measure the effectiveness of programmatic - automated display advertising targeted to specific audiences based on artificial intelligence and real-time bidding.

The good news is that by following the advice of the Father of Advertising, David Ogilvy, “Never stop testing, and your advertising will never stop improving,” that model can be disrupted and replaced by a true, transparent view into the incremental revenue increases driven by programmatic ads.

Before we explore the test that allows us to do that, let’s take an honest look at the problems inherent in the current cost per click reporting and last click attribution models, specifically for a merchant spending on both programmatic and email marketing:

  • If you’re using different vendors for each channel, with both using a 30-day attribution window, they are often both taking credit for the same conversions, inflating the true return on your investment.
  • Thanks to the prevalence of click bots and fat finger clicks, a great number of reported clicks are not meaningful, but are driving up ad spend and setting false expectations for programmatic ad performance. 
  • Marketers have gotten so accustomed to justifying ad spend based on falsely inflated click counts that moving to a more transparent model showing true incrementality could reflect poorly upon them in the short-term even though it is the wisest thing to do in the long-term. 
  • Not all clicks are created equal. It is important to understand that when a shopper clicks through an email, she is an engaged subscriber showing intent to explore a product even if she gets interrupted before converting. When she later visits weather.com, for example, her true intent is to find out the temperature, however, she may be interrupted by an ad that reminds her to go back to your website and complete her purchase. But would she have anyway? 

To-date we have not had a model that reflects real-world consumer behavior and scientifically proves incrementality in a single channel. The only truly accurate way to quantify the incremental revenue lift provided by programmatic is to conduct a PSA (Public Service Ad) test.

In a PSA test, a number of recent site visitors - the test group - are targeted with a merchant’s branded ad, and an equal number - the control group - with a PSA, for example, from the Red Cross, that has nothing to do with the merchants’ brand. By doing this and comparing the number of conversions from each group for each type of campaign, it becomes immediately apparent how many of your retargeted visitors would have returned to your site and purchased even without seeing your branded display ad.

Example results may look like this:


From this example, we can conclude that of the 60 conversions vendors and marketers would typically attribute to programmatic efforts, half would have occurred anyway.

So why is this so important?

It is critical that marketers understand what digital marketing efforts are truly driving incremental revenue. The model above reflects the real world incremental lift attributable to programmatic. But what’s more, this type of testing is instrumental in optimizing programmatic efforts.

This particular test indicates that post purchase messages are the most effective, but that programmatic is seemingly ineffective in converting higher funnel prospects. Armed with this information, a retailer can do further tweaking and testing to maximize programmatic spend.

Looking for more information on display ads? Read our blog "Cross-Device Personalization Tactics for Display Ads" or contact us for more information.






Keith Brown
New Product Development




Matt Vollmer
Product Manager

0 comments: