Moving from Static A/B Testing to Dynamic Testing

A/B testing

Last month Blue Conic stepped up to speak about their thoughts regarding campaign testing. Their approach for testing is to have it “always on”. To begin, the hypothesis that A/B testing no longer meets today’s marketing challenges was laid out. Three key problems with A/B testing were noted:

  1. First, the process relies on measuring the “average” customer, which is not a good proxy for individual customers.
  2. Second, there is an inherent delay caused by the testing process. It takes time to run a proper test. This prohibits marketers from reaping the benefits of the better performing message until after the test is complete.
  3. Third, the tools used to measure results from A/B testing will often declare a winner based on a significance threshold, not a sample-size threshold. If we are testing on a single conversion event then we also run the risk of missing out on events that can have impact on our bottom line.

Blue Conic defines “always-on optimization” as a method that utilizes machine learning to serve the optimal message from an available set in response to actions or derived intent from an individual or segment in real-time. They point out three advantages of an always on approach:

  1. Quicker speed to performance for the leading variant.
  2. Progressive decisioning against a baseline.
  3. Machine learning eliminates human bias.

The table below lines up the differences between traditional A/B testing and an always-on optimization model. The advantages of an always on approach include the ability to learn continuously and serve up the most appropriate content for the individual visitor.


Key takeaways – traditional A/B testing has its place in smaller contextual scenarios, but as organizations grow and their reliance on online traffic increases, there comes a time when a more robust testing scenario is needed. An always-on methodology allows for real-time learning that removes the human bias. It is focused on multiple outcomes, not just a single conversion metric. It is also suited for cross channel testing.

Greg Timpany directs the research efforts for Global Knowledge in Cary, North Carolina, and runs Anova Market Research. You can follow him on Twitter @DataDudeGreg.


Speak Your Mind