Last month Blue Conic stepped up to speak about their thoughts regarding campaign testing. Their approach for testing is to have it “always on”. To begin, the hypothesis that A/B testing no longer meets today’s marketing challenges was laid out. Three key problems with A/B testing were noted:
- First, the process relies on measuring the “average” customer, which is not a good proxy for individual customers.
- Second, there is an inherent delay caused by the testing process. It takes time to run a proper test. This prohibits marketers from reaping the benefits of the better performing message until after the test is complete.
- Third, the tools used to measure results from A/B testing will often declare a winner based on a significance threshold, not a sample-size threshold. If we are testing on a single conversion event then we also run the risk of missing out on events that can have impact on our bottom line.
Blue Conic defines “always-on optimization” as a method that utilizes machine learning to serve the optimal message from an available set in response to actions or derived intent from an individual or segment in real-time. They point out three advantages of an always on approach:
- Quicker speed to performance for the leading variant.
- Progressive decisioning against a baseline.
- Machine learning eliminates human bias.
The table below lines up the differences between traditional A/B testing and an always-on optimization model. The advantages of an always on approach include the ability to learn continuously and serve up the most appropriate content for the individual visitor.
Key takeaways – traditional A/B testing has its place in smaller contextual scenarios, but as organizations grow and their reliance on online traffic increases, there comes a time when a more robust testing scenario is needed. An always-on methodology allows for real-time learning that removes the human bias. It is focused on multiple outcomes, not just a single conversion metric. It is also suited for cross channel testing.