PCBB
BID Daily Newsletter
May 22, 2012

BID Daily Newsletter

May 22, 2012

EXPERIMENTATION - A/B TESTING

We are fans of banks getting more experimental in almost every facet of business. Models and outside advice are helpful, but are no substitute for your own experience in your market with your customers. If you are a marketer, survey professional or software designer you are familiar with A/B testing. However, if you are in bank management, product design or sales, you are probably not as well versed. Today, we want to highlight a method of experimentation called A/B testing to garner knowledge to help you become more effective. At its core, A/B testing is exactly what it sounds like: you have one category of product/service with a main feature that comes in two versions - A and B. In addition, you have a metric that defines success. Maybe it's the lowest cost of funds or maybe it is the most new customers. To determine which version is better, you subject both to experimentation simultaneously. In the end, you measure which version was more successful and select that version for further testing or real-world use. This is similar to the scientific method you first learned back in middle school. First you have to pick what to test. It does not have to be one thing, but it should be limited in scope so you can ascertain what is driving the difference. Disclosure statements, a call to action in sales materials, monthly fees and online layout are the most popular items for banks to test. After deciding what to test, the next question is how to test. While there are services that focus on this, for the basics, a bank can do this themselves by offering the same product using two representative samples of the marketplace that have the same demographics and composition. In addition to live testing, there are several applications and companies that can do this with focus groups for more accurate results. While the downside is that this is faster, the focus group may not exactly be your customer set. In addition, there is always some error in the question "would you buy" as opposed to actually buying. Another tip is to make sure you test both simultaneously, as interest rate, media headlines and other factors can change over time and influence the test. The control group has to be as close to the variant as possible. Before you test, make sure you record related variables such as present interest rates, credit quality of the bank, market share and other factors, so you can look back and determine if the results are still valid. Additionally, suppress the desire to interject your opinion or bias into the test. Get the results first and then draw conclusions. About 30% of the time, test results are the opposite of what we would have thought and are counterintuitive. At one bank, the marketing department had everyone convinced that the variant layout that was being tested was ugly. While true, the variant happened to be more effective at getting customers to respond, which was the goal of the test. The length of time of testing depends on the life of the decision cycle. If you are testing a loan offering, the length of test may be 6 months. If it is a deposit product, the cycle may be 3 weeks, while a marketing offer may be just 5 days. As a rule of thumb, the goal is to set the length of test about 30% past the average sales or decision cycle. Once you have a sufficient response rate that is representative of your market, you can reach a conclusion of which variation drove the most conversion. Finally, test and retest. Every test will result in one of three outcomes, positive, negative and no result (too few responses, ties, etc.). Once you get a positive result, consider retesting to be sure. Maybe the outcome is sufficient or maybe there is room for improvement and you can pit the winner to a second test. The goal is to find what works best. A/B testing is an example of the new quantitative face of banking that we are highlighting at our EMC Conference. There, we discussed how a 5% variation in a deposit promotion in a $1B institution made the difference by adding more than $2mm of additional terminal customer value that would have gone to another institution. This all occurred for a test that cost about an addition 15 hours of time - not a bad return.