Most marketers test various concepts through simple AB testing or other more advanced structured testing approaches (fractional factorials, Taguchi methods etc.). A common mistake that most analysts make involves calculating the lift on the overall gains, rather then individual tests. In cases where the test/control sizes are similar proportions, they lead to the same answer, however the reverse is not true. The answers could even lead to contradicting conclusions, as they will be outlined below.
Suppose we have two groups within our file (this could be more, but for simplicity, we'll stick with two). These two groups could be anything, for example, it could be
- Male versus female,
- People who bought X and who didn't buy X
- People who are highly responsive to marketing or non-responsive
- High value customers , Low value customers
Suppose we're trying to examine the effectiveness and lift of a specific campaign on these groups.
Examine the calculation below:
|Total Marketed||Responders||Response Rate||Total Marketed||Responders||Response Rate||Lift||Incremental Responders|
If you follow the above example, the incremental customers coming in from each group is 40+5 = 45. However, when we sum up the test and control groups into one group, where we have 1500 test and 600 control subjects, then the incremental customers is calculated at 50.
How come? Why are the two results different? Which one is correct? Let me know at firstname.lastname@example.org what you think. Answer will come in a few days...