Significance of Statistically Significant Results in A/B Testing

You are running an A/B test (or multivariate test) but are in a hurry to make a decision to pick the winning page. Should you pick a winner based on initial few days of the data before your tool has actually declared a clear winner? This question comes up quite often during the conversations with the clients.

I always warn against such an approach and advice to be patient and get statistically significant results before pulling a plug on an underperforming variation o declaring a winner. I wanted to share few graphs with you to illustrate my point.

The following example is from a test that I am currently running.
First Two Weeks: If you make a decision without completely getting through the test, you will pick “Yellow” as the winner while declaring “Blue” a loser.

Month Later: There does not seem to be a clear winner.

Few More Days Later: Seems like “Blue” is trending higher. Should you pick “Blue” now?

Finally: We don’t have a clear winner yet.

As you can see, picking a winner or dropping a loser in the early stages of test, without having a statistical significant result, would have been a wrong decision.

Comments? Questions?

---

Looking to fill your Web Analytics or Online Marketing position?

Post your open jobs on Web Analytics Job Board
--------------------------------------------------------------------------