A/B Testing is the art of learning about your users with experiments: show half of them “A”, show the other half “B”, and measure the response in each group.
How many users do you need to run an A/B Test? If you have too few users in the experiment, you’ll never see a difference between the two groups. If you have too many enrolled, the experiment will never end.
That’s where A/B Buddy comes in: A/B Buddy helps you perform a statistically rigorous sample size calculation, with the slide of a finger, on your phone or on your iPad.
Let’s say your current conversion rate is 10%, and you want to see if a proposed change from the design team can boost that number up to 12% — that is, a 20% lift in the conversion rate. Simply pop open A/B Buddy, set the Baseline conversion rate to 10%, and the Minimum Detectable Effect to 20%, and your answer is waiting for you at the top of the screen.
A/B Buddy includes a Share button, which links to a web page displaying your sample size calculation. Your numbers, however, remain 100% private — all the information is stored in the URL fragment, and is never transmitted to any servers.
By default, A/B Buddy uses the most common statistical thresholds: 5% for the significance level (false-positive rate), and 80% for the power level (ability to detect the Minimum Detectable Effect). Power users can adjust these values after unlocking the Pro Features in-app purchase. But most users will find the defaults adequate for their needs.
Questions? Suggestions? Ideas? Feedback is always welcome at emmiller@gmail.com