Description:

The Wikipedia fundraising team was performing up to 100 AB tests per week. It wasn't enough to find the gains we needed. We needed to use statistics to interpret our AB tests accurately, but also to estimate smallest acceptable sample sizes to increase testing frequency. We were not comfortable trusting methods proposed by other practitioners or academics who could not prove that their methods would work accurately for our data.

In this webcast talk we'll present simple methods that we believe accurately predict future performance from AB test results, and that allow us to determine the smallest acceptable sample size. Using four years of AB testing data, we'll show that these methods really work. By taking a walk through four years of data, we'll show how our current methods would have predicted the actual outcomes of long tests with much smaller samples, and the implications of various decisions about statistical confidence, power, sample size and choice of statistical method.

This webcast session will be useful to anyone who performs AB tests, manages teams that rely on AB testing, or holds responsibility for implementing or maintaining AB testing platforms.

About Zack Exley

Zack Exley is a practice director at the global IT consultancy ThoughtWorks. Previously, Zack was the Chief Community Officer and Chief Revenue Officer at the Wikimedia Foundation (Wikipedia), where he built an aggressive AB testing program that accomplished a 350% percent increase in revenue while radically reducing the length of the annual fundraiser.

Zack was a pioneer in online fundraising and organizing. He helped bring U.S. politics online as MoveOn.org's first Organizing Director, advisor to the Howard Dean campaign, and Director of Online Communications and Organizing at John Kerry's presidential campaign. He is the Co-Founder of the New Organizing Institute, a technology and strategy training center for social justice organizations.

About Sahar Massachi

Sahar is a startup founder, online organizer, and software developer recently out of Brandeis with a BS and MA in Computer Science. After founding a start up to provide community organizations with better mobile tools, he joined up with the Wikipedia fundraising team to help complete a review of their AB testing methods, analyzing and preparing for publication hundreds of past AB tests.

You may also be interested in: