Growth @ Facebook
Let’s start with a look at the demographics of this set of people.
# if only this would work select * from people;
The dimensions to look at distributions include
Don’t forget to see how how your population divides among active users. Obviously, you need to define what active means in your context.
We made a big assumption here to simplify matters, that people are equally likely to return. In reality having cohorts who differ in their likelihood of returning complicates your test analysis.
We have three different metrics in this dataset. Let’s examine what they are, and their properties.
We will look at the following for people not currently in tests
Think of this as a ‘status update’ event. We care about it a bit.
Think of this as a ‘athletic activity’ event. About as important to us as A.
Think of this as a ‘uploaded a photo’ event. We care about it a lot.
You really should understand these next few lines.
SQL is an indispensable tool in industry.
select c.metric from conversions as c join exposures as e on c.uid = e.uid where e.test1 = 0 and e.test2 = 0
We are running two tests on this unfortunate group of guinea pigs. We are going to examine the results for each, and their impact on the metrics that we care about.
Let’s say this is a big promotion for the Amgen Tour of California on the homepage.
Let’s make the status update button bigger. We want more people to be sharing their thoughts.
We need to know how confident we should be of these results.
What happens when you look at people in both tests? It all goes to pieces.