Ship Early, Ship Often

How Facebook Test and Improve

It’s interesting to see recent details of how Facebook are now testing changes to their experience and ads using the vast amount of data at their disposal. I’ve been taking a look and thinking what this means for smaller businesses without this data and resource.

Facebook recently disclosed in a briefing on how it uses Big Data that its system processes a mind-boggling 2.5 billion pieces of content and 500+ terabytes of data each day. It’s processing 2.7 billion Like actions and 300 million photos per day, and it scans roughly 105 terabytes of data each half hour.

I’ve been reading about how they manage this in the Facebook engineering blog in this post by Andrew (Boz) Bosworth who is the Director of Engineering at Facebook and is naturally from Harvard and FWIW looks a cool dude!

"Every day, we run hundreds of tests on Facebook, most of which are rolled out to a random sample of people to test their impact. For example, you may have seen a small test for saving news feed stories last week.

Other products might require network effects to be properly tested, so in those cases we launch to everyone in a specific market, like a whole country.

That’s why we’ve developed a sophisticated and flexible tool called Gatekeeper to make sure tests don’t collide with each another and that they provide statistically meaningful results. This allows us to roll things out slowly and make improvements as we go. Not every test we do ends up being integrated into the product, but even failed tests help us understand how to make Facebook easier to use, faster and more engaging".

It’s no surprise since Google and Amazon have used a similar approach for testing new search experience and ad features for years, but they haven’t often given this much detail.

Boz also gives an example of “Getting it Wrong” when they launched a new product without testing that it reduced chat interactions by 9%. Boz says:

"…ship early, ship often … means that when we’re able to iterate, people on Facebook get better experiences sooner than they would otherwise.

When you see such dramatic results from the smallest tweaks, you realize how much opportunity there is to improve things—and we feel a constant sense of urgency to do so.

When a test goes out we look at the data immediately and adapt the products quickly. We do this on a daily basis. This cycle of iteration is the engine of progress and the people who use Facebook are not just the beneficiaries but are also intimately a part of the process. We don’t just develop this product for them, we develop it with them".

You can read more in this personal interview with Boz how Facebook manage innovation.

Are you investing enough in analysis and testing?

So what does this mean for smaller companies without this resource (the rest of us)? Well, it prompts these questions:

• When did you last test a change to make your home page or key landing page more effective?
• Are you using analytics to find out what you should be testing?
• Are you using Google’s new Content Experiments (formerly Google Website Optimizer) or similar AB/MVT testing tools
• If not, why not? How can you reallocate budget/people from other activities to give them the time to test
• Which resources do you have to test - do you need to invest in a new set of skills internally or bring in a new specialist agency to help you?
• What is your process for testing? How do you define time to make structured testing happen?