Why These Tech Companies Keep Running Thousands of Failed Experiments

Technology and advanced data analysis capabilities have made it faster and cheaper to test new ideas. JL

Ben Clarke reports in Fast Company:

Innovative businesses experiment quickly and cheaply, knowing full well that most tests will be duds. Innovative run thousands—Intuit: 1,300, P&G: 7,000–10,000, Google: 7,000, Amazon: 1,976, and Netflix: 1,000—thanks to a combination of new technologies and "lean" business approaches. Instead of making ideas trickle up through a long process of approvals, meetings, egos, and politics, junior level decision-makers can perform low-risk, low-cost experiments
Twenty-odd years ago, most big companies would run just a handful of experiments each year. Today, the most innovative businesses run thousands—Intuit: 1,300, P&G: 7,000–10,000, Google: 7,000, Amazon: 1,976, and Netflix: 1,000—thanks to a combination of new technologies and "lean" business approaches. And it isn't just quantity that's rising but the quality and pace of experimentation, too. These days, the true test of how innovative a company can be is how well it experiments.

The Costs Of Innovating

This is hardly a secret. Amazon chief Jeff Bezos has said, "Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day."

Most experiments fail, no matter how well-designed.

What makes that possible? For one thing, Bezos explained in a 2011 book interview, "We’ve tried to reduce the cost of doing experiments so that we can do more of them," he tells The Innovator's DNA coauthors Jeff Dyer, Hal Gregersen, and Clayton M. Christensen. "If you can increase the number of experiments you try from a hundred to a thousand, you dramatically increase the number of innovations you produce."
Second, you only need a few big wins to make all those experiments worth it. After all, most experiments fail, no matter how well-designed. One paper that reviewed experiments' success rates found that less than 50% of those conducted at Amazon, Microsoft, and other software companies actually improved the metrics they were designed to improve. Those failures cost Amazon billions of dollars, but Bezos gladly accepts that as the cost of innovating. In a recent SEC filing, he explains why:

Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in awhile, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold. Big winners pay for so many experiments.

You don’t have to look further than Amazon Web Services (AWS) to know Bezos isn't exaggerating. In just 10 years, AWS, which started off as an experiment unrelated to Amazon's e-commerce business, has become the fastest growing B2B company in history. By the close of 2016 alone, it's expected to bring have raked in over $10 billion in revenue. And that's just one of other multibillion-dollar experiments at the company—Prime, Echo, Kindle, and Amazon's third-party sellers—that have all paid off in the long run.
Researcher and author Nassim Taleb sums it up in this graph:

[Image: Nassim Taleb]

How The Top Tech Companies Experiment

Amazon, of course, is no outlier when it comes to experimenting. Facebook CEO Mark Zuckerberg says the company conducts tens of thousands of experiments at any given time. As far back as 2008, when it was a much smaller company, Google was already leveraging its search engine users to run 50–200 experiments at once.

"It’s important to be bold. Big winners pay for so many experiments."

Netflix created a whole division of 300 people to help users discover content. In a 2014 interview, chief product officer Neil Hunt claimed that this $150-million annual investment was yielding $500 million in value for Netflix. And in a recent Netflix blog post (winkingly titled, "It’s All A/Bout Testing: The Netflix Experimentation Platform"), the company even offered an inside, technical look at an experimentation platform it built and explained what the company
is looking to do with it next.
According to author Michael Shrage, P&G now has the capacity to churn out ideas for new product features at 10,000 times the rate it could a decade ago. For example, on the marketing front, the company uses virtual store environments to test consumer reactions, then optimizes how to best package and place products on store shelves at a fraction of the cost.

[Photo: via McKinsey & Company]

Intuit performs hundreds of experiments on each of its products every year. Not unlike Netflix, it's also built technology expressly to improve how well and how quickly the company experiments.

[Experimenting] resolves the tension between creative teams . . . and senior executives who want guaranteed results fast.

Finally, Capital One runs 80,000 marketing experiments per year in order to target its customers with personalized credit card offers. Its data-focused marketing has helped lower the cost of acquiring customers by 83% over three years while increasing market share.

Experimenting Changes The Way Decisions Are Made

Another reason why these and other innovative companies are experimenting so much is simple: It resolves the tension between creative teams—who want to try big, new, unproven ideas—and senior executives who want guaranteed results fast.
Using real-world data, experiments make it possible to break decisions down into their core
assumptions and test those in bite-size chunks, lowering the risk baked into the decision-making process. If you want to start running more experiments at your own company, here are a few steps to take and criteria to aim for:

1. Work toward building a platform or approach that makes it possible to run not just twice as many experiments as you already do but 10 to 100 times more.
2. Empower your most junior employees to conceive and perform their own experiments.
3. Come up with a standard methodology for validating results.
4. Share the results and lessons learned from experiments across the company, then use those to inform new ones; even failed experiments should help you ask better questions.
5. Select experiments to go forward with that have the highest potential ROI (in other words, stop testing the color of the package).
6. Set aside a percentage of your marketing and product-development budget strictly for experimentation. Most companies spend 0%–5% of their time and money on this, when it should be closer to 10%.

Suddenly, instead of making ideas trickle up through a long process of approvals, meetings, egos, and politics, junior level decision-makers can perform low-risk, low-cost experiments—and let the results speak for themselves.