Today’s most successful companies, the ones that are “crushing it,” started as a series of crazy ideas, followed by experiments to test just how viable those ideas might be.
Experimentation is a crucial mechanism for driving breakthroughs in any organization.
If you want to create a successful, hyper-growth company, you’ve got to focus on empowering your teams to rapidly experiment.
Over the years I have had the pleasure of sitting down with wizards of experimentation, including Jeff Holden, Uber’s Chief Product Officer; Astro Teller, CEO of X; and Jake Knapp, Design Partner at Google Ventures.
Through my conversations, I have compiled a suite of best practices for running great experiments and building a culture of experimentation at your company.
In this post we will discuss:
Building a Culture of Experimentation
- Running effective experiments
- Google Ventures design sprints
- Building a culture of experimentation
The only constant is change, and the rate of change is increasing.
Ultimately, standing still equals death, and the only way to succeed is to be constantly experimenting and innovating (think of it as Darwinian evolution at hyperspeed).
Hyper-growth and experimentation are very closely linked.
Jeff Bezos likes to say, “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…”
Jeff Holden, who has built experimental engines at Amazon, Groupon, and Uber, agrees: “The philosophy is you have to build your company to be a big experimental engine and it has to start right at the beginning.”
It’s not easy to just “retrofit” your company with that engine later—it’s a cultural shift. You have to be in the mindset of constantly testing crazy ideas, new business models, new products and new processes.
At Amazon, in the early days, they created a standard experimental platform that was available to almost everyone—meaning, if somebody wanted to test a new button or new feature on the website, they could.
The problem was that many of these experiments were useless.
Jeff Holden continues, “They had no chance of yielding any value. There wasn’t any point to them. We were just kind of curious. We were just running a lot of experiments—which have a cost, by the way—and were taking up experimental slots [so others couldn’t experiment], and things started colliding with each other.”
Their solution was to create an ‘Experiments Group’—if you wanted to do an experiment, you had to run it through this group.
The first question the group would ask was: What’s your hypothesis?
The second question: What’s the value proposition to our company?
“If you couldn’t articulate your hypothesis crisply, or your hypothesis didn’t matter for Amazon or Uber or Groupon, then they must not do that experiment. Oftentimes you’ll send folks back to the drawing board or ask them to recast the experiment. The company learned, and we got much better.”
Finally, “You have to be able to interpret the experimental results really well. It’s statistics. Know the difference between statistically significant and insignificant results.”
Uber, for example, runs thousands of experiments per month to test different features. They A/B test key features that are core to the business and choose the one that performs best.
“Build a team inside your organization that has an experimental ethos, and make sure that the experiment, value proposition, and hypothesis are really thought through before you invest the time and energy to actually do them.”
In general, only hire people who are familiar with the experimentation/data-driven mindset and set the stage for experimentation in the beginning.
How to Launch Good Experiments
Astro Teller, Chief of Moonshots, explains that the following three principles describe a good experiment:
Principle 1: Any experiment where you already know the outcome is a BAD experiment.
Principle 2: Any experiment when the outcome will not change what you are doing is also a BAD experiment.
Principle 3: Everything else (especially where the input and output are quantifiable) is a GOOD experiment.
Seems simple enough, right?
You must ask the kind of questions to which you don’t currently know the answer, but if you did, you’d change the way you operate.
If you already know the answer, or if you are testing an insignificant detail that doesn’t matter, you’ll just be wasting time and money.
To get good questions/experiments, you must create a culture that incentivizes asking good questions and designing good experiments.
Astro describes a very unique approach to doing just this:
“At X, we set up a ‘Get Weirder Award.’ The whole point of the Get Weirder Award was to focus the team on experiments and to drive home they needed to think in terms of experiments.”
Teams would be challenged to ask “weird” questions—to put forth crazy ideas around framing problems differently and to design experiments that really push the limits.
Critically, Astro only gives out the Get Weirder Award after the experiments are run.
“If you give out the award after they’ve run the experiment, independent of the results, then people start to really feel that you don’t actually care about the outcome. You care about the quality of the question. So every two weeks, we would give out an award for the best experiment.”
Doing so constantly (and viscerally) reinforced the behavior of asking good questions—accordingly, at X, they’ve built a culture around celebrating the questions themselves.
Google Ventures: Design Sprint
A sprint, invented by my friends Jake Knapp and John Zeratsky of Google Ventures, is a fantastic tool for rapid experimentation in your company.
I have leveraged the sprint process across all of my companies.
Participating in a sprint orients the entire team and aims their efforts at hitting clearly defined goals.
Sprints are useful starting points when kicking off a new feature, workflow, product, business or solving problems with an existing product.
Here are the five phases of a sprint, typically done sequentially over the course of five days, that you can try with your team:
Day 1: Understand: Develop a common understanding of the working context, including the problem, the business, the customer, the value proposition and how success will be determined. By the end of this phase, you should also aim to identify some of your biggest risks and start to make plans to mitigate them. Common understanding will empower everyone’s decision-making and contributions to the project. Understanding your risks enables you to stay risk-averse and avoid investing time and money on things that rely on unknowns or assumptions.
Day 2: Diverge: Generate insights and potential solutions to your customer’s problems. Explore as many ways of solving the problems as possible, regardless of how realistic, feasible, or viable they may or may not be. The opportunity this phase generates enables you to evaluate and rationally eliminate options and identify potentially viable solutions to move forward with. This phase is also crucial to innovation and marketplace differentiation.
Day 3: Converge: Take all of the possibilities exposed during phases 1 and 2, eliminate the wild and currently unfeasible ideas and hone in on the ideas you feel best about. These ideas will guide the implementation of a prototype in phase 4 that will be tested with existing or potential customers. Not every idea is actionable or feasible, and only some will fit the situation and problem context. Exploring many alternative solutions helps provide confidence that you are heading in the right direction.
Day 4: Prototype: Build a prototype that can be tested with existing or potential customers. Design the prototype to learn about specific unknowns and assumptions. Determine its medium by time constraints and learning goals. Paper, Keynote, and simple HTML/CSS are all good prototyping tools for software products and 3D printing for hardware. The prototype storyboard and the first three phases of the sprint should make prototype-building fairly straightforward. There shouldn’t be much uncertainty around what must be done. A prototype is a very low-cost way of gaining valuable insights about what the product needs to be. Once you know what works and what doesn’t, you can confidently invest time and money on more permanent implementation.
Day 5: Test and Learn: Test the prototype with existing or potential customers. It is important to test with existing or potential customers because they are the ones for whom you want your product to work and be valuable. Their experiences with the problem and knowledge of the context have influence on their interaction with your product that non-customers won’t have. Your customers will show you the product they need. Testing your ideas helps you learn more about things you previously knew little about and gives you a much clearer understanding of which directions you should move towards next. It can also help you course-correct and avoid building the wrong product.
Sprints offer a path to solve big problems, test new ideas, and accelerate the decision-making process. By the way, you can learn a lot more about the sprint process here.
Happy experimenting!
Join Me
Abundance Digital Online Community: I’ve created a digital/online community of bold, abundance-minded entrepreneurs called Abundance Digital.
Abundance Digital is my ‘onramp’ for exponential entrepreneurs — those who want to get involved and play at a higher level. Click here to learn more.
Image Credit: Shutterstock
