This post explores how to run great experiments in your company, based on recent conversations with my friend Astro Teller, Chief of Moonshots at “X” (formally Google X, Google’s R&D factory).
X’s mission is to invent and launch “moonshot” technologies that could make the world a radically better place…dare I say, help create a world of Abundance.
Astro leads a team of brilliant engineers, scientists and creatives developing solutions to dozens (perhaps hundreds) of the world’s toughest problems. Some of their publicly known projects include: the self-driving cars, the smart contact lens, high-altitude wind-power generation, and Project Loon, just to name a small fraction.
All of these projects started as a series of experiments.
Today’s most successful companies, the ones that are “crushing it,” started as a series of crazy ideas, followed by experiments to test just how viable those ideas might be.
Experimentation is a crucial mechanism for driving breakthroughs in any organization.
By the way, you may also want to check out Astro’s 2016 TED talk which was just released.
What Are Experiments and Why Do They Matter?
Experiments help you test a hypothesis about your product or service and help you find answers to your most difficult questions.
Good questions are questions that, if answered, fundamentally change (and improve) the way you operate.
Understanding how to focus your team on asking good questions and turning good questions into experiments is critical.
So what makes a good experiment?
The Three Principles of a Good Experiment
Astro explains that the following three principles describe a good experiment:
Principle 1: Any experiment where you already know the outcome is a BAD experiment.
Principle 2: Any experiment when the outcome will not change what you are doing is also a BAD experiment.
Principle 3: Everything else (especially where the input and output is quantifiable (i.e., measurable)) is a GOOD experiment.
Seems simple enough, right?
You need to be asking questions to which you don’t know the answer but such that if you did know the answer, you’d change the way you operate.
If you already know the answer, or if you are testing an insignificant detail that doesn’t matter, you’ll just be wasting time and money.
How to Ask Good Questions and Design Good Experiments
In any organization, you get what you incentivize.
In order to get good questions/experiments, you have to create a culture that incentivizes asking good questions and designing good experiments.
Astro describes a very unique approach to doing just this:
“At X, we set up a ‘Get Weirder Award.’ The whole point of the Get Weirder Award was to focus the team on experiments and to drive home why they needed to think in terms of experiments.”
Teams would be challenged to ask “weird” questions — to put forth crazy ideas around framing problems differently and to design experiments that really push the limits.
Critically, Astro gives out the Get Weirder Award before the experiments are run.
“If you give out the award before you’ve run the experiment, then people start to really feel that you don’t actually care about the outcome. You care about the quality of the question. So every two weeks, we would give out an award for the best experiment.”
Doing so constantly (and viscerally) reinforced the behavior of asking good questions — and as such, at X, they’ve built a culture around celebrating the questions themselves.
How to Manage Experiments
Once you’ve designed a good experiment and assembled an intellectually diverse team to tackle it, what are the best management principles to keep from screwing it up?
Management Principle 1: Don’t Be a Bottleneck!
As a CEO or manager, it is critically important that you don’t get in your team’s way by micromanaging them or by demanding to be the sole decision maker.
If you need all of the information, all of the time, your team will never get their work done.
Astro explains, “Your job as a manager is to give your team your recommendations and empower them to do whatever they think is right. Allow them to learn.”
He continues, “I work super hard for me not to be the bottleneck at anything that goes on here. Ironically, that’s a full-time job.”
One funny story involved two employees who had a major strategic conflict. They wanted Astro to make a decision as to which one was right.
Rather than do so, he said, “I believe I already know which of you is right and which of you is wrong. I can just make that decision right now, but I’m not going to. The problem is: If I tell the two of you who’s right and who’s wrong, in my opinion, the next time you have conflict, you will come back and ask me to do it again, and that does not scale. I will spend however as much time it takes to either train the two of you to work well together or figure out that you can’t.”
Not being a bottleneck means deliberately letting your team learn. Sometimes it’s hard to do, but it’s a necessary step if you want to derive the most value from your experiments.
Management Principle 2: The Value of Secrecy
In a previous blog, I mentioned the notion that ambitious entrepreneurs (and companies) need to get comfortable with being misunderstood.
Interestingly, experiments are one of the most publicly misunderstood domains within a company.
We talked in great depth in the last blog about why you have to focus on killing your ideas — in line with that discussion, an experiment that proves an idea won’t work is as successful of an experiment as proving that one will work!
The problem is: the public (and especially the press) doesn’t understand this.
They see “failed experiments” and think, “failed company” — which adds enormous pressure and stress to your employees and investors.
Thus, it can be useful to keep your experiments secret.
Astro explains, “The main value of secrecy is not ‘to hide our awesome ideas’, it’s to make it easier to kill ideas. Secrecy offers air cover. It allows the team to have the emotional space to solve the problems without outside pressure.”
Management Principle 3: Don’t Worry about Success — Worry about Progress & Learning
Finally, if you are worrying about success, you are going to be paying attention to all of the wrong indicators and misguiding your team.
You need to focus on progress and learning — and success will follow.
Rather than creating a culture that only celebrates big wins, create one that celebrates progress on tough projects and running good experiments.
You’ll be amazed by the difference in your organization with such a small mindset and cultural shift.
Image credit: Shutterstock.com
