Good Product Experiments & Bad Product Experiments

1. Good experiments improve product strategy | Bad experiments improve metrics

2. Good experiments are built around a product and help a team to better understand users’ behavior | Bad experiments are delivered in a spontaneous brainstorm that is not associated with a growth strategy

3. Good experiments influence the result by solving users’ problems | Bad experiments change metrics, but don’t solve users’ problems

4. Good experiments are like bets: the higher risk, the greater growth | Bad experiments are endless optimizations and attempts to improve minor details that as a result don’t contribute to product growth

5. Good experiments answer the question: “What should we do with what we’ve learned?” They help to understand why something works and something does not; to confirm or disprove hypotheses; to continue an experiment or switch to another | Bad experiments boil down to the question, “Keep it or stop it?” and are soon forgotten

6. Good experiments are an “obedience instrument”. You test what you believe in but are not sure of | Bad experiments are used to delay decision making, resolve differences, or let data tell you what to do next

7. Good experiments use success metrics from the start | Bad experiments are launched without understanding what metrics to track. It turns out, teams look at those metrics that are improving to justify an idea, rather than test a hypothesis

8. Good experiments are considered as a system and involve compromise | Bad experiments encourage personal wins and OKRs, even if they are detriment of other team members

9. Good experiments are well documented at launch | Bad experiments restart due to lack of data

10. Good experiments are on the table. They are recorded and documented for the future | Bad experiments are launched, but never complete the loop

11. Good experiments are integrated into the main product development process. They focus on the long-term success and are culturally sensitive | Bad experiments generate endless controversies and contribute to mistrust and lack of cooperation

12. Good experiments have a plan for success. Successful experiments become part of the product and are integrated into the launch process. | Bad experiments ignore the MVP.