Metrics are the foundation for effective assumption testing

The core theme behind The Salt Test is defining your riskiest assumptions and then testing them to see if they pass or fail. It is therefore imperative to have a set of numbers that define what constitutes a pass or a fail. 

These numbers will vary, depending on the test, and it is vital that they are defined beforehand. Doing this will ensure that you remove the emotional response to your experiments. This is particularly true when the test doesn’t turn out as you hoped.  

For example, imagine my goal is to sell my product to 10 people at $800 each. What happens if I only manage to sell it to eight people, or six, or maybe only four? You could argue that eight people is close to 10, so that is probably a pass. You may find reasons to justify why you only got six, or even four.  

This is why it is imperative to state upfront the number you will use to define success, and even more importantly, the number that you define as a fail. It is crucial that the fail number is immovable, to ensure you aren’t swayed by an emotional response.  

I have also discovered that coming up with a fail number forces me to consider what could go wrong in a test, which allows me to address any concerns beforehand. For example, I recently defined a test with the goal to run a service from start to finish within 24 hours. My fail metric was 72 hours. When considering that number, I thought 72 hours was very comfortable. However, with more insight, I realised that I relied on third parties for this experiment, and if they were not available, I could easily miss that goal. Therefore, I redefined the test to ensure those third parties were available and ready, which is what would happen in a realistic scenario anyway. My new test was consequently a better representation of what would happen in the real world.  

So, if you pass the experiment, you sign it off and move on to the next risky assumption. If you fail the test, you stop, as tempting as it might be to make changes and to keep testing (failure to do this will lead to long and expensive experiments). However, what happens if you get a number in between the two? This is the opportunity to keep testing. Take the time to assess why you didn’t reach your goal and see if there is anything you can do to improve your chances of success. If you do find something, make those changes and test again. Ideally, every iteration will get you closer to your goal, until you achieve it.  

With this in mind, my preference is to use these three metrics, ‘Pass’, ‘Fail’ and ‘Figure it out’.