You have a feature to implement, and there are several implementation solutions available. How do you choose the best one?
Start out with all your potential solutions for a feature idea. Next, filter based on how the solutions perform using a set of verification methods. Finally, implement the feature knowing that you’ve found the solution that meets your needs.
Verification Methods
The following are the verification methods I’ve experienced most often on the projects:
- Data – Analyze the possible solutions using existing data. Maybe you’ve done a similar feature before, or you can answer some of the open questions by looking at data you already have.
- Mock – Mock up the possible solutions, walk around the office, or the street and ask people for input. What kind of feedback do you get? How do they react to the mocks? What are the pros and cons? Are they using the feature the way you envisioned they would?
- Prototype – Test your feature solutions in the product. Use branches or feature flags, and this time deploy them on a test site/device, and try them out them on volunteer testers. Again, listen to what the testers tell you about what works and doesn’t work.
- A/B test – Build all the solutions into the product, and split them among your user base. These are real users, using your product in the right context, make sure you’re tracking the KPI’s that indicate if the feature is doing well or not, and base your decision on the data gathered.
- Gut feel – This method might feel counter-intuitive. However, there are times when you don’t get conclusive data from any other method, and you need to rely on your experience to pick a solution. Make sure to follow up on how the solution is performing.
Setting up a verification funnel
You can use any combination of these methods in your feature verification process. You can choose to use the same method several times, you can also exclude a method. Each feature will have it’s own combination of methods.
I try to apply the verification methods in the order listed above so that the most expensive method (implementing in the application) and the most subjective (gut feel) are not used too early while we still have a lot of possible solutions.
Example feature verification funnel
- Mock – there is no data, so this is where we start, we make changes based on these first tests
- Prototype – we take three ideas from the mocks that we prototype
- A/B test – finally we merge our ideas into two solutions and we A/B test
- Data – in the meantime, some other part of the organization has completed a test that’s applicable to our situation. Our first A/B test shows one strong candidate, and now we have another potential solution
- Mock – We mock the new potential solution
- A/B test – We A/B test both solutions, they’re both performing well, but one of them is more consistently strong, we focus on it.
- A/B test – We A/B test variations of the solution that’s performing best.
- A/B test – We re-test to make sure the results are not based on timing, and we pick the best implementation
Outcome
The verification funnel is built to help you get the best possible solution to the problem you’re trying to solve. There are no milestones or gates. This is where ideas start their journey to becoming the next implementation.
Get in touch via my homepage if you have questions or comments!
