How to validate new product ideas (the data-driven, no-bullshit way)

3 common mistakes to avoid; 3 principles for finding great product ideas; 3 simple steps to validate whether your idea is worth pursuing. Actionable advice, real examples — without the bullshit.

Most blog posts will bore you to death with 5,000 words on *why* you should validate your startup ideas before building. I’m gonna skip all that and jump straight to the *how* part.

Over the past ten years, I’ve launched loads of different types of products, from software startups and consumer goods brands to international tech conferences and even a government lobbying campaign. What I’ve learned is that idea validation doesn’t change depending on the product category — it’s the same ingredients required every time.

This no-bullshit guide covers three topics:

1. The Most Common Mistakes

2. What Makes An Idea Great

3. Idea Validation In Three Steps

Let’s jump in…

You’ve Probably Already Made These Mistakes (Everybody Does)

Mistake 1: “So… what do you think?”

If your approach to date has been to tell people your startup idea and ask what they think, then you should trash everything and start over with a clean slate — 90% of people will find something nice to say in these situations, even if they think your idea is terrible.

Instead of pitching our startup ideas to anyone who’ll listen, we want to find target customers and get them to talk about their lives (the problems, goals and frustrations they experience). A good place to start is to follow the advice from the first few chapters of The Mom Test by Rob Fitzpatrick.

Mistake 2: Idea > Problem

Customers don’t search for ideas; they search for solutions to their highest-priority problems. They’ve got more problems every day than time, money, or energy. If you want them to use your product, you’ll need to be solving one of the problems at the very top of their priorities list.

Shreyas Doshi, former Product Leader at Stripe, says this is the big undiagnosed reason so many new products fail — the product teams validate that their idea solves a problem for customers, but they fail to learn whether it’s solving a high-priority problem or just a mild inconvenience.

Mistake 3: Target Customer = Everybody

Your idea will launch into an extremely noisy market where nobody knows who you are or what you do. The best way to stand out is to articulate your customer’s high-priority problem so clearly that they instantly know you truly understand the problem they’re trying to solve.

The more customer segments you target, the vaguer your description of that problem becomes as you try to make your messaging relevant to different types of people. It’s worth reading how WeatherBill burned $16M before realizing that hedging their bets on ten different customer segments was killing their potential (they exited for $930M exit 4 years later).

The Search For High-Priority Problems

You’re hopefully already realizing that we’re not really looking to “validate an idea” at all — instead, we’re trying to discover a high-priority problem we can solve.

But what kind of problems are high-priority?

The best problems tend to fall into one of these categories (or, even better, overlap two or three of them simultaneously):

1. Expensive: Problems that cost more time, money or energy than we can afford to give.

2. Frequent: Problems that can be fixed quickly or inexpensively can still become high-priority when they appear multiple times a day or week, burying us in repetitive work that drains our energy or attention.

3. Boring: Problems that are monotonous, bureaucratic, or time-consuming tend to get kicked down the road until a deadline forces action.

So how do we discover these kinds of problems in the first place?

—

Data-Driven Validation: The Discovery Sandwich

There are three steps in our search for our high-priority problem:

Step 1. Discover: Find all the different problems that target customers experience in our area of focus.

Step 2. Stack Rank: Get those customers to collectively rank these problems from highest to lowest priority.

Step 3. Deep Dive: Take the highest-ranking solvable problem and learn as much as possible about it.

I like to think of these three steps as the layers of sandwich…

Step 1. Problem Discovery

Our goal is to learn about all the problems that one group of people experience during a specific task, occasion or objective (ie. whatever topic you’re interested in).

You’re 100% banned from pitching them your idea — this is just an informal interview where you’ll ask open questions about your area of interest and allow them to introduce problems to the conversation organically.

Three quick tips on discovery interviews from The Mom Test:

1. Use follow-up questions to turn any vague or hypothetical answers into concrete recent examples full of rich detail.

2. If they suggest specific features or product ideas, keep asking ‘why’ until you understand what problems those suggestions were intended to solve.

3. Ignore compliments — if you’re getting them, it’s probably because you’re pitching your idea, which you’re not allowed to do anyway.

Step 2. Problem Stack Ranking

Once you’ve uncovered a bunch of problems, next you’ll need to separate high-priority problems from mild inconveniences.

To do this, we’ll want to mimic how people make decisions in real life — comparison. I’m a big advocate for Pair Ranking survey, where respondents see two problem statements at a time and have to pick the one that’s more relevant, important or urgent to them. Whichever problem statement wins the highest percentage of these pairs gets ranked as your highest-priority option.

I built a free research tool for Pair Ranking called OpinionX because no other tools offer that survey method. But if you’re not keen on me promoting my own product, here’s a list of alternative survey tools you can try.

There are just two things you need to create a stack ranking survey:

1. Ranking Question → include the context of the use case / topic you want respondents to consider when they’re voting on problem statements, for example: “Which problem is more frustrating for you when it comes to monthly goal setting with your team?”

2. Problem Statements → Translate all the problems mentioned during your interviews into a list of short statements (here’s my guide to writing great problem statements for research projects). This should include problems you could solve but also other problems too because your goal is to find the relative importance when comparing all the problems your customers are dealing with.

Share your Pair Ranking survey with a sample of target customers. Once the survey starts collecting votes, you’ll quickly see their top priorities rise to the top of the stack-ranked results.

Some optional things to consider including in your stack ranking setup:

1. Collect some basic demographic data and use a segmentation filter to calculate separate stack rankings for each subgroup within your target customer base (eg. managers vs. juniors).

2. Gather extra problem statements directly from your early batch of survey respondents and add them to your voting list mid-survey. Oftentimes your customers have insider vocabulary for describing their problems in ways you haven’t discovered yet.

3. Once you’ve found your high-priority problem, you can repeat this process a layer deeper into the problem to explore the hierarchy of specific nuances and variables — but before that, check out the next step of the sandwich…

Step 3. Problem Deep Diving

Now we’ve got two really valuable data points for our final step: (1) our stack ranked data helps us pick a high-priority problem to focus on, and (2) we can see which individual participants ranked that problem as high-priority for themselves personally.

We had to be very careful during Step 1 not to bias our interviews by introducing problem statements ourselves — now our stack ranked data allows us to jump straight to the big questions like:

• When was the last time this problem occurred for you? Can you tell me that story behind it?

• How did you solve the problem then? What solutions did you use to address it?

• What downstream impact or knock-on effects does this problem create?

• Why haven’t you been able to solve this problem before now?

• What are the costs of this problem for you in terms of time, energy and financial budget?

A large dose of curiosity and empathy in these interviews will surface the rich details about the problem that you’ll use to design every aspect of your solution — from the product itself to your marketing, sales and growth strategies.

—

Example of a stack ranking survey that follows the Discovery Sandwich method

I had spent 7 months and 150+ interviews trying to figure out what problem our startup was meant to solve. Within 2 hours of launching a stack ranking survey, I could see that the problem I was chasing was dead last to our target customers — 45th out of 45 problems (here’s the full story).

Today, OpinionX is a free research tool for stack ranking people’s priorities (used by thousands of companies like Google, Amazon and Shopify). Since that first experience of the power of problem stack ranking, I’ve seen thousands of stack ranking surveys built by so many different types of teams around the world. To help you get started, I’ve created a sample ‘Idea Validation’ stack ranking survey on OpinionX — check it out in our Sample Survey Gallery.

^ Ranking problem statements related to online meetings and then filtering to see what’s ranked highest by people who with a negative feeling towards online meetings.

—