How Stripe validates ideas for new products

On this day three years ago, a single Reddit post turned our startup from a guaranteed failure into a growing startup — here's what happened...

On this day three years ago, I posted something on Reddit that turned OpinionX from a guaranteed failure into a growing startup.

Overnight we went from zero revenue to paying customers. An investor even sent me the post and recommended I follow its advice, not realizing I was the one who wrote it.

You can still find the original post here on Reddit, where it remains in the top 10 most-upvoted posts of all time in the r/Startup community of 1.6 million members. I’ve included the full text below, edited to remove many embarrassing typos that I just noticed today for the first time :)

— — —

How Stripe validates ideas for new products

80% of products and features are rarely or never used. Why? Because they're solutions for problems that customers don't care enough about.

I spent over a year doing 100+ customer discovery interviews only to follow the wrong problem. I've also spent 5+ years as a Techstars Community Leader and Global Facilitator helping hundreds of early-stage entrepreneurs from around the world to validate their ideas and build first-concept products. If there is one thing I've learned, it's that validation is hard as f*ck.

At the end of March 2021, I came across this Twitter thread by product management genius Shreyas Doshi describing an idea validation method called Customer Problem Stack Ranking that he previously used at Stripe.

via Twitter

When we found this thread, we had a grand total of zero customers despite launching 6-months prior. So I decided to give Customer Problem Stack Ranking a try to see if it could help explain our lack of traction.

We asked 150 product managers to stack rank our value proposition (which we had spent months crafting) and it came dead last.

Dead. Last.

We learned more in 2 hours using this stack ranking approach than we did in 100+ target customer interviews across an entire year.

To explain how Customer Problem Stack Ranking works, I'll use an idea that I've heard pitched at countless hackathons — Splitzies, an app that makes it easier to split the cost of group holidays.

Btw I used OpinionX (a free tool I created) to carry out this stack ranking method using the pair ranking format but feel free to use whatever research platform you prefer.

— — —

What is Customer Problem Stack Ranking?

Customer Problem Stack Ranking (CPSR) tells you how important your idea is compared to the other problems your target customers experience. It's a simple data-driven approach to understanding whether your idea solves a burning pain point or just a mild inconvenience.

Step 1: Write Your Question

Customer Problem Stack Ranking is a type of survey so it needs a question, which usually goes along the lines of "What is the most frustrating aspect about ____ ?". Your question should be broad enough that it allows participants to explore all the problems associated with an activity rather than just the specific problem that you're trying to solve.

For our imaginary app Splitzies, our ‘activity of focus’ is booking a group holiday so our question should be → What is the most frustrating part about booking a group holiday?

Step 2: Turn Your Idea Into A Problem Statement

Asking target customers to rate your idea is a bad idea. As Rob Fitzpatrick explains in The Mom Test, if you ask people about your idea they'll just tell you it's great so they don't hurt your feelings. Instead, we need to turn our idea statement into a problem statement and compare it to the other problems that customers face in our 'activity of focus'.

The problem statement for Splitzies could be "Dividing the cost of a hotel with friends is frustrating and complicated when planning a group holiday." You can create multiple problem statements to explore the different pain points your idea might solve and the different words your target customers might use to talk about the 'activity of focus'.

If you're struggling to turn your idea into problem statements, here are some quick guides for brainstorming and writing problem statements.

Step 3: Peripheral Problem Statements

If we only consider the problems your idea solves, we won’t have an accurate picture of our target customers’ true priorities. Peripheral problem statements fall under the same 'activity of focus' but aren't solved by your idea.

A handful of interviews or a quick search on some forums will help you find some problems that fit in here. You can always ask each CPSR participant to submit new problems that you can add to your stack-rank list as you go to fill in any missed gaps.

Here are some peripheral problem statements for Splitzies:

• Keeping a list of potential Airbnbs and hotels turns into a giant messy spreadsheet.

• Agreeing on dates that suit everyone is a pain.

• It's difficult to plan activities when I haven't researched transport methods yet.

• Finding good info about the best time of year to visit each destination is a challenge.

• It's hard to know how expensive a destination is for general things like food and transport.

Step 4: Send Your Stack Rank To Target Customers

Before you start your stack ranking research, it’s important to consider who you’re going to ask to participate.

If you’re using a manual process like interviews, it’s best to focus on one specific segment at a time to make it easier to spot patterns. For example, if I send my Splitzies stack rank to both young parents planning a family holiday and student backpackers, they're going to have very different top-priority problems and my data will get all messed up.

If you’re using a stack ranking survey, it’s worth including some short multiple-choice questions to bucket participants into groups so you can filter and compare results of different customer segments afterwards.

If you haven't got a userbase or waitlist to send this to, just send people DMs on online communities, forums, and social media. We joined some Slack communities for Product Managers, sent out 600 direct messages in one evening, and ~25% of the people we messaged completed our survey. A well-written non-salesy message helps a lot here too!

Step 5: Iterate!

For our survey, we used a voting method called Pairwise Comparison. Each survey participant was shown 10 pairs of problem statements and asked to pick the one they cared about more. We ranked the problem statements based on which ones were picked most often in these pair votes.

Once votes started rolling in, we could see the top priorities emerge pretty quickly. We included an open-ended question after the pair voting so that participants could submit new problem statements for ranking, which we used to double our total list of problem statements during the first 1-2 hours of distributing the survey.

Step 6: Results

In one click, you can sort all the problems from highest to lowest importance and see how important your value proposition is compared to the other problems your target customers face.

Other than our key problem statement ranking dead last, what was really surprising was how many of the highest-ranked problems were things that could be solved using our product, but we just weren’t telling anyone to use it in these ways.

So, we took 5 of the top 7 problem statements and rewrote our entire landing page and onboarding experience to focus solely on these (more on how we rewrote our landing page here). Every metric skyrocketed — most notably, we’ve finally landed our first paying customers. We realized that the real mistake we made was inventing our own vocabulary instead of learning exactly how our customers described the problem themselves.

Step 7: Go.

Whether you’re trying to launch a new startup idea, validate the need for a new feature, or sharpen product strategy at your company, Customer Problem Stack Ranking is an easy way to measure what your customers actually care about.

Create your own stack ranking survey today using the same pair ranking format we used for our experiment in 2021 → OpinionX is a free tool for stack ranking surveys where you can create unlimited surveys with unlimited participants.

— — —

PS. I hope this serves as proof to many early-stage teams that sometimes it really is just one more push, product update, or blog post until things ‘click’ and you kickstart momentum for your startup. Since writing this post, we’ve onboarded +14,000 teams including paying customers like GitHub, Lionsgate, and The Economist. Keep pushing!

— — —