How to Choose a Product Experiment: 55 Methods by Risk and Evidence
Most ideas lose when tested. Name the riskiest belief, decide how strong the evidence must be, then pick the cheapest of 55 product experiments that gets you there.
Share
Product experiments are small, planned tests of one belief your plan depends on, with a pass signal written before you start. To choose one, name the belief, name its risk (desirability, usability, feasibility or viability), decide how strong the evidence must be, then pick the cheapest method that reaches that strength. This guide sorts 55 methods that way, with time and effort for each.
Key takeaways
Most ideas lose when tested. Only one third of the ideas tested at Microsoft improved the metric they were designed to improve, and about 10% of Google's controlled experiments led to business changes (Kohavi and Longbotham, 2023).
Test the riskiest belief, not the whole idea. Marty Cagan's four risks are value, usability, feasibility and business viability (SVPG, 2017).
Evidence comes in strengths. What people say is weakest; money or a signature is strongest. Strategyzer's library for Testing Business Ideas rates 44 experiments from 1 to 5 on cost, setup time, run time and evidence strength.
Cheap does not mean weak. In that library, a concierge test scores 1 of 5 for cost and 5 of 5 for evidence.
Write the pass signal before you start. Moving it afterwards is how most teams fool themselves.
Why do product ideas need experiments at all?
Most product ideas do not fail because the team built badly. They fail on a guess nobody checked.
The record from teams that measure is humbling. Ron Kohavi and Roger Longbotham collect it in one place: a third of ideas tested at Microsoft moved their target metric, Jim Manzi wrote that about 10% of Google's experiments led to business changes, and Netflix is reported to consider 90% of what it tries to be wrong (Kohavi and Longbotham, 2023). These companies employ excellent product people. Their first guesses still lose most of the time.
Skipping the check costs more later. CB Insights studied 431 venture-backed companies that shut down since 2023 and found reasons for 385 of them: 70% ran out of capital, 43% had poor product-market fit, 29% cited bad timing and 19% had unsustainable unit economics (CB Insights, 2026). Companies could give more than one reason. Poor fit and broken unit economics are exactly the doubts a cheap experiment can probe months before the money runs out.
A product experiment is how you check a guess before you spend real money on it. You do not need a research team, and you do not need all 55 methods in this guide. You need the one test that could change your mind about the belief your plan depends on most.
What is a product experiment?
A product experiment is a small, planned test of one belief, with a result you define before you start. Good product experiments have four parts:
The belief, written as a sentence that could be false. "Freelancers lose work because follow-ups slip," not "freelancers need this."
The method: the smallest action that produces evidence about that belief.
The measure: one number, plus one thing that must not get worse.
The pass signal: the result that would make you say yes, set in advance.
Teresa Torres puts the core move plainly: stop testing whole ideas and test the assumptions they depend on (Product Talk, 2023). An idea like "a reminder app for vets" holds a dozen beliefs. Most are safe. One or two would kill it, and those are the ones worth an experiment. If you are starting from a feature request, the product problem statement template turns it into a belief you can test.
What are the four product risks an experiment can test?
Every belief carries a kind of risk, and the kind decides which methods fit. Marty Cagan of Silicon Valley Product Group names four (SVPG, 2017):
Risk
Cagan's question
Strategyzer's word
Usual owner
Cheap first test
Strong test
Value
Will customers buy it, or choose to use it?
Desirability
Product manager
Customer interview, fake door
Concierge, letter of intent
Usability
Can users figure out how to use it?
Tagged as desirability
Product designer
Paper prototype
Usability test on a real build
Feasibility
Can engineers build it with the time, skills and technology we have?
Feasibility
Lead engineer
Expert interview, technical spike
Wizard of Oz, mash-up
Business viability
Does it work for the rest of the business?
Viability
Product manager
Pricing survey, market research
Funnel test, letter of intent
Bland and Osterwalder's Testing Business Ideas (Wiley, November 2019, 368 pages) groups risk into three questions: do customers want it, can we build it, and will it earn sustainable revenue (Wiley; Strategyzer, 2025). Its experiment library tags usability-style tests, such as card sorting and clickable prototypes, as desirability. Cagan keeps usability separate, and so do we, because "they want it" and "they can use it" fail in different ways and need different tests.
Torres adds a fifth category, ethical: should we build this at all? Cagan's advice on order is blunt: tackle the big risks early, especially value and business viability. For a deeper look at what counts as proof for each risk, read what counts as evidence for the four product risks.
How strong is the evidence from each kind of experiment?
The higher the step, the more you can trust the result.
Not all evidence is equal. A kind word and a signed letter are both "positive feedback," but only one should move your roadmap. Alex Osterwalder makes the same point: what customers say and what they do are two different things, so interviews and surveys give valuable but weak evidence, while a simulated sale gives strong evidence (Strategyzer, 2017).
Strength
What it means
Example
Treat it as
What people say
Someone told you something
An interview, a survey
A reason to run a stronger test
Early signal
Real behaviour, but cheap or indirect
An ad click, a search trend
A clue about where to look
What people do
Someone took a real action
Finishing a task, leaving an email
Good evidence for that specific action
Money or a signature
Someone gave up something of value
A payment, a signed letter
The strongest evidence you can get before launch
Specky uses the same ladder. A belief can only be marked Supported by what people do, or by money or a signature. Anything weaker stays an Early signal, however good it sounded.
Does a stronger test always cost more?
No. Strategyzer's experiment library poster rates every method from 1 to 5, and cost and evidence strength move independently (Strategyzer experiment library). A sample:
The lesson: the expensive part of a strong test is usually your time, not your budget. A concierge test costs little money but weeks of manual work.
How do you choose the right product experiment?
Answer five questions, in order:
What is the belief? Write it as one sentence that could turn out to be false.
What kind of risk is it? Do they want it (desirability)? Can they use it (usability)? Can we do it (feasibility)? Will it make money (viability)?
What would it cost to be wrong? A guess that decides the next six months deserves stronger evidence than one that decides a button colour. Barry O'Reilly makes the same point about evidence thresholds: a rocket launch needs a higher bar than a sign-up flow (Thoughtworks, 2014).
What is the cheapest test that gives enough evidence? Start low on the ladder when you know little. Climb as the stakes rise.
What would count as a pass? Write it down before you begin. O'Reilly's reason is the right one: signals stated in advance reduce biased readings of the result.
A rule of thumb we use: if being wrong costs a week, what people say is enough. If it costs a quarter, get what people do. If it costs a hire, a year or your runway, get money or a signature.
When you have several beliefs and cannot tell which is riskiest, map them first. Our guide to assumption mapping and the free assumption map tool sort beliefs by impact and evidence, so the test goes where it matters.
Product experiment decision table: which test for which risk and evidence level?
Find the row for your risk. Move right until you reach the evidence strength that being wrong deserves. Pick the cheapest method in that cell.
Times and effort come from the method tables below and assume a small team. Read across a row to see how to climb. Read down a column to compare risks at the same strength.
Which product experiments answer which question? The full library of 55
Every method below links to its own guide, with a copyable template, a pass signal set in advance and a worked example.
Desirability: is the problem real, and who has it?
Start here when you are still unsure the pain exists. Most of these cost a few days and no code.
What does climbing the evidence ladder look like in practice?
This sequence is made up to show the idea. Imagine you think independent gym owners lose members because they do not notice who has stopped coming.
Five customer interviews confirm owners have tried to fix this. That is what people say, and it costs a week.
A fake door shows owners will leave an email for an offer. Say 40 of 400 visitors from a gym-owners' newsletter do, which is 10%. Now you have behaviour.
A concierge test sends weekly lists by hand to five gyms, and three pay for a second month. Now you have commitment.
Using the times in the tables, steps 1 to 3 take about five weeks: one for interviews, one for the fake door, three for the concierge. Each step cost more than the last, and each one was earned by the one before it. If any had failed, you would have stopped early and saved everything after it.
Notice what the sequence skipped. Nobody ran a survey to ask whether gym owners "would use" a churn tool, and nobody built a dashboard to see if it got traffic. Each test answered the next riskiest question and nothing more. For where this sits in a full discovery cycle, see the product discovery process guide.
How many product experiments and participants do you need?
There is no fixed number of experiments. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. Strategyzer's summary of the book suggests a practical cadence: run three to five small experiments on your highest-priority assumptions in the first month, and start with the smallest test that could confirm or kill each one (Strategyzer, 2025).
Pair weak and strong. One "say" test tells you what to look for; one "do" test tells you whether it holds. Two "say" tests that agree are still two opinions.
Participant numbers depend on the method:
Usability tests: five people per round. Jakob Nielsen's model found five users uncover about 85% of usability problems and 15 find nearly all, so he advises spending the budget on several small rounds (Nielsen Norman Group, 2000).
Surveys: aim for about 50 valid answers from one named segment before you trust a percentage. Our discovery survey guide explains why.
Split tests: these need real traffic. Kohavi and Longbotham note that a 50/50 split gives the most statistical power once a short ramp-up has ruled out obvious errors. If you lack the visitors, use a fake door or landing page instead. Our guide to A/B testing covers the maths.
Letters of intent and presales: a handful can be enough when each one is a real buyer with a budget.
What are the most common mistakes with product experiments?
Testing the idea instead of the riskiest belief. "Will this work?" cannot fail cleanly. "Will owners pay €50 a month for a weekly list?" can.
Picking the method you already know. Teams that like surveys survey everything. Let the risk choose the method.
Treating what people say as what they do. Twenty enthusiastic interviews are still the bottom rung.
Writing the pass signal afterwards. A result you judge after seeing it is a story, not a test.
Split-testing without traffic. A few hundred visits rarely prove that a change caused a result.
Skipping the unglamorous risks. Founders love desirability tests. Cagan's advice is to tackle value and business viability early, and feasibility can sink a loved idea just as fast.
Climbing too fast. A crowdfunding campaign before a single interview spends weeks to learn what a conversation could have told you.
When should you not run a product experiment?
Skip the experiment when the decision is cheap and easy to reverse; shipping and watching is the test. Our piece on reversible and irreversible product decisions helps you tell the two apart. Skip it, too, when you will build regardless of the result. A test you would ignore is theatre. And when none of the 55 methods fits your setting, such as a year-long public-sector sale, design a custom experiment rather than forcing one.
Is it ethical to test something that does not exist yet?
Several methods, including the fake door, the 404 test, presales and the Wizard of Oz, show people something that is not fully real. Keep them honest:
Tell people on the next screen that the feature is not ready, and thank them.
Never take money you cannot refund or deliver on. Say when delivery would happen.
If a person secretly does the work in a Wizard of Oz test, tell participants afterwards when it would matter to them.
Ask before recording anyone, and only keep the email addresses people chose to give you, for the purpose you stated.
How to run this in Specky
Fill in a Lean Canvas on the Assumptions page, and Specky drafts the assumptions behind it. Each one appears on a board by impact against evidence, with the riskiest numbered. Click a note, pick a test, and the plan fills in with steps, a pass signal, and a time to run.
Who does the work depends on the method. Some tests are AI interviews in Vouch, at vouch.specky.space. For some, Specky builds a page and collects responses. For some, it reads your connected analytics. For the rest, you run the test and Specky keeps the plan and the result. Each guide above says which.
Recording a result moves the note and updates any bet resting on it. One contradicted assumption marks the whole bet as contradicted. A bet only counts as supported when every assumption under it is.
Frequently asked questions
What is a product experiment?
A product experiment is a small, planned test of one belief about a product, with a pass signal defined before you start. It lets a team learn whether something is true before spending real money on it. A good one names the belief, the method, the number it will measure and the result that would count as a yes.
How do I choose the right product experiment?
Write the belief you are least sure about, name its risk (desirability, usability, feasibility or viability), and decide how strong the evidence must be given what being wrong would cost. Then pick the cheapest method that reaches that strength. Early on, conversations are usually enough. As stakes rise, ask people to act or pay.
What are the four types of product risk?
Marty Cagan's four risks are value (will they buy or use it?), usability (can they figure it out?), feasibility (can we build it with our time, skills and technology?) and business viability (does it work for the rest of the business?). Strategyzer's Testing Business Ideas uses three, desirability, feasibility and viability, and its library tags usability tests as desirability.
What is the strongest evidence in product validation?
Money or a signature is the strongest evidence, such as a payment, a pledge or a signed letter of intent. It beats what people do, which beats what people say. Osterwalder's rule of thumb is that evidence gets stronger the closer it comes to a real purchase, which is why a simulated sale outranks any interview or survey.
Should I start with customer interviews or a landing page?
Start with interviews when you are still unsure the problem is real, because you need people's own words before you can write an offer. Start with a landing page when the problem is clear and you can state your offer in one sentence. Many teams run both inside two weeks: interviews first, then a page built from what they heard.
How many experiments should I run before building?
There is no fixed number. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. As a cadence, Strategyzer's book summary suggests three to five small experiments on your highest-priority assumptions in the first month. Stop early when one fails clearly.