Product Experiment Template: Design Your Own Test When None Fits
When no standard test fits your situation, design your own in seven steps, and make sure it is a test that can actually fail.
Share
A product experiment template is a one-page plan for testing one belief: what you believe, what you will do to test it, with whom, what you will measure, and the result that counts as a pass, all written before you start. Use this custom version when none of the standard methods fits your situation, and give the method a name so you can repeat it.
Key takeaways
A custom experiment is a last resort. Testing Business Ideas already covers 44 experiment types, each rated for cost, time and evidence strength (Strategyzer, 2025).
Every good test plan makes four things explicit: the belief, the test, the metric and the threshold for success, as in Strategyzer's Test Card (Strategyzer, 2015).
Write the pass signal first. Signals stated before a test reduce biased readings of the result (Thoughtworks, 2014).
Expect to be wrong. Only a third of ideas tested at Microsoft improved the metric they targeted (Kohavi and Longbotham, 2023).
What is a custom product experiment?
A custom product experiment is a small test you design yourself when none of the standard methods fits, with a named belief, a method, and a result you set in advance.
The structure is not new. Alex Osterwalder's Test Card asks what needs to be true, how you will test it, what you will measure and what the threshold for success is. Barry O'Reilly's hypothesis format at Thoughtworks ends the same way: we will know we have succeeded when we see a measurable signal. This template adds who could prove you wrong, time and cost limits, and your next move after a yes or a no.
When should you design your own experiment instead of using a standard one?
Use it when your situation is unusual: a regulated setting, a small group of buyers, an odd channel, or a question that mixes several risks.
Do not use it just because a standard method feels like too much work. Check our guide to choosing a product experiment first. Most questions already have a tested method with well-known pitfalls. If the idea itself is still a blur, fill in a Lean Canvas first. And give your test a clear name. A test named "that thing we tried" cannot be compared with anything later.
Your situation
Check this standard method first
Design your own when
Buyers are a small, named group
Letter of intent, expert interviews
You cannot reach the person who signs
The sales cycle is a year or longer
Partner and supplier interviews
The decision runs through a chain of people
Regulation blocks fake doors or presales
Concierge, Wizard of Oz
Even manual delivery needs approval first
Which risk does it test, and how strong is the evidence?
You decide which risk this tests, and name it: desirability (do they want it?), feasibility (can we do it?), or viability (does it make money?). In Specky, a custom test defaults to desirability and directional evidence. Judge honestly how strong yours is, by asking: did people say something, do something, or commit something?
Strategyzer grades the evidence a team needs as directional, moderate or strong. Osterwalder's rule decides where your design lands: what customers say and do are two different things, and a simulated sale beats any survey (Strategyzer, 2017).
Product experiment template (copy and use)
Open the image at full size, or copy the text version below.
Who to recruit and how many
Whoever can actually show the belief to be wrong. State the group and how many. If you cannot say who could disprove it, the test is not ready.
Step-by-step plan
Write the belief as one sentence that could be false.
Name the risk it belongs to.
Describe the smallest action that would give you real evidence.
Decide who you need, how many, and for how long.
Write your pass signal, with a number.
Run it without changing the rules halfway.
Write down what you learned.
Test plan
Field
Your answer
Name of this method
We believe that
Risk
Desirability / Feasibility / Viability
To find out, we will
With these people
How many
We will measure
We'll say yes if
We'll say no if
Time limit
Cost limit
What we'll do if yes
What we'll do if no
Check your evidence level
What people say: They told you something.
Early signal: Real behaviour, but indirect or cheap to do.
What people do: They took a real action in a real setting.
Money or a signature: They gave up something of value.
What to measure
One primary number that your pass signal uses. One thing that must not get worse.
Pass signal to set before you start
Specky's starting point: write the number that would make you say yes. This is a starting point, not a benchmark. If you cannot write a number, describe what you would have to see, in enough detail that a colleague could judge it.
What we learned
Field
Your answer
What happened
The number
What surprised us
What we now believe
What we'll do next
How do you write a belief and a pass signal that can fail?
The belief and the pass signal are where most custom tests go wrong. A belief that cannot be false and a pass signal written afterwards turn an experiment into a story.
The base rates say you should expect a no. Ron Kohavi and Roger Longbotham report that a third of ideas tested at Microsoft improved their target metric and about 10% of Google's experiments led to business changes. They also cite Avinash Kaushik's line that "80% of the time you/we are wrong about what a customer wants." A plan with no believable path to "no" ignores those odds.
Weak version
Strong version
Why it is stronger
"Teachers like our tool"
"Teachers will ask their IT director to approve it"
It names an action, not a feeling
"We'll see how it goes"
"At least 4 of 12 directors reply with a next step"
A colleague can judge it without you
"Engagement goes up"
"Weekly active schools rise from 20 to 26 within 4 weeks"
It has a baseline, a target and a deadline
"People are interested"
"At least 3 of 10 buyers book a second call within 7 days"
Interest has to cost them something
Match the bar to the stakes. O'Reilly's example: a rocket launch needs a higher threshold than a sign-up flow. When being wrong would cost months, set a higher number or climb a rung on the evidence ladder.
Why track one thing that must not get worse?
Because a win on one number can hide a loss on another. Kohavi and Longbotham describe a Bing test where new site links raised revenue but degraded user metrics and page-load time. In a small custom test the protected number can be simple: support tickets, unsubscribes, or the goodwill of the people you asked for a favour.
How do you build a custom experiment from standard parts?
You rarely need to invent a method from scratch. Teresa Torres groups assumption tests into four kinds: prototype tests, one-question surveys, data mining and research spikes (Product Talk, 2023). Most custom designs combine one of these with an unusual audience or channel.
Building block
Evidence strength
Example of a custom twist
A one-question ask
What people say
One question in a professional association's newsletter
A prototype or mock-up
What people do
A printed mock-up at a trade fair, with a QR code to sign up
Existing data
Early signal
Public tender records showing how often districts buy similar tools
A research spike
Early signal
One integration run by hand in a partner's sandbox
A request that costs the person something
What people do, or money or a signature
Users forwarding a request to their budget holder
How many people, and how long, should a custom test run?
Let the evidence type decide. For commitments (requests, signatures, payments), a few of the right people is enough: 12 teachers who each reach a budget holder beat 200 anonymous clicks. For counts, aim for about 50 valid answers from one segment, as in our discovery survey guide. For watching people use something, start with five per round, as in the usability test template.
For time, Kohavi and Longbotham advise a main metric you can measure over a short period, such as two weeks. One to four weeks fits most custom tests. If your signal only appears after a year, test an earlier step that predicts it.
Custom experiment vs standard method: which should you use?
Standard method
Custom experiment
Known pitfalls
Documented by others
You find them yourself
Fit to odd settings
Sometimes poor
Designed for your setting
Use when
A method in the library fits
None of the standard methods fits
Worked example (hypothetical)
This example is made up to show the method. A team sells software to school districts, where a purchase takes a year, and a landing page would not reach anyone useful. They believe that district IT directors will approve a tool if a teacher asks for it.
No standard method quite fits. So they design a "teacher request test": they ask 12 teachers who use the free version to send a one-paragraph request to their IT director, and count the replies. They name it, and write a pass signal: at least four directors reply with a next step.
Five reply. Two ask for a security document. The team records it as a pass, under the name "teacher request test," so they can run it the same way next year.
Their filled-in plan could read:
Field
Their answer
We believe that
IT directors will approve a tool if a teacher asks for it
To find out, we will
Ask 12 teachers on the free version to email their director
We'll say yes if / no if
At least 4 reply with a next step / fewer than 2 reply
Must not get worse
No teacher reports a complaint from their director
Time limit / cost limit
3 weeks / €0 plus 2 hours of the team's time
Result
5 of 12 replied (42%); 2 asked for a security document
The two security requests are the real finding. The next test is whether a ready-made security pack turns replies into approvals.
What are the most common mistakes with custom experiments?
Not naming it. You will not be able to compare it to anything later.
Building a test that cannot fail. If nothing could make you say no, it is not a test.
Inventing a method to avoid a hard truth. Check the library first.
Testing too many things at once. One belief per test.
Overrating the evidence. Be honest about whether people merely said something.
Moving the line. Changing the pass signal once results arrive erases the test.
Custom tests often ask real people for favours, sometimes in sensitive settings like schools. Tell participants what you are testing, never ask them to claim something untrue, and never take money you cannot deliver on or refund.
How to run it in Specky
On the Assumptions page, click the note and choose Pick a test, then Other. Specky gives you a blank starting plan.
Name the method in the description, so that the evidence still has a label.
You run this one. Specky keeps the plan and the result.
Add your pass signal and click Start running. Then write down what you observed, and record the result.
Since this is logged as directional evidence, a pass shows as Early signal on the board, not Supported, even if your test was strong. If your test asked people to act or pay, say so in your notes.
What should you do with each outcome?
Supported (shows as Early signal): Follow up with a standard method that captures stronger evidence, such as a concierge test or a letter of intent.
Unclear: Your pass signal may have been fuzzy. Tighten it and run the test again.
Contradicted: The belief did not hold. Decide what to do next, using the "if no" line you wrote before you started. Any bet resting on this note now shows as Contradicted.
Write the belief as a sentence that could be false, choose the smallest action that would give you real evidence, decide who and how many, and set a pass signal before you begin. Add a time limit, a cost limit and one thing that must not get worse. Then run it without changing the rules halfway.
What should a product experiment template include?
At minimum: the belief, the risk it tests, the method, the people and how many, the metric, a pass line and a fail line, and a time and cost limit. Strategyzer's Test Card reduces this to four parts: belief, test, metric and success threshold. Add what you will do after a yes or a no.
What makes a good pass signal?
A good pass signal is a number or a specific outcome, written before the test starts, that a colleague could judge without asking you. If nothing could make you say no, it is not a pass signal. Raise the bar when being wrong is expensive, and write a separate "no" line.
When should I create my own experiment instead of using a standard one?
Create your own only when no standard method fits your situation, such as a year-long public-sector sale, a regulated product or a tiny buyer group. Standard methods come with well-known pitfalls, which saves you from learning them the hard way. Check the full library of 55 methods first.