Wizard of Oz Test: Template, Operator Guide and Real Examples
Before you spend six months building the hard part, put a person behind the curtain and see if the experience works.
Share
A Wizard of Oz test is a product experiment where customers use what looks like a finished, automated product while a person quietly does the work behind it. It shows whether the experience works, and what the machine would need to do, before you spend months building the hard part. Below is a free Wizard of Oz test template, an operator guide and the ethics rules.
Key takeaways
The method dates to 1973; Jeff Kelley coined the name in 1983 (NN/g, 2024).
Strategyzer rates the Wizard of Oz 2 out of 5 for cost and 5 out of 5 for evidence strength (Strategyzer poster).
It tests the experience, not the technology: a pass says people want what the machine would do, not that one can.
For AI products, log how often the human overrides the model. Kromatic reads an approve rate near 90% as a sign an off-the-shelf model may do, and near 30% as the human doing the work (Kromatic).
It involves deception: avoid high-stakes advice, protect data, debrief everyone.
What is a Wizard of Oz test?
In a Wizard of Oz test, the customer sees a working interface: a chat window, an app, a search box. Behind it, a person (the "wizard" or operator) reads each request and produces the response by hand. NN/g describes it as an interface that appears autonomous but is fully or partially controlled by a human. In moderated studies, a facilitator runs the session while the wizard stays out of sight.
Startups also call it a Wizard of Oz MVP. Of the 44 experiment types in David J. Bland and Alex Osterwalder's Testing Business Ideas (Strategyzer), it is a validation experiment in the "simulation" group.
Where does the Wizard of Oz name come from?
From L. Frank Baum's story, where the wizard is an ordinary man behind a curtain. The research history:
1973: Don Norman and Allen Munro used the method to test an automated airport travel assistant, the first documented use according to NN/g.
1983: Gould, Conti and Hovanyecz published "Composing letters with a simulated listening typewriter" in Communications of the ACM. Speech recognition was not good enough, so they simulated recognisers with 1,000-word, 5,000-word and unlimited vocabularies (CACM, 1983). A 1990 replication notes that their operator typed the dictation on a QWERTY keyboard (Newell et al., 1990).
1983-1984: Jeff Kelley named the method in his Johns Hopkins dissertation on natural-language interfaces, and published his iterative design method for natural-language office systems in ACM Transactions on Information Systems in 1984 (Kelley, 1984).
What are some Wizard of Oz examples?
Zappos. In 1999 Nick Swinmurn photographed shoes at a Sunnyvale store, sold them through a website, and bought each pair at full price only after an order came in (Fortune, 2012). Shoppers saw an online store, not a man with a camera.
AI assistants and chatbots. Kromatic notes most tests today are hybrids: a language model drafts each reply and a human approves, edits or rewrites it. The test then shows whether the experience works and how much the human is fixing.
When should you use a Wizard of Oz test, and when not?
Use it when your product promises something that is hard to automate, like AI that really understands a request, and you need to know if the experience works before you spend months on the hard part. NN/g names chatbots, recommendations and real-time lookups as typical fits.
Do not use it when a wrong answer could harm someone, as in health, legal or safety matters, unless an expert is doing the work. If customers can know a person is helping, a concierge MVP is simpler and more honest. And if a static mockup can answer your question, NN/g's advice is to use that instead.
Which risk does a Wizard of Oz test check, and how strong is the evidence?
This tests feasibility from the customer's side: if the machine could do this, would the experience work, and what would it have to do? The evidence is behaviour, because real customers use it. It says little about whether software could really do the job. See our guide to choosing a product experiment.
Strategyzer scores it 2/5 for cost, 3/5 for setup time, 3/5 for run time and 5/5 for evidence strength. The real cost is operator time.
Wizard of Oz test template (copy and use)
Open the image at full size, or copy the text version below.
Who to recruit and how many
Five customers with a real job to do. Keep the group small, since each one requires a person at the controls. Do not promise anything you would be unable to deliver by hand.
Step-by-step plan
Write what the customer will see and what the person will do.
Set your pass signal, and the longest acceptable wait.
Build a simple front end.
Choose an operator and write down their rules.
Run it for a set number of hours per day, and be clear about those hours.
Log each request, the operator's answer, and the time taken.
Tell customers how it worked at the end of the test.
Operator guide
Reply within [X] minutes.
If you cannot do it, say: "I can't do that yet."
Do not do anything the real product could never do.
Log every request, including those that you could not answer.
Questions to ask afterward
"What did you expect to happen when you asked for that?"
"Where did it surprise you?"
"Did anything feel off?"
"If it worked like this every day, what would you use it for?"
After you explain how it worked: "Does that change how you feel about it?"
What to measure
Jobs completed. Time per request. Share of requests the operator could answer within the rules. How many customers noticed, or minded.
Pass signal to set before you start
Specky's starting point: at least 3 of 5 finish the job and don't notice or don't mind. This is a starting point, not a benchmark. Add a limit of your own on operator time per job.
Results log
Customer
Request
Operator time
Answered within rules?
Job finished?
Noticed or minded?
Comments
How do you prepare the operator?
The operator decides whether your results mean anything. NN/g's setup steps and Kromatic's operator advice come down to five habits:
Pick a response method.Closed (choose from prepared answers) is fast but rigid; open (write each answer) is flexible but varies; hybrid mixes both.
Write the rules down, and ask an engineer which responses are feasible.
Build a console the customer never sees: request, draft, send, log.
Pilot with a colleague to find the responses you forgot.
Check the operator. If a model drafts, Kromatic suggests blind double-checks on about 10% of replies, because operators start rubber-stamping.
How many participants do you need for a Wizard of Oz test?
Specky's template starts with five customers, enough to see whether the experience works and to collect the requests your rules missed. For a clearer read on behaviour such as return visits or payment, Kromatic suggests 10 to 30 users over 1 to 3 weeks, at about $40 to $400.
Do the operator maths first: five customers, 10 requests each, 5 minutes per answer is roughly 4 hours. Thirty customers is 25 hours. Open the product only when someone can staff it.
What should you measure in a Wizard of Oz test?
The template's measures say whether it passed. For AI products, add override data:
Measure
What it tells you
Jobs completed
Whether the experience works at all
Time per request
Whether the wait is acceptable, and what the machine must beat
Does the experience work if a machine could do it?
Do they return to one slice, unprompted?
Does the chain of tools hold end to end?
Strategyzer cost / setup / run / evidence (of 5)
1 / 2 / 3 / 5
2 / 3 / 3 / 5
4 / 3 / 4 / 5
3 / 3 / 4 / 5
Specky pass signal
3 of 5 come back or pay
3 of 5 finish and don't notice or mind
4 of 10 use it again in week 2
3 finish end to end and ask to continue
The deciding question: would people judge a human service differently from software? If yes, use Wizard of Oz. If not, a concierge is simpler and needs no deception.
What does a Wizard of Oz test look like in practice?
This example is made up to show the method. A team is planning a chatbot that answers tenants' questions using lease documents. Before building anything, they set up a chat window, and a team member reads each lease and types the answers within five minutes.
Five property managers try it for a week. All five finish their tasks. Three say the wait is fine. Two ask for answers to cite the exact clause. The log shows 40 of 52 questions could be answered from the lease alone.
The signal passes. Better still, the team learned that citing the clause matters, and that 12 in 52 questions need someone to make a judgment call. Both things shape what they build.
The log also sizes the build: 40 of 52 is 77% answerable from the document, for roughly 3.5 hours of operator time at four minutes an answer. Next, a spike tests whether a model can cite the right clause for those 40 questions.
What are the most common Wizard of Oz testing mistakes?
Promising what you can't do. The operator is bound by what the real product could eventually do.
Not writing rules. Without them, your operator becomes a very clever human.
Forgetting privacy. A person is reading customer data, so handle it with care.
Never telling people. Share how it worked once the test is over.
Leaving out response time. The delay is part of the experience.
Is a Wizard of Oz test ethical?
It can be, if you limit the deception. NN/g calls for a debrief at the end. Kromatic adds rules worth adopting:
No high-stakes advice through the wizard: nothing medical, legal or financial unless a qualified person is doing it.
Collect only what the shipped product could collect, and say in your privacy notice that your team may review requests.
Refund anyone who paid for a capability that was not built.
Debrief everyone, and keep any promise the wizard made.
In the EU, a person reading customer messages is processing personal data, so settle your lawful basis first.
How to run it in Specky
On the Assumptions page, click the note and choose Pick a test, then Wizard of Oz. Specky fills in the steps, the pass signal, and a two-week timeline.
Click Test with a page. Specky builds a page to invite customers in, and collects the responses.
The front end and the person behind it are your part. Specky does not play the wizard for you.
Click Start running. Paste your operator log onto the test, and record the result.
A pass shows on the board as Supported, because customers using it is behavior.
What should you do with each outcome?
Supported: The experience works. Now find out if a machine could do it with a technical spike.
Unclear: People finished, but the operator's work was too heavy or too clever. Tighten the rules and run it again.
Contradicted: People did not finish, or the experience was not good. Do not automate something that does not work by hand. Any bet resting on this note now shows as Contradicted.
Frequently asked questions
What is a Wizard of Oz test?
A Wizard of Oz test presents what looks like a working product while a person secretly does the work. It lets you test the experience before building the difficult technology behind it. Teams use it most for chatbots, voice interfaces and AI features.
Is a Wizard of Oz test deceptive?
It can be, so be careful. Avoid it where a wrong answer could harm someone, protect any data your operator reads, and tell participants how it worked once the test is over. Nielsen Norman Group recommends a debrief, and anyone who paid for a capability that does not exist should get a refund.
What is the difference between Wizard of Oz and concierge testing?
In Wizard of Oz testing, customers think they are using software. In concierge testing, customers know a person is doing the work. Choose Wizard of Oz when people would judge a human service differently from an automated one; otherwise, a concierge is simpler and avoids deception.
Who invented the Wizard of Oz method?
Nielsen Norman Group dates the first documented use to 1973, when Don Norman and Allen Munro simulated an airport travel assistant. Jeff Kelley coined the name in his 1983 Johns Hopkins dissertation and published the method in 1984.
How many people do you need for a Wizard of Oz test?
Five customers is a practical start, because each needs an operator at the controls. That is enough to see whether the experience works. To measure behaviour such as repeat use or payment, Kromatic suggests 10 to 30 users over 1 to 3 weeks; budget the operator hours before you recruit.