Spec Hallucination: The 15-Minute Claim Trace for AI-Written Specs
AI writes specs that sound sourced but often aren't. The Claim Trace is a 15-minute audit that labels every claim Sourced, Inferred, Assumed or Invented before engineering builds from it.
Share
Ask an AI to write a PRD and you get a confident, well-formatted document in under a minute. Ask where each sentence came from and the confidence disappears. That gap has a name now: spec hallucination. A "Show HN" project this week pitched a multi-persona library built specifically to fight it, and an essay circulating among product leaders makes the adjacent point that AI made feasibility almost trivial while value, viability and usability stayed exactly as hard as before.
Both point at the same problem. When writing a spec costs nothing, the scarce thing is not the document. It is the part of the document you can defend.
This post gives you a 15-minute method for auditing any AI-written spec before engineering, or an AI coding agent, builds from it. We call it the Claim Trace.
What spec hallucination actually looks like
Spec hallucination is rarely a made-up feature. It is a made-up reason.
The feature is plausible, the structure is clean, and buried in the "Problem" and "Success metrics" sections are sentences like these:
"Admins currently export data manually every week."
"Reporting was the most-requested capability among churned accounts."
"We expect a meaningful lift in activation."
Each reads like a finding. None may be one. The model has seen thousands of PRDs, so it produces the shape of evidence: a behavior, a segment, a direction of change. If your inputs were thin, the model fills the gaps with whatever is statistically typical for a spec like this.
The danger is not that the sentences are always wrong. It is that they are indistinguishable from the ones that are right. A reader skimming the document cannot tell which claims came from your interviews and which came from the model's priors.
Why this got worse in 2026
Three things changed.
Drafting cost collapsed. A spec used to take a PM a day, and the day forced contact with the evidence. Now the draft arrives before anyone has reopened the research.
Downstream cost did not collapse. If an AI coding agent builds from a spec, a fabricated premise becomes working code quickly. You discover the premise was invented after the feature ships, not before.
S
Specky Team
Writing about AI-native product development at Specky.
Review habits didn't adapt. Teams review specs for clarity, completeness and tone. Those are exactly the properties language models are best at faking. Almost nobody reviews for provenance.
The fix is not "don't use AI to draft." It is to add the review step the draft skipped: trace every claim to a source.
The Claim Trace: four labels, one pass
Read the spec once, sentence by sentence. Every sentence that asserts something about users, the market, the product or the future gets exactly one label.
1. Sourced
The claim points to something you can open: a specific interview, a support ticket, a usage query, a signed decision. "Three of the last five onboarding calls mention exporting to spreadsheets" is sourced if you can link those three calls.
The test is mechanical. Can a teammate who wasn't there click through and see the same thing?
2. Inferred
The claim is a reasonable conclusion from sourced material, but the conclusion itself is new. "Admins likely export because they report upward" is inferred from the calls above. Fine, as long as it is labelled and the underlying sources are attached.
3. Assumed
The claim is something the team believes but has not checked. Assumptions are not shameful. Specs run on them. The failure is when an assumption is written in the same voice as a finding.
4. Invented
The claim has no source, no inference chain, and nobody on the team can say where it came from. This is the label that matters most. Invented claims are the model's contribution to the document, and they are the ones to delete or convert.
Do not argue about edge cases between labels. If you hesitate between Inferred and Assumed, pick Assumed. The label is a prompt to look closer, not a verdict.
Running the audit in 15 minutes
Minutes 0-3: extract. Paste the spec into a scratch list and pull out every factual claim. Skip pure requirements ("the export must be a CSV"). Those are decisions, not claims. You are looking for sentences that begin with "users", "customers", "admins", "currently", "most", "because", or a number.
Minutes 3-8: trace. For each claim, try to find the source in under 60 seconds. If you cannot, it is not Sourced. Do not go hunting for a justification after the fact. Hunting until something loosely fits is how invented claims get laundered into sourced ones.
Minutes 8-12: act on each label.
Sourced: attach the link inline.
Inferred: attach the link and rewrite so the inference is visible ("Based on the three calls linked, we think...").
Assumed: move it to an assumptions list with an owner and a cheapest test.
Invented: delete it, or turn it into a question to answer.
Minutes 12-15: re-read the spec. If the Problem section collapsed when the Invented claims were removed, you have learned the most valuable thing the audit can tell you: the feature does not have a problem yet.
A worked example
Take a made-up but typical draft for an admin export feature. The AI-written Problem section says:
Admins export data manually every week, spending hours in spreadsheets. Reporting is the top request among churned accounts. A native export will reduce churn.
Run the Claim Trace:
Claim
Label
Action
Admins export data manually every week
Assumed (one call mentioned it, nothing about frequency)
Move to assumptions, test with a usage query
Spending hours in spreadsheets
Invented
Delete
Reporting is the top request among churned accounts
Invented (no ranking exists anywhere)
Delete; replace with a question
A native export will reduce churn
Assumed
Keep as a hypothesis with a metric and a date
Four claims, two deleted, two demoted. The rewritten Problem section is shorter and honest: one call mentioned exporting, the rest is a hypothesis, and here is the cheapest way to test it. That document can still lead to a build. But the team now knows what they are betting on.
The assumption ledger is the real output
The audit's most useful artifact is not the cleaned spec. It is the list of Assumed claims, because that list is your discovery plan.
For each assumption, record four fields: the claim, who owns checking it, the cheapest test (a query, a call, a fake-door), and the date by which you will have decided. A spec with a short, owned ledger is a spec that can move fast safely. A spec with no ledger either has no assumptions, which is nearly never true, or has hidden them.
Making provenance the default instead of an audit
A manual trace works, but it relies on discipline at the exact moment discipline is cheapest to skip. Better to make provenance part of how the draft is produced.
This is the design stance behind Specky. Signals, interviews and decisions live as connected nodes in the product graph, and when the AI drafts an insight, decision or spec, it is expected to cite the specific evidence nodes it used. Drafts land in the PM Inbox for human review rather than going straight into the record. A reviewer can see which sentences carry citations and which carry none, which is the Claim Trace done at write time instead of after the fact.
You can approximate this in any tool. Require the model to attach a source to each factual claim, and treat any claim it cannot source as a candidate for the Invented label. A model that says "no source" is giving you a better answer than one that fills the gap fluently.
Where the audit stops
Be clear about what this method does not do. A fully sourced spec can still describe the wrong thing, because sources can be biased, thin or unrepresentative. The Claim Trace tells you whether a claim has a source, not whether the source is good. Pair it with a check on evidence quality: how many independent sources, how recent, how close to the actual buying or usage decision.
It also does not replace judgment about what to build. It makes judgment visible. When the document says "we are assuming this", the people reading can disagree with the assumption instead of absorbing it.
The takeaway
When a spec costs nothing to produce, the question to ask of it changes from "is this well written?" to "which parts of this can I defend?" Label every claim Sourced, Inferred, Assumed or Invented. Delete the invented ones. Own the assumed ones. Attach the links to the rest.
Do it once on a spec you are about to hand to engineering or an agent this week. If more than a third of the Problem section comes back Invented, you have found the problem before it cost the team a sprint.
Specky is the AI product workspace that shows its work — every AI draft cites the evidence behind it, so you can see what's sourced and what's guessed. → specky.space