How PMs Should Annotate AI Outputs: Separate Product Failures from Preferences
A practical review method for AI products: separate policy failures, missing context, and personal preferences so real output feedback can shape better product behavior.
Share
A rating is not an explanation
A reviewer gives an AI answer four stars. What should the product team change?
Maybe the answer missed the user’s question. Maybe it included a claim with no source. Maybe the facts were right, but the tone felt stiff. Maybe the reviewer prefers a different structure even though the answer did the job.
Those reactions are useful, but they describe different things. If they all become “quality: 4/5,” the team loses the information needed to improve the product.
A practical annotation process starts by separating what the output did, which product rule it met or missed, and what the reviewer personally preferred. That turns review into a way to clarify product behavior, rather than a popularity vote on generated text.
Decide what kind of judgment you are collecting
Before asking people to label outputs, decide what the labels should represent.
Sometimes the product needs a consistent policy. For example, an AI product brief must not present an unsourced customer claim as a fact. Reviewers should apply that rule consistently. This is a policy judgment: the team is defining one intended product behavior.
Other times the product should preserve differences in people’s needs. One product manager may prefer a concise summary; another may need more detail for a stakeholder review. If both formats work for their jobs, recording the preference can teach the product about different contexts. This is a preference judgment, not automatically a product failure.
Research on annotation for subjective language tasks describes a similar distinction between collecting different perspectives and instructing annotators to apply one defined policy. The choice depends on what the dataset is meant to support. The same decision matters when a product team reviews AI outputs: are you learning what users prefer, or deciding what the product must do? Röttger et al., “Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks”
Write the purpose above the review form:
We are collecting these labels to decide whether the AI output follows the product’s agreed behavior for this task.
We are collecting these reactions to learn which valid output styles work better for different users and situations.
A team can collect both kinds, but the labels should keep them separate.
Label the behavior before judging the prose
For each output, record three things:
Observed behavior: What did the system actually do?
Criterion: Which agreed product rule applies?
Impact: What could happen to the user if this behavior stays as it is?
For a research synthesis feature, a small set of behavior labels might be:
Label
Use it when
Task miss
The output does not answer the job the user asked it to do
Unsupported claim
A factual statement cannot be traced to the provided material
Important omission
Relevant evidence or a meaningful contradiction is missing
Context mix-up
The output combines sources or people that should remain distinct
Wrong next step
The recommendation does not follow from the evidence and stated goal
Preference
The output works, but the reviewer would choose a different style or structure
Cannot judge
The source or task context needed to review it is missing
The labels are only a starting point. Adapt them to the product’s actual user job and risks. A label such as “hallucination” is often too broad to guide a fix. “Claim about a customer’s reason for leaving has no supporting interview passage” tells the team what failed and what evidence would settle it.
Make every failure inspectable
When a reviewer marks a product failure, ask for a short evidence trail:
Select the exact phrase or section in the output.
Link the source or task instruction it should follow.
State the expected behavior in one sentence.
Describe the user impact.
Mark severity separately from confidence in the label.
For example:
Field
Example
Output excerpt
“Users left because setup was too complex.”
Source
Three interviews mention setup, but none state why the users left
Criterion
Do not state a cause unless a source supports it
Expected behavior
Say that setup was mentioned; leave the reason open
Impact
A team could prioritize the wrong fix
Severity
High for this decision
Label confidence
Clear
Severity and certainty answer different questions. A reviewer can be unsure whether a claim is unsupported while recognizing that, if it is, the consequence would be important. Keeping those fields separate helps a PM decide what needs immediate investigation.
Do not ask the reviewer to diagnose the technical cause. “The retrieval system failed” is a hypothesis about implementation. The annotation should preserve the observable failure; engineers can investigate why it happened.
Treat disagreement as a signal
When two reviewers choose different labels, do not automatically settle it with a majority vote. First classify the disagreement.
Rule is unclear: The reviewers cannot tell what the product promises. Rewrite the criterion and add an example.
Context is missing: A reviewer lacks the source, user goal, or permission needed to judge. Fix the review case.
Policy tradeoff is unresolved: Both judgments are reasonable, but they imply different product behavior. The PM should make or facilitate that decision with domain experts.
Valid preferences differ: Preserve the preference and ask whether it maps to a user segment, task, or context.
Review execution differs: The instructions may be clear, but someone may have overlooked evidence or misunderstood the task. Clarify the review guide and repeat the case.
This framing avoids turning every difference into “human error.” A disagreement can reveal an incomplete policy or a real variation in user needs. A 2022 study of annotation paradigms makes the key point: sometimes the goal is to model differing perspectives; other times it is to apply one defined belief. Decide which goal fits the product before collapsing labels into a single answer.
Run a short calibration before scaling review
Start with a handful of representative outputs: a straightforward case, a missing-source case, a contradiction, a style preference, and a boundary case. Ask two or more reviewers to label them independently, then compare the evidence and reasoning together.
The goal of calibration is not to get a perfect agreement score. It is to find where the instructions fail to distinguish:
a factual error from an unsupported inference
a missing detail from a detail that is irrelevant to the user’s task
a style preference from a usability problem
a low-impact flaw from a release-blocking behavior
a model error from missing or poor input context
Update the guide with the resolution and a compact example. Keep the initial independent labels too; they show whether the disagreement came from the case, the policy, or the review instructions.
The NIST AI Risk Management Framework calls for teams to define responsibilities and oversight around human-AI use. For a product team, a review guide is one practical way to make those responsibilities visible: who supplies domain judgment, who applies the rule, and who follows up on implementation and reliability.
Connect annotations to product decisions
An annotation set is not a launch decision by itself. It is evidence for decisions such as:
Is the product behavior defined clearly enough to build?
Which failure needs a product-policy change?
Which failure belongs to implementation or reliability?
Does the team need more representative examples?
Is this a valid preference that calls for customization?
What must be re-reviewed after a prompt, model, or workflow change?
Keep a short record of the review batch: task and user context, product-rule version, example sources, labels, unresolved disagreements, decisions, and owner. When the rule changes, version it and revisit the examples that depend on it. That lets the team tell whether an output changed because the product policy changed or because the system behaved differently.
PMs own the judgment about the intended user behavior, domain tradeoffs, and what the examples should teach. They can write the prompts, review outputs, and define acceptance criteria because that work specifies the product. Engineers own production implementation, access controls, reliability, and the systems that make the behavior work.
A copyable annotation card
Use this for the next review session:
Task and user: Product rule being checked: Output excerpt: Source or context: Observed behavior: Expected behavior: User impact and severity: Label confidence: Policy failure, missing context, or preference? Decision, owner, and next review:
If the team cannot fill in the product rule, pause the scoring and decide what the product should do first. The rubric is only useful when it reflects a product decision the team is willing to own.
Specky helps teams keep user evidence, product decisions, and next steps connected. Explore the product workspace for technical PMs or start shaping a requirements document with the PRD generator.