Most "AI Features" Shouldn't Be AI Features. Here's the Test That Tells You.
Most "AI features" shipped in 2026 are rules engines wearing an AI costume. Here's the four-question test to run before you spec one — and what it looks like when a feature fails all four.
Share
Most "AI Features" Shouldn't Be AI Features. Here's the Test That Tells You.
Every roadmap review in 2026 has the same moment. Someone proposes a feature, and somewhere in the pitch — usually within the first thirty seconds — the word "AI" shows up. Not because the problem demands it. Because it's 2026, product management job postings are up 14% year over year after two rough years, and nobody wants to be the PM who shipped the boring version.
Pragmatic Institute puts AI adoption inside product teams at 64% and climbing. That's not the problem. The problem is how many of those AI features are solving problems a five-line rule would have solved better, cheaper, and in a way a support engineer could actually explain to an angry customer at 11pm.
The AI PM's real first job isn't writing the model spec. It's deciding whether the feature should touch a model at all. Most teams skip that step, because "let's add AI" feels like a decision and "let's check if we need it" feels like stalling. It's the opposite. Skipping the check is how you end up maintaining a probabilistic system for a problem that never needed one.
Why this keeps happening
Three forces are pushing AI into places it doesn't belong:
Investors and boards ask about it. "What's your AI strategy" is now a standing agenda item, and "we use it where it earns its keep" is a less exciting answer than a roadmap slide with a chatbot icon on it.
Building the wrong version is cheap now. A working AI prototype takes an afternoon. That speed is genuinely useful for testing ideas — and it's also exactly what makes it easy to ship something before anyone asks whether it should exist. Fast to build isn't the same as right to build.
"AI feature" reads as progress to people who don't read the changelog closely. A deterministic improvement — a better default, a smarter form, a rule that removes three clicks — often delivers more user value than a model-backed version of the same thing, but it doesn't photograph as well in a release announcement.
None of these are bad reasons to want AI in the product. They're just bad reasons to skip the check.
The AI Fit Test
Before a feature gets an AI spec, run it through four questions. If it fails three or more, you're looking at a rules engine wearing an AI costume — and the rules engine is the better product decision.
S
Specky Team
Writing about AI-native product development at Specky.
AI earns its cost on messy, unstructured, high-variance input: free text, images, a support ticket that could mean six different things. If the input is a clean, bounded set of fields — a dropdown, a status enum, a number with a known range — a model is solving a problem that if/else already solved, just with worse latency, worse cost, and a failure mode nobody can fully explain.
Test question: can you write the input's shape as a TypeScript type in under five lines? If yes, be suspicious.
2. Does the mapping from input to output generalize, or is it really a fixed rule?
This is the one teams get wrong most often. "Categorize this ticket" sounds like a classic ML problem until you notice there are only six categories, the rules for each haven't changed in two years, and a support lead could write the routing logic in an afternoon. Generalization is the thing you're paying for with a model. If the actual mapping is stable and enumerable, you don't need generalization — you need a lookup table.
Test question: if the "right answer" for every case were written down today, would that list still be right in six months? If yes, it's a rule, not a model.
3. Can the product tolerate being wrong sometimes?
Models are probabilistic. That's fine for a first-draft summary, a suggested reply, a ranked list — anywhere a human reviews the output before it matters. It's not fine for billing calculations, access control, compliance determinations, or anything where "usually right" creates a support queue, a refund, or a regulator's attention. The EU AI Act framework raises the cost of getting this one wrong even further for teams selling into Europe — a "usually right" system in a high-stakes category isn't just a UX risk anymore, it's a compliance one.
Test question: if this is wrong 1 time in 20, does a person notice before it causes damage, or after?
4. Do you actually have the data to make it good — not just the API key to make it exist?
An AI feature backed by thin, unrepresentative, or entirely synthetic data isn't a shipped feature, it's a demo wearing a production URL. This is the check teams skip most under deadline pressure, because "we'll improve it with real usage data" sounds like a plan and is usually a hope. If you can't point to the examples the model will actually see in production, you don't know if it'll work — you're finding out live, with users as the test set.
Test question: could you show a skeptical engineer 20 real examples of the exact input this feature will see, today? If the honest answer is "not yet," the feature isn't ready — the data collection is the actual next step, not the model.
What "failing" the test looks like in practice
A B2B SaaS team we've talked with was building an "AI-powered" onboarding checklist that reordered setup steps based on a new user's signup answers. Four form fields, a plan tier, and a company size band — a bounded, enumerable input if there ever was one. The mapping from those fields to "which three steps matter most" was something their own sales team already knew and could write down. There was zero tolerance for getting it wrong, because a reordered checklist that buries the one step that actually matters kills activation. And they had exactly nine users' worth of real behavioral data to train on.
Four questions, four fails. They shipped a rules-based checklist instead — six weeks faster, and it's the version still in production a year later, because there was never a model quietly drifting out of date in the background.
That's not an anti-AI story. It's a story about where the four minutes it takes to run the test would have saved six weeks of the wrong build.
Where AI actually clears the bar
The test isn't a case against AI — it's a filter for where it earns its cost. Drafting a first-pass PRD from messy interview notes: ambiguous input, no single "correct" mapping, low stakes because a human reviews it, and plenty of real examples to learn from. Summarizing a week of scattered customer feedback into themes: same shape. Surfacing which of a hundred backlog items are duplicates of each other: genuinely hard to write as a rule, tolerant of being wrong sometimes, and gets better with more data. These pass on all four counts, and they're exactly the categories where AI-assisted product work has actually stuck instead of getting quietly turned off six months after launch.
Running the test on what you've already shipped
The test isn't only for new specs. Point it at your last three "AI-powered" features and it usually turns up at least one that's failing quietly — not broken, just costing more than it's worth. A model that classifies a fixed, unchanging set of categories is a maintenance burden pretending to be a capability: it needs monitoring, it drifts, it has a bill, and none of that shows up on the roadmap slide that shipped it.
The retrofit version of the test asks one extra question: what would it cost to replace this with the boring version today, and what would we lose? Sometimes the answer is real — the model genuinely generalizes to cases the rule-based version missed, and that's the evidence that justifies keeping it. Sometimes the honest answer is "nothing, we'd lose nothing, we'd just stop paying for inference on a lookup table." Either way, you now have a documented reason instead of a feature that exists because nobody revisited the decision.
This is worth doing on a cadence, not just once. A feature that correctly needed AI at launch — because the input was genuinely ambiguous — can stop needing it a year later, once the categories stabilize and the "ambiguous" cases turn out to cluster into six predictable buckets. The test doesn't get run once and filed away. It's a periodic check, the same way you'd revisit a pricing assumption or a churn hypothesis.
The objection worth taking seriously
The strongest pushback on this framework isn't "AI is always better" — most experienced PMs already know that's false. It's "we don't have time to run a four-question test on every feature." That's fair, and it's also why the test is scoped the way it is: four questions, each answerable in a sentence, not a two-week discovery sprint. The cost of running it is minutes. The cost of skipping it is a model in production that a rule would have replaced, still getting maintained eighteen months later by someone who's forgotten why it's there.
The teams that get the most value from AI in 2026 aren't the ones that shipped it fastest. They're the ones that can point to why each AI feature is AI — and, just as importantly, why the features around it aren't.
Make the test part of the spec, not a debate
The reason this check gets skipped isn't that PMs don't know it matters — it's that there's no forcing function. Nobody wants to be the person in the room who says "wait, do we need a model for this" after three people have already nodded at the AI-feature slide. The fix isn't a stronger opinion. It's making the four questions a mandatory section of the spec, right next to the success metric — answered in writing, with the actual examples cited, before a single prompt gets written.
That's the same discipline behind evidence-based product decisions generally: the choice is only as good as the reasoning that's visible next to it. A feature spec that shows its work — this is the input shape, this is why it's ambiguous, here are the 20 real examples, here's why "usually right" is tolerable here — is a spec a skeptical engineer can actually push back on. A spec that just says "AI-powered" isn't a decision. It's a placeholder for one.
Specky is the AI product workspace that shows its work — every AI-or-not call, every framework, every real example lives in one searchable decision trail instead of a Slack thread nobody can find again. → specky.space