Do 95% of AI Pilots Fail? What the MIT Study Actually Measured
The 95% figure is not a universal AI failure rate. Here is what the 2025 MIT NANDA report measured, why critics dispute it, and how product teams can judge an AI pilot on real outcomes.
Share
No, the evidence does not show that 95% of AI pilots fail. The figure comes from a July 2025 MIT NANDA report built on 52 interviews and 153 survey responses. It says 95% of organisations saw no measurable P&L return from generative AI. Among firms that actually piloted a custom tool, about one in four reached production, and the report calls its own numbers directional.
Key takeaways
The report says "95% of organizations are getting zero return" on $30-40 billion of enterprise GenAI spending (MIT NANDA, 2025). That describes organisations and P&L impact, not the outcome of every pilot.
For task-specific tools, 60% of organisations evaluated one, 20% piloted, and 5% reached production. Of the firms that piloted, roughly 25% (5 of 20) succeeded.
"Success" meant a marked and sustained productivity or P&L impact, measured six months after the pilot. That is a high bar, and the report does not define it precisely.
Critics point to the small sample, no peer review, internal inconsistencies, and the authors' commercial interest in the agentic AI they recommend (80,000 Hours, 2026).
The useful lesson for product teams is not a failure rate. It is to set a baseline, a success threshold and a decision date before the pilot starts.
Where does the "95% of AI pilots fail" statistic come from?
The number comes from The GenAI Divide: State of AI in Business 2025, written by Aditya Challapally, Chris Pease, Ramesh Raskar and Pradyumna Chari and dated July 2025. It was produced with Project NANDA, an MIT Media Lab group that builds infrastructure for networked AI agents (MIT Media Lab). The report labels itself as preliminary findings.
Its stated method has three parts, covering January to June 2025:
a review of more than 300 publicly disclosed AI initiatives
structured interviews with representatives from 52 organisations
survey responses from 153 senior leaders collected at four industry conferences
The statistic went mainstream on 18 August 2025, when Fortune reported that 95% of generative AI pilots at companies are failing (Fortune, 2025). Fortune described the research as 150 interviews with leaders, a survey of 350 employees and 300 public deployments. Those numbers are two to three times larger than the 52 interviews and 153 responses in the report itself.
The report contains several different numbers that headlines merged into one. Separating them changes the story.
What people repeat
What the report says
What it actually describes
95% of AI pilots fail
95% of organisations are getting zero return
Organisations, judged on measurable P&L impact
Only 5% of pilots succeed
5% of integrated AI pilots are extracting millions in value
A share of all organisations, not of pilots
Pilots don't reach production
60% evaluated task-specific tools, 20% piloted, 5% reached production
A funnel across all surveyed organisations
AI tools don't get used
Over 80% explored or piloted ChatGPT-style tools; nearly 40% report deployment
General-purpose tools, judged separately
Most AI fails
Success means a marked and sustained productivity and/or P&L impact
Self-reported by users or executives, measured 6 months after the pilot
The funnel matters most. If 60% of organisations looked at a task-specific tool, 20% ran a pilot and 5% reached production, then 80% never piloted anything. Among the 20% that did, 5 in 20 succeeded: a pilot-to-production rate of about 25%.
80,000 Hours host Rob Wiblin made this point in April 2026: the figure describes all surveyed organisations, most of which never ran a custom pilot (80,000 Hours, 2026). The report invites the confusion by calling the result a 95% failure rate in one passage.
The report also flags its own limits. It says the pilot figures are "directionally accurate based on individual interviews rather than official company reporting", that sample sizes vary by category, and that success definitions may differ across organisations.
What are the main criticisms of the MIT study?
Five problems come up repeatedly. None proves enterprise AI is going well, but together they mean the 95% figure cannot carry the weight placed on it.
1. The sample is small
The headline rates rest on 52 interviews. 80,000 Hours estimates that the 5% success figure may reflect two or three companies, so moving one or two interviews would change the headline. The report's own methodology note admits the sample may not represent all enterprise segments or regions, and that firms willing to discuss AI may differ from those that declined.
2. It was not peer-reviewed or easy to check
The report was not an academic paper. When the Fortune story went viral, the PDF sat behind a request form, so journalists repeated the headline without reading the method (80,000 Hours, 2026).
3. The numbers do not always agree with each other
Two examples from the same document:
One takeaway says 50% of GenAI budgets go to sales and marketing; the paragraph below says sales and marketing captured approximately 70% of a hypothetical $100 allocation.
The executive summary says only 2 of 8 major sectors show structural change; a later section says 7 of 9 sectors show no real structural change.
4. "Success" is a high, loosely defined bar
A pilot counted as successful only if users or executives described a marked and sustained productivity or P&L impact within six months. A tool that helped individuals but had not yet moved the P&L counted as a failure. The report even concedes that a six-month window may be too short to judge complex enterprise systems.
5. The authors recommend what they build
The report concludes that agentic systems with memory and learning will close the gap, and NANDA is a project for agent infrastructure. 80,000 Hours argues this conflict of interest should have been disclosed.
So what is the real AI pilot failure rate?
There is no single, reliable AI pilot failure rate, because studies define "failure" differently. A model can give wrong answers, users can stop using a feature, a pilot can be cancelled, a workflow can stall before production, or a team can fail to measure financial impact.
RAND's 2024 report cites estimates that more than 80% of AI projects fail, twice the rate of IT projects without AI (RAND, 2024). RAND attributes that figure to outside estimates; its own evidence is interviews with 65 data scientists and engineers with at least five years' experience.
Self-reported success is also unreliable. In METR's randomised trial of 16 experienced open-source developers working on 246 real issues, AI tools made them 19% slower. The developers had expected a 24% speed-up, and afterwards still believed AI had made them 20% faster (METR, 2025). The NANDA success figures are self-reported too, which is one more reason to measure your own baseline instead of borrowing anyone's rate.
Why do AI pilots fail, according to the research?
The headline is weak, but the qualitative findings match other research.
From the MIT NANDA report:
The "learning gap". Users said tools do not learn from feedback, need too much context each time, cannot be customised to their workflow, and break in edge cases.
Simple work versus complex work. 70% of respondents preferred AI for drafting emails and 65% for basic analysis, but for complex, multi-week work humans were preferred by 9 to 1.
Shadow AI. Only 40% of companies said they had bought an official LLM subscription, yet workers at over 90% of surveyed companies reported regular use of personal AI tools for work.
Buy versus build. External partnerships reached deployment about 67% of the time versus about 33% for internal builds. The report notes these are self-reported and that the correlation does not prove causation.
Speed. Top mid-market performers reported about 90 days from pilot to full implementation; large enterprises took nine months or longer.
From RAND's 65 interviews, five root causes: the team misunderstood or miscommunicated the problem; the organisation lacked the data; it chased technology instead of the user's problem; it lacked infrastructure to deploy and manage models; or the problem was too hard for AI (RAND, 2024).
Write the pilot plan before you expand an AI feature or coding assistant to more users. This seven-point checklist keeps the decision about your evidence, not someone else's statistic.
User and job. Who is doing what task, and what is difficult about the current workflow?
Baseline. How does the task work today? Record time, completion, errors, rework or another relevant measure.
Success outcome. What user or business outcome should improve? Set a threshold before the pilot.
Quality boundary. What makes an answer or action acceptable? Include representative examples, edge cases and known failure modes.
Fallback and ownership. What happens when the model is uncertain or wrong? Name the person who can review, override or stop the workflow. The guide to bounded AI decisions shows how to write that handoff.
Cost and operating limits. Track latency, model cost, review effort, escalation and support load alongside task outcomes.
Decision date. Agree when the team will scale, revise or stop the pilot, and who owns that call.
Measure in layers; each answers a different question:
Whether people can rely on the output for this task
Delivery
Cycle time, review effort, rework, escaped defects
Whether the work moves through the team differently
Product impact
Task completion, repeat use, retention, revenue or cost to serve
Whether the change matters to users or the business
Operating cost
Latency, model cost, human review, support load
Whether the value is sustainable
Google Cloud's framework for AI coding assistants, co-written by DORA lead Nathen Harvey, uses a similar ladder: adoption, trust, acceleration and impact. It warns against jumping from adoption straight to business impact, and suggests at least 6 to 8 weeks, or four two-week sprints, before acceleration may show (Google Cloud, 2025). Accepted suggestions are an early signal, not proof that customers got more value. The AI evaluation framework for product teams covers the quality layer in detail.
Worked example: AI-generated acceptance criteria
This example is illustrative, not a customer result. Say a 6-person team pilots AI-drafted acceptance criteria for tickets. Counting drafts would tell it nothing, so it writes the plan first:
Baseline (last 20 tickets): 3 days from request to approved brief; reviewers catch 4 missing edge cases per 10 tickets; 2 in 10 tickets need engineering rework after kick-off.
Threshold: approved brief in 2 days or less, with rework no higher than 2 in 10.
Quality boundary: no invented requirements; every criterion traceable to the request or a linked customer note.
Cost limit: reviewer time per brief must not rise above 30 minutes.
Decision date: after 8 weeks or 40 tickets, whichever comes first.
If briefs arrive in a day but rework climbs to 4 in 10, the pilot failed on the outcome that matters, however fast the drafts were. If drafts take as long but rework halves, it may be worth scaling.
Common mistakes when reading AI pilot statistics
Treating a survey of organisations as a rate for your project. Your odds depend on your problem, data and workflow, not a 52-company average.
Counting activity as success. Prompts sent, drafts generated and suggestions accepted show adoption, not value.
Trusting how fast it feels. METR's developers felt 20% faster while measuring 19% slower. Time the work.
Skipping the baseline. Without "before" numbers, no "after" number means anything.
Leaving the end date open. Pilots without a decision date tend to drift on as permanent experiments.
The checklist has limits. It will not rescue a pilot aimed at a problem users do not have, and some benefits, such as better decisions, take longer than eight weeks to show; record those as open questions. For who owns the final call, see what product managers should never delegate to AI and the AI product manager guide.
Where Specky fits
Specky reads customer calls, feedback, Slack, Jira and analytics into a Product Graph, drafts specs with the supporting evidence cited inline, and checks after launch whether the bet worked. For an AI pilot, that means the baseline, the threshold and the decision date live next to the spec instead of in a slide nobody reopens. Your own measurements still decide whether the pilot succeeded.
Frequently asked questions
Is it true that 95% of AI pilots fail?
No. The MIT NANDA report says 95% of surveyed organisations saw no measurable P&L return from generative AI. Its own funnel shows that only 20% of organisations piloted a task-specific tool and 5% reached production, so about a quarter of piloting firms succeeded. The sample was 52 interviews and 153 survey responses.
Who wrote the MIT study on AI pilots?
Aditya Challapally, Chris Pease, Ramesh Raskar and Pradyumna Chari wrote The GenAI Divide: State of AI in Business 2025, published in July 2025 with MIT Media Lab's Project NANDA. It presents preliminary findings from January to June 2025 and was not peer-reviewed.
What counts as a failed AI pilot?
The MIT report counted a pilot as successful only if users or executives described a marked and sustained productivity or P&L impact within six months. Other definitions include cancellation, low adoption or harmful errors. Define failure for your own pilot before it starts, with a measurable threshold and a date.
Why do most AI pilots fail?
Research points mostly to product causes: unclear problems, poor fit with the workflow, missing data, tools that do not learn from feedback, and no agreed measure of success. RAND's interviews with 65 practitioners found the most common root cause was misunderstanding or miscommunicating the problem AI was meant to solve.
How long should an AI pilot run?
Long enough to compare against a baseline on the outcome you care about. Google Cloud suggests at least 6 to 8 weeks before productivity gains from AI coding assistants may appear. Set the decision date and sample size, such as 8 weeks or 40 tasks, before the pilot begins.
GDPR-aligned. EU workspace data is stored in Frankfurt, Germany. GDPR and EU data Built in Vienna, Austria. A product of influence solutions fm GmbH (FN 675374y). Imprint 14-day trial. Card required, and nothing is charged until day 14. 30-day refund after subscribing. See pricing