Blog

Thoughtbubble Blog

How to Spot Realistic AI Promises vs Vaporware

Judge AI vendor claims with three criteria: technical feasibility, historical precedent and practical utility, before any contract is signed.

An AI promise is realistic when the vendor can explain how the system works, point to something comparable already in production, and name the workflow it improves. Vaporware, in a business context, is a capability that is sold or roadmapped but never delivered as described. A structured AI vendor evaluation built on three criteria, technical feasibility, historical precedent and practical utility, lets you separate the two without reading a line of code.

The executive dilemma in the AI era

A professional woman in a dark blazer sits at a wooden boardroom table looking at a notebook.

Executives are now asked to approve budgets for systems they cannot inspect personally. The vendor knows whether the demo was rehearsed, whether the data was curated and whether the accuracy figure came from a lab test or live use; you do not. That information gap, not the technology itself, is the procurement risk. The fix is procedural: treat every claim as a hypothesis and score it against the three criteria below before money moves.

Criterion 1: assess technical feasibility

Feasibility questions are generic by design. You are not asking how the model works internally; you are asking how its performance was measured. Request the headline accuracy figure and the method behind it. Was it measured on the vendor's curated examples or on data like yours, in a demo environment or after deployment? Every capable AI system has an error rate, so a vendor who cannot name one, or who answers with adjectives instead of numbers, is describing ambition rather than a product.

Ask what the system needs to run: how much historical data, in what format, how clean, and who does the cleanup. Preparation work rarely appears in the demo, and it lands on your budget. Then ask what happens on a bad day: when the output is wrong, who notices, how quickly, and what that failure costs. Feasible claims arrive with limits stated plainly. A pitch with no stated limits has undisclosed limits, not absent ones.

Criterion 2: look for historical precedent

Precedent is the executive's strongest tool, because judging evidence is already your skill. Ask for the closest thing to this system that is running in production today, ideally for an organisation like yours. Then test how verifiable each example is. A usable case study names the client or at least the industry, the scope of the deployment, roughly when it went live and the outcome in numbers. A logo wall with an anonymous quote is marketing, not precedent.

Run the test in the other direction too. If the claim has no precedent anywhere, treat it as a research project and price it accordingly, not as routine procurement. Check that any precedent transfers: a capability proven on text summarisation does not automatically survive a regulated, high-stakes domain. Note any resistance to supplying references; enthusiasm about the future with vagueness about the past is the classic vaporware signature.

Criterion 3: measure practical utility

A closed leather planner and a pen sit on a white desk next to a mug.

Utility is decided before the meeting. Write down the specific workflow the tool should improve and the number that should move, whether hours per week, correction effort or response time. A pitch that cannot be tied to one named workflow is a novelty, whatever the demo suggests. Some problems are better served by conventional software; ask whether the task truly involves judgement or unstructured input, and for stable, rule-based processes, choosing between AI and traditional automation is the more productive question.

Then test integration, because value disappears in handoffs. Ask who uses the output, where it lands and what changes in that person's day. Ask for a time-boxed pilot on your own data with a success threshold agreed in advance, the same discipline behind testing an AI workflow before building the full app. A return you cannot specify before purchase is one you cannot verify after it.

Questions for your next AI vendor evaluation meeting

Two men in professional attire sit across from each other in a glass-walled conference room.

These questions carry the three criteria into the room. Ask them in order; a vendor who stalls on feasibility has already saved you the rest of the meeting.

  • What accuracy figure do you quote, how was it measured, and on whose data?
  • What does the system need from us: data volume, format and cleanup, and who does that work?
  • What is the error rate, and what does one wrong output cost our team?
  • Where is the closest comparable deployment running today, and can we speak with that customer?
  • How long did that deployment take to reach production?
  • Which of our named workflows will this change, and which metric will move?
  • What would a four-week pilot on our data prove, and against what threshold?

Score the answers, not the demo.

From hype to implementation

Discipline beats intuition in AI procurement: a cautious review costs a few meetings, while an unexamined claim can cost months of integration effort. Keep the three tests attached to every claim: is it feasible, is it precedented, is it useful. Where a pitch passes but the stakes are high, isolate the single riskiest assumption and validate it with a small build before committing, the approach behind the riskiest assumption framework. That converts the vendor's promise into your own evidence. When the answers hold up, run the pilot on your terms. Bring your AI vendor evaluation to Thoughtbubble.