Most growth-stage executives evaluating AI vendors don't have the technical depth to ask "are these claims real?" The vendors know it. Polished proposals, confident demos, every benchmark anyone could ask for — and a six-month engagement later, no working feature in production.
These five questions, asked in the first or second call, surface real capability without requiring an engineering background. They aren't gotchas. They're the questions a senior engineer would ask if they were sitting next to you.
1. "Walk me through a project that didn't work and what you learned."
Every experienced AI team has had a project that didn't work the way they planned. Models that didn't produce useful output. Use cases that turned out to be a bad fit. Costs that ran over. If a vendor doesn't have an answer to this question, they're either inexperienced or hiding something. Both are problems.
What you want to hear: a specific story with concrete details — what they tried, why it didn't work, what they'd do differently. The story should make them sound more credible, not less. Real expertise is comfortable acknowledging failure.
2. "What's the one approach you usually try first, and why?"
Bad answer: "We tailor our approach to your specific needs." That's a sales line. Every team has defaults — the things they reach for first, because experience has taught them what usually works.
Good answer: something specific. "We start with prompt engineering plus RAG over the customer's existing documentation, before considering anything fancier. About 70% of the time that's enough, and the 30% where it isn't, we know quickly and can scope the next step." That's an actual operating philosophy.
3. "Who would actually be working on this, and can I meet them?"
Sales calls are run by partners and senior strategists. The people who actually build the system are often more junior — and more variable. You want to meet them before you sign. Ask about their experience with your specific stack and use case. A good vendor will set up the conversation; a bad vendor will keep deflecting.
Bonus question: "Will they still be on the project in six months?" Consultancy turnover is real, and a project that loses its lead engineer in month four often fails.
4. "How will we measure success? What metrics do we agree on now?"
If a vendor doesn't have a sharp answer to this, they don't know what good looks like for your use case. Worse, you'll find yourselves three months in, unsure whether the engagement is working — and the vendor will assure you it is.
Good metrics for AI projects are concrete: "reduce average customer-support ticket resolution time by X%," "convert Y% of inbound documents to structured records with under Z% manual review," "answer N% of internal questions correctly without human escalation." Vague success criteria — "improve user experience," "add AI capabilities" — are how you end up with no project to point to and a six-figure invoice.
5. "What happens to the IP and the data?"
Some vendors retain rights to fine-tuned models or use your data to train shared models that benefit their other clients. Some require you to keep using their inference platform after the engagement ends. Some happily sign over everything.
You need to know which kind you're dealing with before signing. The right answer depends on your situation — sometimes the shared-model arrangement is cheaper and acceptable. But it should be a deliberate choice, not a surprise three months in.
Bonus: the no-team test
Ask: "What would you do if we said we want this built but we don't have an internal engineering team to maintain it?" A good vendor won't agree to that without a clear handoff plan and ongoing support. A vendor who shrugs and says "sure, no problem" is selling you a system that will break six months after they leave.
If you'd like a second opinion on a vendor proposal you're evaluating — vendor-neutral, no commercial agenda — reach out. It's something we do regularly for clients.