Three weeks ago, in a product sync, one of our engineers asked: "Should we go GPT or Claude for this feature?"
Everyone looked at each other. I looked at them.
The answer wasn't "which is better?" It should have been "better for what?" Nobody in that room asked that question.
In 2022, this wasn't a real decision. One option dominated. Now there's Claude 4, GPT-5, Gemini 2.5 Pro, Llama 4, Mistral, and dozens more — each with different benchmark scores, pricing tiers, API behavior, and distinct output personalities.
And this choice is no longer purely technical.
Who actually owns this decision?
Engineers handle integration. But which model you pick directly shapes:
- How long users wait
- Your cost per interaction — which hits your margins
- What users actually see: output quality, tone, consistency
That's not infrastructure. That's the product.
What benchmarks don't tell you
"MMLU: 92.3" means nothing to a user waiting for a clinical summary. What they care about: Did the system ask the right question? Is the output actually useful? Does it feel off?
Last February, we tested two models for a specific workflow. Model A scored 8% higher on standard benchmarks. Model B triggered 30% fewer error responses in real user testing and felt noticeably more natural.
We went with Model B. Benchmarks gave us the wrong answer. Real users corrected it.
Lock-in risk nobody prices in
You integrate with Model X today. Six months later, they raise prices 40% or change how the API returns structured data. What then?
SaaS teams know vendor lock-in. But with AI models, there's an extra layer.
When you switch models, it's not just a config change. Your prompts need rewriting. All your output quality tests need to run again. Users' mental model of "how the AI behaves" gets reset.
Model switching is a product revamp. Not a migration.
The persona problem
Every model has a distinct voice — how verbose, how formal, how creative, how cautious. These vary significantly across providers.
If your product is supposed to feel clean and precise, but your chosen model produces long, enthusiastic, slightly meandering responses — there's a mismatch. Users feel it before they can name it. They just say something feels off.
This is exactly the kind of decision that goes wrong when PMs hand it entirely to engineers.
The cost model — the hidden variable
A PM I know added an AI summarization feature to their product. They picked a capable, premium model. First month: that single feature consumed 40% of total infrastructure costs.
Nobody had forecasted it. Because the pricing model lived in a technical spreadsheet, not in the product planning process.
When model selection gets treated as an engineering problem, the economic consequences still land on the PM — just later, when options are narrower.
The wrong question and the right one
Wrong: "Which model is best?"
Right: "What does our user need in this specific task, and which model delivers it most reliably — at a cost structure that actually makes sense?"
Sometimes that's the cheaper, faster, slightly-less-capable model. Sometimes it's the slower, more precise one. Sometimes you run two different models in the same product — one for retrieval, one for generation.
Yes, that's real. More teams do it than admit it.
What PMs should actually do
Treat model selection as a product decision, not a delegated technical detail. Put it on your roadmap. Understand the cost model before launch, not after. Test persona fit yourself. Talk about lock-in risk before you're locked in.
This conversation will come back every six months. You might as well own it from the start.