The vendor’s own model list is marketing copy, not a contract.
I learned that the expensive way while building this site’s own AI assistant. Google’s Gemini API has a GET /models endpoint that lists every model you could theoretically call. Two of the models on that list return a bare 404 the moment you actually call them. A third is marked available but is, in Google’s own words, “no longer available to new users.” The listing and the product are not the same thing, and nothing in the API tells you which model you are looking at until you try it.
I own the Gemini integration behind this site’s Ask AI feature and its realtime voice mode, both server-proxied so the API key never reaches a browser. Getting that integration to actually work, not just demo, is the same evaluation problem any PM faces when a vendor pitches a model: capability, cost, latency, reliability, lock-in, and whether your evaluation process catches the gap before your users do.
So that’s today’s post.
Today’s Post
Six checks a vendor pitch has to survive before it becomes a product dependency.
- Why capability is fit to your task, not a benchmark score
- Why the model catalog you’re shown is an advertisement, not an entitlement list
- Why cost, latency and reliability are one bill
- Why lock-in gets mitigated by an abstraction boundary, not a vendor promise
- Why evaluation has to be a regression habit, not a one-time RFP
- What a decision memo and its kill criteria actually buy you
1. Capability is fit to your task, not a leaderboard score
Start with a blunt question: does the model reliably produce outputs good enough for your task, under your constraints. Not the benchmark’s constraints. Yours.
I run two model families behind Ask AI: Gemini’s gemini-3.1-flash-lite as the default, and Google’s Gemma 4 models (gemma-4-31b-it, gemma-4-26b-a4b-it) as alternates. Both families answer the same retrieval-augmented question reasonably well on paper. Only one of them has a landmine underneath.
Gemma 4 always emits a reasoning block in its response, a “thought” part, and it rejects the thinkingConfig setting that would normally suppress that. Gemini 3 doesn’t emit that block at all. If you don’t filter thought: true parts out of every response parser, Gemma’s scratchpad prints as the answer, in front of the user, every time. That is not a benchmark failure. It’s a behavioral quirk a leaderboard score would never surface, and it only shows up when you run your own inputs through your own code.
The lesson generalizes past this one API. Translate capability into product language before you sign anything:
- Task success rate. What share of real interactions meet your bar without a human fixing them up.
- Failure shape. When the model fails, is it obviously wrong, subtly wrong, or confidently wrong. Confidently wrong is the expensive one; Gemma’s leaking scratchpad is a version of it.
- Coverage. Does it hold up for the segments you actually serve, not the segment in the vendor’s demo.
If you can’t write your acceptance criteria before you test, you’re not ready to evaluate the vendor. You’re ready to be impressed by them, which is a different thing.
2. The model catalog is an advertisement, not an entitlement list
Every provider will hand you a list of models you can call. Treat that list the way you’d treat a restaurant menu during a supply shortage: it’s an aspiration, not an inventory count.
Gemini’s GET /models for this site’s API key lists gemini-2.5-flash (Google’s own note: “no longer available to new users”), gemini-3.5-flash, and gemini-3.6-flash. Calling either of the last two returns a bare 404 with an empty body. Only gemini-3.1-flash-lite, gemma-4-31b-it, and gemma-4-26b-a4b-it were verified live as of 11 Aug 2026, by hand, by actually calling each one. NVIDIA’s NIM catalog sets the identical trap on its nvidia/llama-3.3-nemotron-super-49b-v1 listing. So the working model set for both providers lives in an allowlist this site maintains itself, not in whatever the vendor’s endpoint reports.
There’s a second version-pinning trap sitting right behind the first one. Gemini’s REST calls have to hit v1, not v1beta. Gemini 3 models don’t exist on v1beta at all: the identical request that 404s on v1beta answers normally on v1. Gemma and the embedding model work on both, which is exactly the kind of inconsistency that makes a team assume the whole API surface behaves one way when only part of it does. The realtime voice websocket is a separate surface again and stays on v1beta deliberately, because that’s the surface Google ships it on.
None of this is a knock on Google specifically. It’s what every model API looks like once you stop reading the marketing page and start reading the response bodies. Read the status history and the actual response codes, not the slide.
3. Cost, latency and reliability are one bill
Vendors quote a clean number, per token or per seat. Your users experience a bundle: how fast the first token arrives, whether the response streams smoothly, and whether the system is even up when they ask.
This site’s free tier for Gemini runs at 15 requests per minute, shared across every visitor on the site at once, not per user. That’s not a hypothetical constraint, it’s the ceiling the whole feature lives under, and it shapes everything from how eagerly Ask AI prefetches to whether a second visitor gets rate-limited by a first one’s long session.
Latency shows up as a measured number too, not a guess. Ask AI’s chat page currently ships around 10MB to a phone: a 4.1MB ONNX wasm runtime, all twelve embedding text shards eagerly, a 1.99MB WebLLM bundle, and a second reranker on top. The measured Interaction to Next Paint on that page is 7,256ms. That number is the actual user experience, and it is on this site’s own backlog to fix, which is the honest version of a latency section: you publish the number you measured, not the number you hoped for.
Reliability is where the fallback design earns its keep. If NVIDIA_API_KEY is unset, the NVIDIA provider doesn’t error, it simply disappears from the model picker. That’s a design decision, not an accident: a missing dependency should degrade the feature, not crash it. Every environment variable in this site’s schema is optional for exactly that reason.
4. Lock-in gets mitigated by an abstraction, not a promise
Lock-in shows up as proprietary formats, orchestration glued to one vendor’s SDK, and workflows that quietly assume one provider’s quirks. You don’t eliminate it. You decide, in advance, where the boundary sits that lets you swap without a rewrite.
This site’s boundary is one file: gemini-catalog.ts. The API version, the endpoint, the default model ID, and the verified-live allowlist all live there, and the serverless route, the browser RAG client, and the completion helper all read from it. A model the picker offers is a model the proxy actually accepts, because there’s exactly one place that mapping can drift. Swapping a default model is a one-file change instead of a grep across six files hoping you found every call site.
Pixis made the same bet at the data layer, before any model was involved. While the AI layers were shipping, my team ran a separate MMP and third-party integration exploration, evaluating measurement partners including AppsFlyer and peers, before wiring the Google, Meta, Amazon, and GA connectors the rest of the product feeds on. That evaluation happened before the connectors were built, not after, because unwinding a data-gravity dependency after 400+ enterprise customers depend on it costs a different order of magnitude than deciding up front.
The Anthropic product-experiments portfolio I’ve been building applies the same discipline one level earlier, before any code ships at all: six draft experiment specs, each scored for reach, leverage, and confidence, with a stated run order, before committing engineering time to any one of them. Only one of the six, an Atlassian sync MCP slice, has shipped as runnable code so far. Workflow infrastructure like that beats raw model capability precisely because it’s the layer that survives a model swap; the CLAUDE.md, the slash commands, and the MCP server don’t care which model is answering.
5. Evaluation is a regression habit, not a one-time RFP
A demo selects for the input the vendor chose. Production selects for the input your actual user typed. The gap between those two is where evaluation earns its cost.
NVIDIA’s model gets a dedicated liveness probe, pnpm probe:nim, that re-measures model availability on a schedule. Gemini’s allowlist has no equivalent yet. It’s verified by hand, and the honest note on that gap, written into this site’s own documentation, is that the allowlist “will rot.” That’s not a hedge, it’s a kill criterion waiting to be written: the day nobody has re-checked the Gemini allowlist in some fixed number of weeks is the day the feature is running on unverified assumptions again.
The same discipline applies to your own test suite, not just the vendor’s model. Twenty of this site’s 175 unit tests are currently red on main, across five files, and the cause in every case is that an assertion drifted from an intentional code change, not that the code broke. One test literally expects a maxOutputTokens value the code stopped sending months ago. That’s the unglamorous version of “regression testing”: someone has to diff the failure list against a clean checkout before assuming a change caused it, every time, or the red tests stop meaning anything.
Build your own labeled set from real inputs. Score models and prompt versions against it the same way every time. Blend automatic checks, like the thought: true filter that has to survive every model swap, with human review for the subjective calls a filter can’t make. A/B tests in production catch what your users actually do; offline evaluation catches the catastrophic failure before it reaches them. You need both, and neither one substitutes for the other.
6. The decision memo and the kill criteria
Executives don’t need thirty rows of model trivia. They need a decision, with the tradeoffs named. A useful memo states the problem, two or three real candidates, a recommendation with a backup, the risks with owners, and a kill criterion: the signal in the first thirty or ninety days that would make the team pivot.
This site already has one kill criterion built into its architecture rather than written down anywhere: if NVIDIA_API_KEY is unset, that provider vanishes from the picker instead of half-working. That’s what a kill criterion looks like when it’s enforced by code instead of by someone remembering to check a dashboard.
Security belongs in the same memo, not a post-signing surprise. GEMINI_API_KEY is server-only, reached through a proxy route so the browser never sees it. PUBLIC_GEMINI_API_KEY, the legacy bring-your-own-key path, is the opposite: anything set there gets inlined into the browser bundle and is visible to anyone who opens dev tools, which is exactly why this site leaves it blank and defaults to the server proxy. ADMIN_TOKEN gates the PostHog visitor dashboard the same way. None of these decisions is exotic. They’re the boring, checkable kind that a real memo should force onto the table before the vendor contract, not after someone finds a key in a bundle six months later.
Takeaways
Six checks, one per section. Run them against whatever a vendor is pitching you this quarter before you write the contract, not after.
Final Words
Pick the model or vendor your team is currently trusting without having re-verified it, and write down the one signal that would make you pivot away from it. If you can’t name that signal in one sentence, you haven’t actually evaluated the vendor. You’ve just liked the demo.
Cheers, Hoshank