A model that scores 94% offline can still ship a product your users hate.
That gap is the whole job. Two teams I have watched up close, a 20-person AI creative team at Pixis and the data platform under InMobi’s ad exchange, both learned it the expensive way: the metric that looks good in a notebook and the metric that matters to a customer are not automatically the same number.
This post is the delivery playbook for the space between those two metrics: what changes in planning, in evals, in rollout, and in how you work with your ML engineers when the system you are shipping is never going to be perfect.
So that’s today’s post.
Today’s Post
Six habits that separate a PM who ships AI features from one who ships AI demos.
- Why waterfall plans rot and experiment-first delivery is the honest default
- Evaluation metrics, and why they are where product intent becomes math
- Prompts and model versions as product assets you manage like code
- Build versus buy, and the question that actually decides it
- Latency, cost and failure modes as the real user experience
- The cross-functional rituals that decide whether any of this survives contact with users
1. Waterfall plans rot, run experiments instead
Traditional software lets you sequence design, build, test and launch with dates that mostly hold.
ML work does not. The unknowns hide inside the data and the real-world distribution the model has to survive, and you only find them by training, evaluating and failing in controlled ways.
That does not mean chaos. It means the unit of planning changes from a feature milestone to hypothesis, experiment, decision: a bounded spike, a labeled eval set, an offline benchmark, a shadow run, a gradual rollout.
Pixis’s AdRoom team ran exactly that shape without calling it a research process. Every new brand went through a two-week onboarding sprint with one hard goal: get the brand self-serving live, on-brand ads inside the window. The team instrumented activation, meaning a published on-brand ad, and ignored logins on purpose. Four automotive and consumer-tech brands later, the sprint had produced $800K ARR in two months, and the full motion reached $0 to $10M ARR in five months with a 20-person team.
Product implication: your roadmap should show learning milestones next to feature milestones. “Model beats baseline on held-out brands” is a legitimate line item even when the UI has not moved.
2. Evaluation metrics are where product intent becomes math
Your engineers will talk about loss, BLEU, perplexity, AUC, F1. Your job is to connect each one to something a user actually feels.
Ask two questions of every metric on the table. Which one, if it improved, would make the experience meaningfully better? And which one could improve while the product gets worse anyway, because latency crept up, cost climbed, or false positives started annoying people?
Pixis’s Advance platform is the guardrail version of this done right. Before any AI layer shipped, a 12-person pod set a hard freshness target and treated data delay as a feature with a number attached: 4 hours down to under 15 minutes, at 99.6% end-to-end sync reliability across 1M-plus keywords a day for 400-plus enterprise customers. None of that demoed well. All of it is why the recommendations sitting on top of it were worth acting on.
Product implication: co-own a small metric stack with your ML lead. Offline metrics for fast iteration. Online metrics tied to real behavior. Guardrail metrics you refuse to regress, latency, cost, toxicity rate, support volume. A team that optimizes only the offline number eventually ships something users hate or the company cannot afford.
3. Prompts and model versions are product assets
For an LLM feature, the prompt is not trivia. It is behavior. A small wording change can shift tone, safety and how often the model invents a fact.
You need versioning, change logs and access controls sized to the risk, plus a practice for regression testing whenever a provider quietly updates the base model underneath you.
Pixis learned this from the other direction, on generated creative rather than generated text. AdRoom’s editor let a marketer skip the designer and the manual QA gate entirely, and that was only safe because of one unglamorous decision: enforce brand guidelines, fonts, color, logo rules, voice, at generation time, not after. The model could not violate a constraint it never saw as optional. That single decision is the reason designer workload dropped 80% without a wave of off-brand ads following it out the door.
Product implication: ask how prompt and model changes get reviewed, how last week’s behavior gets reproduced on demand, and what the plan is for the day a vendor deprecates the model you built around. “We’ll handle it” is not an answer, it is a request for a follow-up meeting.
4. Build versus buy is a bet on differentiation, data and pace
Buy an API or a managed model when speed to market beats owning the weights, when your edge is workflow or data rather than model architecture, or when you do not have the team to train and serve at scale.
Build, custom training, fine-tuning, your own serving, when the domain performance gap is large and persistent in your own evals, when unit economics at your volume demand it, or when you sit on proprietary data an off-the-shelf model cannot see.
Pixis’s answer was a hybrid, and the order it happened in is the lesson. The AI creative and optimization layers, AdRoom and Advance, are where the differentiation lives, so that is where the team built. The plumbing under them, Stripe billing, the MMP and third-party measurement evaluation, the Google, Meta and Amazon connectors, got integrated and stood up rather than reinvented. The 12-person data platform pod came first, specifically because a fast model sitting on late or wrong data is not a differentiator, it is a liability with a nicer UI.
Product implication: force a total-cost-of-ownership conversation early. Look past the sticker price per token to engineering time, review operations and failure handling end to end. The cheapest API stops being cheap the moment it needs an army of human reviewers behind it.
5. Latency, cost and failure modes are the user experience
A brilliant model that answers in eight seconds loses to a good model that answers in eight hundred milliseconds, for anything interactive. A feature that burns margin can win a demo quarter and die in a finance review the next one.
Users will also hit low-confidence outputs, invented specifics, off-brand language, and tool-call errors in anything agentic. Your product needs a defined answer for each: suppression, fallback, human routing, safe defaults, plain language about what just happened.
The rollout is where all of this either gets caught early or gets caught by a customer. Shadow mode runs the new system in parallel without touching what users see, so you can measure disagreement with the old system and calibrate thresholds before anyone is exposed. A gradual rollout by percentage, cohort or region contains the blast radius if you got it wrong. Rollback has to be real: feature flags, a pinned model version, and one named owner who can revert without convening a committee.
Product implication: write rollout steps as acceptance criteria. “Launch” is not a date on a calendar, it is a sequence with exit conditions, and “done” for an AI feature means risk-adjusted, not checkbox-complete: performance under load, monitoring live, a rollback path defined, and everyone aligned on which residual failures are acceptable.
6. Cross-functional rituals decide whether any of this survives
Design, content design, legal, support and sales need to see realistic outputs before launch, not curated winners. The earlier they see a failure, the better the guardrail they build for it.
Pixis’s Prism launch is the concrete version of that discipline. The team ran 15-plus voice-of-customer interviews, every launch, and turned them into exactly five roadmap features, not fifty. The hard part was not gathering the input, it was declining the other forty ideas the interviews surfaced. That only works when a leader treats a killed idea as evidence and evidence as a win, and it is the reason the GTM around Prism moved trial-to-paid up 25% and cut onboarding time 40% instead of shipping a co-pilot nobody finished setting up.
The durable product underneath every one of these habits is a system, not a model: model plus data plus prompts plus tools plus UX plus monitoring plus policy plus operations. InMobi’s system of record under a demand-side platform and ad exchange processes 200 million-plus devices’ worth of ad events a day, and the month-end revenue close that runs on top of it went from 5 days to 12 hours. Nobody outside the finance team ever sees the model behind that number. They see the close.
Product implication: schedule red-team sessions with genuinely messy inputs before launch, not after. If the room is uncomfortable, you are doing it correctly, and a single source of truth that a new engineer can read in an hour is cheaper than the outage that happens without one.
Takeaways
Six habits, one per section. Pick the AI feature closest to shipping on your roadmap and check it against all six before the launch review, not after.
Final Words
You are not shipping a model. You are shipping model, data, prompts, tools, UX, monitoring, policy and operations, all at once, and the PM’s job is to keep that whole stack honest with the user rather than just impressive in a demo.
Pick one AI feature on your roadmap this week and run it through sections 1 through 6 with your team, out loud.
Cheers, Hoshank