Most AI pilots do not fail because the idea was wrong. They fail because nobody built the machinery that lets a probabilistic system run unattended in a workflow that matters. A demo proves the model can do the task once. Production proves it can do the task ten thousand times without a human catching every miss.
We treat the gap between the two as five gates. Each one answers a question someone will eventually ask before letting the system run unattended, and each one is cheap to build early and expensive to retrofit late. Skip any single gate and the pilot stalls — usually not in engineering, but in the review meeting where nobody can answer the question that gate exists to answer.
The five gates
| Gate | The question it answers | What it looks like when it's missing |
|---|---|---|
| Evals | Is it good enough, and did that change make it better? | Every release is an argument about vibes |
| Observability | What did it actually do on this case? | You find out from an angry customer |
| Guardrails | What's the worst thing it can do? | It never clears security or legal review |
| Rollback | What do we do when the model changes under us? | A vendor update silently breaks it |
| Ownership | Whose number is this? | A committee approves it and nobody ships it |
Gate 1 — An eval suite that measures the thing you care about
An eval suite is a fixed set of real cases with known-good outcomes, scored automatically every time anything changes. Not a spot-check, and not does this output look right — a number you can compare against last week's number.
Without one, quality is a matter of opinion. Every prompt tweak becomes a debate, nobody can say whether the change helped, and no one is willing to sign off on a release they can't measure. That's not a model problem; it's the absence of a ruler. You cannot ship what you cannot score — which is why we treat this as the gate that comes before all the others, and why we wrote a whole piece on it.
You've passed this gate when you can swap a model or rewrite a prompt and know within minutes whether quality moved, and in which direction.
Gate 2 — Observability into what the system actually did
Not an aggregate dashboard. Per-run forensics: for any single execution, the input, the context it retrieved, the tools it called, what it returned, what it cost, and how long it took.
Probabilistic systems fail quietly. A deterministic bug throws an error; a model just returns something slightly wrong, confidently, and keeps going. Without traces, those failures are invisible until they accumulate into something a human notices — which is how teams discover a month of bad outputs in a single support escalation.
You've passed this gate when someone can answer why did it do that, on this specific case, last Tuesday in under five minutes.
Gate 3 — Guardrails that bound the worst case
Explicit constraints on what the agent can touch, what it can spend, what it can say, and when it must hand off to a person. Scope limits enforced in code — not instructions in a prompt asking it nicely.
This is the gate most pilots actually die at, and it's rarely engineering that kills them. It's the security review, the legal review, the risk conversation that never resolves because nobody can state the blast radius in a sentence. An agent that probably behaves cannot be granted write access to a production system, and so it stays a demo forever. The fix is to make the worst case small and explicit rather than to argue that it's unlikely.
You've passed this gate when you can state the worst thing the system can do in one sentence, and the person accountable for that risk is comfortable with it.
Gate 4 — A rollback path for when the ground moves
Pinned model versions, versioned prompts and configuration, and the ability to return to a known-good state on demand. The models underneath you are not stable infrastructure — vendors deprecate, retune, and ship behavior changes on their schedule, not yours.
Teams that skip this discover it the same way every time: something that worked for months degrades over a weekend, and there's no previous configuration to fall back to because the current one was edited in place. Reverting turns into re-deriving.
You've passed this gate when rolling back is a deploy, not a rebuild.
Gate 5 — A named owner accountable for the metric
One person. One number. Not a sponsor, not a working group, not a vendor — a named individual whose job is affected by whether the number moves.
This is the least technical gate and the most predictive. Committees are good at approving AI projects and structurally incapable of shipping them: responsibility spread across eight people is responsibility nobody feels at 6pm on a Thursday when the thing needs one more push. We call the failure mode the Steering-Committee Tax, and it is the single most reliable signal we see for a program that will stall.
You've passed this gate when you can name the person and the number without looking anything up.
Why most of it goes wrong in the first 90 days
The failure is usually decided long before anyone notices it — inside the first quarter, in three moves that all feel reasonable at the time.
Month one: the strategy is a list of use cases. A workshop produces fifteen candidate workflows ranked by excitement, and the most impressive one gets picked instead of the most tractable. Nobody asks which of the fifteen has a measurable cost attached to it today, so there's no number to be accountable to later.
Month two: nobody owns the metric. The work has a project sponsor, a vendor, and a steering committee — which is not the same as one named person whose job depends on a number moving. Committees can approve an AI project; they cannot make one work. This is the Steering-Committee Tax, and it is the single most reliable predictor of a stalled program.
Month three: the demo lands and quietly redefines success. It works on the happy path, the room is impressed, and the goal silently shifts from deployed to demonstrated. From here the project doesn't fail so much as never conclude — which is why 95% of enterprise generative-AI pilots deliver no measurable P&L impact (MIT Project NANDA, 2025) and Gartner expects more than 40% of agentic-AI projects to be canceled by the end of 2027 (Gartner, 2025).
The counter-move is unglamorous: pick one workflow that already costs someone money, put a name on the metric, and define done as running unattended in production — not as a demo that impressed a room. That's the whole reason our engagements start with a fixed-price two-week Sprint that ends in something deployed rather than a slide deck. If you're earlier than that, the AI Readiness Assessment is the cheaper first look.
Which gate are you stuck at?
Stalled pilots feel the same from the inside, but the symptom usually points at exactly one missing gate:
- We keep tweaking it and can't tell if it's getting better. — Gate 1. You have no ruler.
- It works, except when it doesn't, and we can't reproduce it. — Gate 2. You have no traces.
- Engineering finished months ago; it's still in review. — Gate 3. Nobody can bound the worst case.
- It used to work and something changed. — Gate 4. You have no known-good state to return to.
- Everyone agrees it's important and nothing moves. — Gate 5. No single person owns the number.
That diagnosis matters more than it sounds, because the gates are cheap in the order above and expensive out of it. Retrofitting evals onto a system already in production means reconstructing ground truth from cases nobody recorded. Adding observability after a failure means investigating something you have no trace of. Almost every expensive AI rescue we're called into is a team paying the retrofit price on a gate that would have cost days at the start.
None of this is exotic. It is the same discipline that turned web demos into reliable software a decade ago, applied to a stack that happens to be non-deterministic. The companies stuck in pilot purgatory are not short on ambition. They are short on the gates — which is exactly what a Gigabit Agents build is built to install.
