Vendor Evaluation · 4 min

How to hand over an AI system — to a new vendor or in-house

The inventory to collect, the acceptance test that proves the handover worked, and the four things that always go missing. Do this properly once and you never repeat the vendor reset.

G

A handover has worked when the receiving team can break the system and fix it without calling anyone. That's the bar, and almost every handover that gets called complete falls short of it — which is why companies end up rebuilding a working system rather than inheriting it.

We've written about the pattern this creates in Vendor Roulette: each engagement ends leaving nothing behind, so each new team starts from a blank page. That piece names what should survive. This one is how to actually collect it.

The inventory

Everything below should be in *your* accounts and *your* repository before the outgoing team's last day — not shared from theirs, not promised at handover.

AssetWhy it's the one that matters
Application codeTable stakes. Rarely the thing that's missing
Prompts, versionedThe accumulated tuning. Often lives outside the repo
Eval suite + test dataWithout it the new team can't tell if they've made things worse
Data pipelinesThe unglamorous majority of the real work
Infrastructure configAs code, not as someone's console clicks
Credentials and accountsIn your name, with your billing, before the last day
Decision logWhy this approach, what was tried and abandoned
RunbookSo somebody can be on call from day one

The two that go missing most often are the eval suite and the decision log, and they're the two that cost the most. Without evals, the incoming team has no way to know whether their first change improved or degraded the system — so they either freeze it or break it. Without a decision log, they re-derive the same choices, frequently re-making an option the last team tried and abandoned for a reason nobody wrote down.

The acceptance test

Don't accept a handover on a walkthrough. Accept it on a demonstration, with the outgoing team present but silent:

  • Reproduce the baseline. The new team runs the eval suite from a clean checkout and gets the documented score. If they can't, the environment isn't actually transferred.
  • Ship a trivial change. A prompt tweak, through the full pipeline to production, and roll it back. This proves the deploy path, the versioning, and the rollback are real rather than described.
  • Diagnose a seeded failure. The outgoing team breaks something deliberately; the new team finds it from traces alone. This is the one that exposes missing observability.
  • Answer three "why" questions. Why this model, why that human checkpoint, why this workflow was scoped the way it was — from the decision log, not from memory in the room.

Pass all four and the handover is real. Anything less and you have a document transfer.

Build in an overlap, and make it work

Two weeks of paid overlap where the new team operates and the old team advises — not the reverse. If the outgoing team is still driving, nothing has transferred; you've just extended the engagement.

The most valuable thing to spend that overlap on is a real incident. If one doesn't occur naturally, seed one. Nothing surfaces an incomplete handover faster than something genuinely being wrong while the people who built it are still reachable.

Taking it in-house instead

Same inventory, one extra question: who is on call? Teams routinely underestimate this because the build looks like the hard part. It isn't — the build ends, and operations don't. Before committing, confirm someone owns the eval baseline, someone watches model deprecations, and someone is genuinely reachable when quality drops, because it will.

In-house makes clear sense when AI is core to your product and you can hire senior AI engineers — we lay out the tradeoff in build vs. buy. It makes less sense when you're absorbing one production system into a team with no capacity to operate it, which is how systems quietly stop being maintained without anyone deciding to stop maintaining them.

Ask for this before you sign, not at the end

Every item above is dramatically cheaper to obtain as a contractual condition at kickoff than as a request during offboarding, when goodwill is lowest and the team is already reassigned. Put the inventory and the four-part acceptance test in the agreement — it's one of the nine questions worth asking any vendor, and the answer tells you a lot about the engagement you're about to enter.

If the answer is that you'd rather not hold the pager at all, Managed AI Operations is the alternative — but even then, insist the inventory lands in your accounts. Being able to leave is what makes staying a decision rather than a condition.

Vendor Evaluation · FAQ

Questions this raises

What should be included in an AI system handover?

Application code, versioned prompts, the eval suite and its test data, data pipelines, infrastructure config as code, credentials and accounts in your own name, a decision log explaining what was tried and abandoned, and an operations runbook. All of it in your repository and accounts before the outgoing team's last day.

How do I know an AI handover actually worked?

Test it rather than accepting a walkthrough. The receiving team should reproduce the documented eval baseline from a clean checkout, ship and roll back a trivial change through the full pipeline, diagnose a deliberately seeded failure from traces alone, and answer three 'why' questions from the decision log. Pass all four and it's real.

What usually goes missing in an AI vendor handover?

The eval suite and the decision log. Without evals the incoming team can't tell whether a change improved or degraded the system, so they either freeze it or break it. Without a decision log they re-derive past choices and often re-make an option the previous team abandoned for an unrecorded reason.

Should we take our AI system in-house?

It makes sense when AI is core to your product and you can hire senior AI engineers. It makes less sense when you're absorbing one production system into a team with no capacity to operate it. The question people underestimate isn't whether they can maintain the code — it's who is on call when quality drops.

Keep reading

Related insights

Vendor Evaluation

Vendor Roulette: why switching AI vendors keeps you at square one

Third vendor, same starting line. Why each switch resets you to zero, what the previous engagement was suppo…

AI Agents

What an AI operations runbook actually contains

Seven entries, each answering a question someone will ask at an inconvenient hour. The document that turns "…

Vendor Evaluation

Nine questions to ask any AI agency before you sign

The questions that separate a firm that ships from one that demos — what each is really testing, and what a …

Stop reading, start shipping

Put a forward-deployed team on it.

If this is the kind of work you're trying to get into production, a 30-minute discovery call is the fastest path to a scoped plan.