Why your AI pilot never reached production
Nine deployments in, the pattern is consistent: the pilot succeeded on the modelling and failed on everything around it.
Le Quang Vinh
Head of Applied AI
· 8 min read
The pilot demo impressed the board. Eighteen months later it is still a pilot. This is the most common conversation we have, and the cause is almost never the model.
There was no evaluation harness
A pilot judged by "the team tried it and liked it" cannot be judged again after a change. Without a labelled set from your own data and an automated evaluation, nobody can say whether version two is better than version one, so nobody will approve it for production.
Build the harness before the model. It is unglamorous and it is the thing that gets you to production.
“Build the evaluation harness before the model. It is the thing that gets you to production.”
The output had nowhere to go
A model that produces a result in a notebook has not changed a process. The output has to land in the ERP, the ticket queue, or the document system that people already use, which is an integration problem with an integration budget.
Nobody owned it after the pilot team disbanded
Production means monitoring, retraining triggers, a rollback path, and an on-call rota. If none of those were scoped, the pilot was a research project that was mislabelled.
Le Quang Vinh
Head of Applied AI
Writes about the work as it actually goes, including the parts that reflect badly on us.
