Most organisations now have at least one generative AI pilot. Far fewer have one running in production, used every day by real people, with a cost and risk profile the business is comfortable with. The gap between the two is mostly engineering, not modelling.
Here's what changes when a pilot becomes a product.
1. "It looks good" becomes "we can prove it's good"
Pilots are judged by demos. Production systems need evidence. Before scaling, build an evaluation set: a few hundred real questions or tasks with answers your subject-matter experts agree are correct. Run every change to prompts, models or retrieval against it automatically, and track the results over time.
If you can't say how accurate your assistant was last week, you aren't ready to put it in front of customers.
2. Retrieval gets serious
Most enterprise assistants depend on retrieval-augmented generation: finding the right internal content and giving it to the model. In a pilot, someone hand-picks a tidy set of documents. In production you need pipelines that keep content current, respect document permissions, handle tables and scanned files, and remove content that's been withdrawn.
That's a data engineering problem, and it's usually the biggest piece of work.
3. Guardrails move from slides to code
- Input checks to detect prompt injection and requests that are out of scope.
- Output checks for sensitive data, unsupported claims and tone.
- Access control so users only receive answers drawn from content they're allowed to see.
- Human review for high-stakes actions, especially when agents can change systems.
4. Cost becomes a design constraint
A pilot with twenty users rarely worries about token costs. Ten thousand users will. Caching, smaller models for simpler tasks, tighter context and usage monitoring can cut running costs dramatically without hurting quality. Measure cost per useful answer, not just cost per call.
5. Someone has to own it
Production AI needs an owner in the business and a team that can operate it: watching quality, handling feedback, updating content and approving changes. Agree this before launch, not after the first incident.
A practical checklist
- An agreed evaluation set and a target score.
- Automated content pipelines with permissions enforced.
- Input and output guardrails, tested and logged.
- Cost and usage monitoring with alerts.
- A named business owner and a support model.
- Sign-off from privacy, security and risk.
If you'd like help taking a pilot to production, see our AI and machine learning services or book a consultation.