The fastest way to ruin a good product is to bolt AI onto it carelessly. The second fastest is to never add it at all. After integrating LLMs into production systems for several years, we've settled on a playbook that avoids both failure modes.
Start with a workflow, not a chatbot
The default instinct is "add a chat box". Resist it. Chat is an interface of last resort - it puts the burden of knowing what to ask on your user. Instead, find one workflow where users already do repetitive cognitive work: summarising, categorising, drafting, extracting. Automate that one thing end to end.
- Good first features: draft replies, auto-tagging, document summaries, smart search
- Bad first features: open-ended chat, "ask me anything" agents, anything customer-facing without review
Wrap the model, don't expose it
Your product should never pass raw model output to a user or a database without a validation layer. We treat the LLM like any other unreliable external API: schema-validate every response, set timeouts, define fallbacks, and log everything. If the model returns malformed output, the user sees your graceful fallback - not a stack trace.
Control costs from day one
Token costs compound quietly. Cache aggressively (identical inputs should never hit the API twice), truncate context to what the task needs, and route simple tasks to smaller models. A two-tier routing setup - small model first, big model only on low confidence - typically cuts spend by half without a visible quality drop.
Measure before you celebrate
Ship behind a flag to a small cohort. Track acceptance rate (how often users keep the AI's output), not just usage. If people trigger the feature but discard the result, you have a demo, not a feature.