The single most useful thing to understand about an LLM is that it produces plausible language, and plausibility is not the same as accuracy. Everything about deploying one safely follows from that.
What it is genuinely good at
- Understanding messy input. Turning a rambling enquiry into structured fields is something rules-based software has always been poor at.
- Summarising. Long call, long thread, long document, short summary.
- Classifying. Routing, prioritising, tagging — often better than keyword rules, and far easier to change.
- Drafting. A first version a person edits.
What it is not good at
- Knowing facts about you. It has never seen your price list. Asked for it, a poorly configured system will produce something that looks like a price list.
- Arithmetic and precise counting. Improving, still not the tool for a ledger.
- Being deterministic. The same input can produce different output. For anything requiring an identical answer every time, use ordinary code.
The practical consequence
Serious business deployments do not ask a model to recall anything. They retrieve the relevant facts from a real source and give them to the model to work with, and they constrain the output to a shape that can be validated. The model handles language; the systems around it handle truth.
Judge a proposal on how it handles that boundary. A vendor who cannot explain where the facts come from is proposing something that will eventually invent one.
Questions
- Which model should we use?
- It matters far less than people expect, and it changes every few months. Design so the model can be swapped. The parts that determine whether a deployment works — grounding, guardrails, evaluation — are the same regardless.
- Will our data be used to train the model?
- It depends entirely on the provider and plan, and it is a question to settle in writing before anything is connected. Business tiers of the major providers generally do not train on submitted data, but that should be verified, not assumed.
Wondering whether this applies to your business? That is a thirty-minute call, not a research project.
Book a Consultation Call