Hallucination: definition, value and limits
Definition
A hallucination is a language model output that sounds fluent and confident but is factually wrong or invented. The model fills gaps with plausible text instead of signalling that it does not know. In an automated flow this is risky, because a wrong answer moves on unchecked. That is why check steps and evidence belong in every AI step.
Why the Term Matters
Anyone building flows with AI has to know that a model does not tell true from false; it produces likely words. Without control, a hallucination lands straight in an invoice, a customer reply or a data field. The term helps you scope the AI step deliberately and put a validation in front of it. That discipline is what turns a prototype into an operable product.
When It Is Relevant
- An AI step writes values into a system other people use.
- Answers contain numbers, names, prices or legal statements.
- The flow runs through without a human check in between.
- The flow is meant to become a product others are accountable for.
When Not
For creative tasks with no factual claim, such as a first draft or brainstorming, a freely worded output is not an error but wanted. And for purely rule-based steps without a generative model, hallucination simply does not arise.
Industries That Use It
- Insurance: claims may only be pre-filled automatically from evidenced fields.
- Legal services: citations and deadlines must be checked against the source, never freely generated.
- Medical technology: automated summaries need a hard limit against invented details.
- Financial accounting: amounts and postings only move on with a plausibility check.
Typical Use Cases
- Allow an answer only when a retrieved document backs it.
- Reconcile numbers and names against the source after the AI step.
- Escalate uncertain cases to a human instead of closing them automatically.
- Attach confidence or a source reference to outputs and log them.
Decision Guide
| Signal | Term fits | Alternative |
|---|---|---|
| AI produces facts, numbers, names | check step and evidence needed | plain rule without a model |
| Output goes into the system unchecked | put validation in front | human approval |
| Creative draft with no factual claim | hallucination uncritical | no check needed |
| A source is available | retrieval as the basis | avoid free generation |
Example
A team lets an AI step summarize incoming requests. To keep invented amounts out, a later step reconciles every number against the original document and flags deviations. Only checked cases move on automatically, the rest go to a human. This describes the method, not an asserted client metric.
Service Connection
Robust check steps usually emerge in workflow automation or an n8n build, and the reliable extension in MVP development.
Sources
This framing describes common practice around hallucinations. Concrete check mechanisms, thresholds and escalation paths depend on the case, the systems and the tools used and must be checked case by case.