Automation & AI

What to do when the answer is confidently wrong

The dangerous output is not the one that fails. It is the one that is plausible, specific, and mistaken.

Anyone who has used these tools for real work has met the pattern: an answer that is fluent, detailed, internally consistent and wrong. It is more dangerous than a refusal, because a refusal makes you go and check.

Sort your tasks by cost of being wrong

Some outputs are self-correcting. A badly drafted paragraph is obvious and gets rewritten. Others are not: a figure, a date, a specification, a claim about what a supplier offers. Those pass inspection precisely because they look like the kind of thing that would be right.

The line to draw is not between easy and hard tasks. It is between outputs where a mistake announces itself and outputs where it does not. Only the second kind needs verification, but it needs it every single time.

Verify against the source, not against plausibility

The practical rule: any number, price, date or capability that reaches a customer gets checked against the system that owns it, not against whether it sounds about right. Sounding about right is exactly the failure mode.

This is faster than it sounds. Checking a figure against a live source takes seconds. Discovering it through a customer takes a relationship.

Do not fix by feel

When something does come out wrong, resist the instinct to rewrite the instruction until it looks better. Find out what the tool was missing — usually context it never had rather than reasoning it failed at — and give it that. Otherwise you are tuning the phrasing of a problem you have not diagnosed.