AI Hallucinations: When Your Smart Assistant Gets Too Creative With The Truth

We're facing challenges with certain confident hallucinations despite instructions to respond with 'Don't know' when uncertain.

This candid admission from a community member sparked an insightful discussion about one of AI's most persistent challenges: knowing when to say "I don't know."

The 80% Problem

The member shared that their AI customer support agent, built with RAG (Retrieval Augmented Generation) and tool calling, was achieving 80% accuracy when answering customer questions by referencing documentation.

But that remaining 20% represents a significant challenge – especially when the AI confidently provides incorrect information rather than admitting uncertainty.

When "I Don't Know" Is The Right Answer

Unlike humans, who often hedge when uncertain, many AI systems struggle with acknowledging limitations. This creates a particular challenge for customer-facing applications where incorrect information can damage trust and create problems.

As the original poster explained, "The expectation is if the LLM is not sure, chat should get transferred to the human in the loop."

Several community members agreed this is a widespread challenge, with one noting: "Currently looks like one area AI is no match for human is in saying a confident 'I don't know'."

Approaches From The Community

The discussion revealed several practical strategies:

1. Prompt Engineering - Spending significant time refining prompts to encourage appropriate uncertainty

2. Evaluation Systems - Building separate evaluator models that check the primary AI's answers

3. Confidence Thresholds - Setting strict thresholds for when to transfer to human agents

4. Controlled Information Sources - Carefully curating the knowledge base to reduce ambiguity

One member shared their success: "We spend enough time improving the prompt to reduce hallucinations. Even though it's not 100% accurate, the rate of hallucination is very low."

Use Cases With Zero Tolerance

The conversation took an interesting turn when the original poster mentioned this was just a warm-up for their real goal:

"The main usecase in mind has zero tolerance for error/hallucination...so was testing the waters with a simpler one."

This highlights a critical distinction in AI applications. For some use cases, occasional errors are acceptable. For others – particularly in fields like healthcare, finance, or legal advice – even rare hallucinations can have serious consequences.

Balance and Expectations

The community converged on a balanced view: while perfect accuracy remains challenging, significant improvements are possible through careful system design.

As one member put it: "Our problem is not 100% accuracy, what we want is 100% accuracy when LLM don't know the answer...even if it says don't know 30% of the time also it's fine."

This highlights an important principle for AI design: sometimes being conservative and admitting uncertainty is better than being confidently wrong.

The discussion showed that while we often focus on making AI systems smarter, teaching them when to acknowledge limitations might be equally important – especially as these systems take on more customer-facing roles.