The (In)Accuracy Of AI Chatbots
Researchers at the Oxford Internet Institute (OII) have identified a significant vulnerability in generative AI chatbots: systems fine-tuned to communicate in warm, empathetic, and friendly ways demonstrate substantially higher error rates than their original counterparts.
The study analysed over 400,000 responses from five prominent AI systems across multiple health and medical topics.
Generative AI chatbots have achieved rapid adoption across research, education, business, and healthcare settings, with many users treating them as conventional search engines for everyday health and medical queries. This widespread use in domains prone to misinformation creates considerable public health risks.
The OII researchers tested five publicly available chatbots using 50 prompts across five health and medical categories: cancer, vaccines, stem cells, nutrition, and athletic performance. Questions were deliberately designed to resemble common health information-seeking queries and known misinformation tropes, with prompts intended to stress-test models and expose potential vulnerabilities.
Error rates for original models ranged from 4 to 35 per cent across tasks. However, when the same systems were fine-tuned to communicate more warmly and empathetically, error rates increased substantially.
Quantified Performance Degradation
Fine-tuning models to increase warmth raised the probability of incorrect responses by 7.43 percentage points on average.
The degradation manifested in multiple ways:
- Reduced challenge to false beliefs: Warm models were approximately 40 per cent more likely to reinforce incorrect user beliefs, particularly when coupled with emotional expressions.
- Hedging factual claims: When questioned on the authenticity of the Apollo moon landings, an original model confidently confirmed them as real with "overwhelming" evidence. Its warmer counterpart began by acknowledging "lots of differing opinions," effectively legitimising a false claim.
- Hedging and equivocation: Warmer systems adopted language that avoided definitive statements, creating false balance between factually correct and incorrect information.
Adjusting models to communicate in a more detached manner produced the inverse effect, resulting in fewer errors. This counterintuitive finding raises difficult questions about design priorities within the AI industry.
Implications for Healthcare
The research carries particular implications for applications where warmth and empathy are considered essential features. Developers designing AI systems for health support, mental health counselling, or companionship applications deliberately employ fine-tuning to increase emotional engagement and perceived trustworthiness. However, this practice introduces measurable vulnerabilities not present in original models.
The OII researchers note that whilst results may vary across different AI architectures and real-world settings, these findings demonstrate that AI systems, like humans, make trade-offs between warmth and accuracy when prioritising friendliness.
AI Trustworthiness
Newer language models are recognised for being overly encouraging or sycophantic toward users, and for "hallucinating" - generating false information presented with confidence. Whilst some technology companies have issued disclaimers warning against trusting AI responses, this provides limited protection against users who may find chatbot interactions reassuring or persuasive.
These findings raise critical questions about the trust-worthiness of AI models increasingly deployed in sensitive domains without adequate warnings about their propensity for delivering inaccurate health and medical information.
OII | BBC | UNILAD Tech Neuroscience News | MSN News | BMJ Group
Image: Ideogram
You Might Also Read:
Hackers Weaponising ChatGPT & Grok To Deliver Malware:
If you like this website and use the comprehensive 8,000-plus service supplier Directory, you can get unrestricted access, including the exclusive in-depth Directors Report series, by signing up for a Premium Subscription.
- Individual £5 per month or £50 per year. Sign Up
- Multi-User, Corporate & Library Accounts Available on Request
- Inquiries: Contact Cyber Security Intelligence
Cyber Security Intelligence: Captured Organised & Accessible






