The (In)Accuracy Of AI Chatbots

Researchers at the Oxford Internet Institute (OII) have identified a significant vulnerability in generative AI chatbots: systems fine-tuned to communicate in warm, empathetic, and friendly ways demonstrate substantially higher error rates than their original counterparts.

The study analysed over 400,000 responses from five prominent AI systems across multiple health and medical topics.

Generative AI chatbots have achieved rapid adoption across research, education, business, and healthcare settings, with many users treating them as conventional search engines for everyday health and medical queries. This widespread use in domains prone to misinformation creates considerable public health risks.

The OII researchers tested five publicly available chatbots using 50 prompts across five health and medical categories: cancer, vaccines, stem cells, nutrition, and athletic performance. Questions were deliberately designed to resemble common health information-seeking queries and known misinformation tropes, with prompts intended to stress-test models and expose potential vulnerabilities.

Error rates for original models ranged from 4 to 35 per cent across tasks. However, when the same systems were fine-tuned to communicate more warmly and empathetically, error rates increased substantially.

Quantified Performance Degradation

Fine-tuning models to increase warmth raised the probability of incorrect responses by 7.43 percentage points on average.

The degradation manifested in multiple ways:

  • Reduced challenge to false beliefs: Warm models were approximately 40 per cent more likely to reinforce incorrect user beliefs, particularly when coupled with emotional expressions.
  • Hedging factual claims: When questioned on the authenticity of the Apollo moon landings, an original model confidently confirmed them as real with "overwhelming" evidence. Its warmer counterpart began by acknowledging "lots of differing opinions," effectively legitimising a false claim.
  • Hedging and equivocation: Warmer systems adopted language that avoided definitive statements, creating false balance between factually correct and incorrect information.

Adjusting models to communicate in a more detached manner produced the inverse effect, resulting in fewer errors. This counterintuitive finding raises difficult questions about design priorities within the AI industry.

Implications for Healthcare 

The research carries particular implications for applications where warmth and empathy are considered essential features. Developers designing AI systems for health support, mental health counselling, or companionship applications deliberately employ fine-tuning to increase emotional engagement and perceived trustworthiness. However, this practice introduces measurable vulnerabilities not present in original models.

The OII researchers note that whilst results may vary across different AI architectures and real-world settings, these findings demonstrate that AI systems, like humans, make trade-offs between warmth and accuracy when prioritising friendliness.

AI Trustworthiness

Newer language models are recognised for being overly encouraging or sycophantic toward users, and for "hallucinating" - generating false information presented with confidence. Whilst some technology companies have issued disclaimers warning against trusting AI responses, this provides limited protection against users who may find chatbot interactions reassuring or persuasive.

These findings raise critical questions about the trust-worthiness of AI models increasingly deployed in sensitive domains without adequate warnings about their propensity for delivering inaccurate health and medical information.

OII  |  BBC  |  UNILAD Tech   Neuroscience News   |  MSN News  |   BMJ Group 

Image: Ideogram

You Might Also Read:

Hackers Weaponising ChatGPT & Grok To Deliver Malware:


If you like this website and use the comprehensive 8,000-plus service supplier Directory, you can get unrestricted access, including the exclusive in-depth Directors Report series, by signing up for a Premium Subscription.

  • Individual £5 per month or £50 per year. Sign Up
  • Multi-User, Corporate & Library Accounts Available on Request

Cyber Security Intelligence: Captured Organised & Accessible


 

 

« Britain Invests In Local AI Venture
Leading Nations In Digital Education & The Evolution Of Digital Learning [extract] »

ManageEngine
CyberSecurity Jobsite
Check Point

Directory of Suppliers

NordLayer

NordLayer

NordLayer is an adaptive network access security solution for modern businesses — from the world’s most trusted cybersecurity brand, Nord Security. 

Clayden Law

Clayden Law

Clayden Law advise global businesses that buy and sell technology products and services. We are experts in information technology, data privacy and cybersecurity law.

Tines

Tines

The Tines security automation platform helps security teams automate manual tasks, making them more effective and efficient.

ManageEngine

ManageEngine

As the IT management division of Zoho Corporation, ManageEngine prioritizes flexible solutions that work for all businesses, regardless of size or budget.

Keygraph

Keygraph

Keygraph builds AI penetration testing for application security teams. Keygraph also builds Shannon, a leading open-source AI pentester

Okta

Okta

Okta is an enterprise-grade identity management service, built from the ground up in the cloud to address the challenges of a cloud-mobile-interconnected world.

DataVantage

DataVantage

DataVantage data masking and data management software helps you prevent data breaches, pass compliance audits and meet regulatory requirements such as HIPAA and PCI DSS.

Datiphy

Datiphy

Datiphy's data-centric security platform uses behavioral analytics, and data-centric auditing and protection capabilities to mitigate risk.

NXP Semiconductors

NXP Semiconductors

NXP is a world leader in secure connectivity solutions for embedded applications and the Internet of Things.

Cygilant

Cygilant

Cygilant is a SOC2 certified service provider that combines MSSP and Incident Detection and Response (IDR) capabilities managed by global SOCs staffed with trained security engineers.

Information and Communication Technology Authority (ICT Authority) - Kenya

Information and Communication Technology Authority (ICT Authority) - Kenya

The ICT Authority is responsible for enforcing ICT standards in Government and ensuring information security.

Squalio

Squalio

Squalio is an information technology group that delivers solutions and services for secure and effective IT management.

ISECURION Technology & Consulting

ISECURION Technology & Consulting

ISECURION is an information security consulting company. We provide a unique blend of services to our customers catering to the current information security landscape.

Trusted Security Solutions (TSS)

Trusted Security Solutions (TSS)

TSS are specialist in IT Security and providing Cybersecurity Solutions & Services combined with storage and backup.

Cloudflare

Cloudflare

Cloudflare is a global network designed to make everything you connect to the Internet secure, private, fast, and reliable.

Ontinue

Ontinue

Ontinue ION is an MXDR service that provides Nonstop SecOps through five key capabilities that enable your organization to respond to attacks and continuously reduce risk.

AHAD

AHAD

AHAD provides cybersecurity, digital transformation, and risk management services and solutions to Government, Fortune 500, And Start-Up Companies in the Middle East region.

Silobreaker

Silobreaker

Silobreaker is a SaaS platform that enables threat intelligence teams to produce high-quality and relevant intelligence at a faster pace.

Infratech

Infratech

Infratech is a leading Saudi company providing cutting-edge services and solutions in IT Infrastructure, IT Security and Digital Transformation.

Seculyze

Seculyze

Seculyze is a cybersecurity solution that integrates with Microsoft Sentinel and Defender to reduce alert fatigue and minimize the risk of cyberattacks.

Fleuret AI

Fleuret AI

Building the future of sovereign AI pentesting. Fleuret AI combines artificial intelligence and offensive expertise to make pentesting accessible, continuous, and sovereign.