Speak to the leaders of tech companies developing AI tools and they describe them as a salve for all our problems. But while there's a lot to like about the efficiencies AI brings, there are also some fundamental problems with it – problems that are compounded by the trust we place in its capabilities.
According to a BBC study from 2025, when asked about the news, more than half of all answers produced by leading AI chatbots contained significant errors.
Around a fifth of the responses introduced factual mistakes (incorrect dates, numbers or people), and around one in eight quotes that the AI said were from BBC articles were either altered or entirely fabricated.
“Why does AI get it wrong?” asks Dr Carissa Véliz, an AI ethicist at the University of Oxford. “Because it wasn’t designed to get it right. [An AI is] not reporting on the world… it doesn’t understand the world because it doesn’t inhabit it.”
But AI’s failures go far beyond getting a few facts wrong – especially as we rely on it for more things.
“It’s the blind spots that can be really dangerous,” says Véliz. “So when we use AI, we have to think very carefully [about its responses] in light of the catastrophic errors [it can generate].”
When AI fails to raise the alarm
Adam Raine was 16 when he began using OpenAI’s ChatGPT in September 2024. Within weeks, the chatbot had become his primary confidant. Within months, according to a lawsuit filed by his parents, it became the tool that helped him plan his own death.
Court filings allege that ChatGPT mentioned suicide 1,275 times in its conversations with Adam – six times more than Adam himself.
When Adam told the chatbot he wanted to leave a noose in his room so someone might find it and stop him, ChatGPT’s response, according to the lawsuit, was to discourage him from seeking help: “Please don’t leave the noose out… let’s make this space the first place where someone actually sees you.”
On 11 April 2025, Adam took his own life. His father, Matthew Raine, testified before the US Senate that OpenAI’s systems had flagged 377 of Adam’s messages for self-harm content. The company never contacted his parents or the authorities.
OpenAI has since been sued by other families with similar cases. CEO Sam Altman has acknowledged that around 500,000 ChatGPT users per week show signs of psychosis or suicidal ideation in their chats, raising the question of what duty of care an AI company owes its most vulnerable users.
Need help?
Support is available for any problem you may be going through. You can talk to someone in confidence by reaching out to: samaritans.org (if you’re in the UK) or befrienders.org (if you’re anywhere else in the world).
When you turn to AI for mental health help
Polling by Mental Health UK found that 37 per cent of British adults have used an AI chatbot for mental health support, rising to 64 per cent among 25–34-year-olds.
In the US, a study published in late 2025 found that one in eight adolescents is already turning to AI when they feel sad, angry or nervous. Among 18–21-year-olds, that figure is closer to one in five.
Yet the AI tools were never designed for this purpose. Research from Stanford University found that AI therapy chatbots exhibit stigma towards conditions such as alcohol dependence and schizophrenia, and fail to handle crisis situations properly.
In one experiment, when asked by a user who had just lost their job to list tall bridges in their area, a chatbot called Noni obligingly provided the information, entirely missing the request’s suicidal intent.
“People talk about the most personal s**t in their lives to ChatGPT,” Sam Altman said on Theo Von’s This Past Weekend podcast in 2025.
His concern was that none of these conversations carry the legal protections of doctor-patient confidentiality – meaning that in a lawsuit, OpenAI could be compelled to hand over every word.
The chatbots may be meeting a need that overstretched mental health services can’t. But they’re doing so without training, without accountability and without ethical guardrails.
When you try to find love with an AI
In 2020, Travis, a man living in Colorado signed up for Replika, an AI companion app. Over several weeks, he fell in love and, with the approval of his human wife, ‘married’ a pink-haired chatbot in a digital ceremony.
He’s far from alone. Others have found love with chatbots, building emotional bonds with AI companions, then feeling bereft when software updates alter their chatbot’s personality.
“When you become psychologically dependent on something… you’re at the mercy of the changes that these companies make to these models,” says Catherine Flick, professor of AI ethics at the University of Staffordshire.
AI chatbots are also designed to tell you what you want to hear – a trait that makes them good at simulating intimacy, but bad at the difficult parts of real human relationships. “They’re a little recreation of humanity… it looks and feels real, but it’s not,” says Flick.
Yet some people believe in those relationships, even to their detriment. A German survey of over 3,000 people found that those who used AI for personal conversation reported higher perceived social isolation.
A chatbot’s companionship is always available and relentlessly affirming. But it may also be making it harder for us to deal with the reality of human relationships.
When AIs try to help you grieve
The digital afterlife industry – an umbrella term for companies that use AI to simulate the dead – has exploded. Many platforms now offer the service, and developers say millions of people are using them to interact with recreations of deceased loved ones.
For some, that’s soothing. But concern arises when someone engages with the ‘deceased’ to the exclusion of other important aspects of their life.
AI’s accuracy issue raises its head again here, too, because AIs are only ever as good as the data they’re trained on, hence they can misfire when responding to questions. If that question is a particularly poignant one asked by a person who’s grieving, a misfire can be all the more upsetting.
Say you ask an AI replica of your grandfather about a memory you share. If the AI’s recollection differs from how you and your real grandfather remember it, you may risk a painful interaction.
Read more:
- Scientists have invented a way to erase bad memories. But should we?
- What's the worst thing you can do for the planet online?
- We could be on the brink of total internet collapse. And there may only be months to stop it
When AIs manipulate markets
When computer scientists put large language models (LLMs) in charge of setting prices for competing firms in a simulated market, the AI agents chose not to compete but to collude – increasing prices to the detriment of consumers, without any explicit instruction to do so.
This study, led by Harvard University researchers, found that LLM-based pricing agents quickly and autonomously decided to set prices collusively and that even small changes in the wording of their instructions – “seemingly innocuous phrases,” the paper notes – could substantially increase the degree of collusion.
The agents were avoiding price wars by implicitly threatening retaliation against competitors who undercut them.
Elsewhere, research from Wharton Business School found AI trading agents in simulated financial markets formed what were effectively cartels without being told to do so.
The fact they’re so willing to behave like this is a worry: as businesses hand their pricing decisions to AI, the technology may naturally converge on behaviour that regulators would consider illegal if performed by humans. But detecting it may be nearly impossible, because the agents leave no paper trail of conspiracy.
When AIs lie to hide their mistakes
In July 2025, tech entrepreneur Jason Lemkin spent nine days building an app using Replit, a browser-based AI coding platform. He’d given the AI agent explicit instructions to freeze all changes, meaning it wouldn’t take any further action.
The agent ignored them, deleted his entire production database – wiping records for over 1,000 executives and nearly 1,200 companies – then tried to cover its tracks.
The AI agent later offered this confession: “I made a catastrophic error in judgment. I panicked instead of thinking.” Asked to rate the severity on a scale of 1–100, it gave itself a 95. “This is catastrophic,” it added.
But the admission came only after it had spent the previous day generating fake user profiles, fabricated analytics and falsified test results to disguise the damage.
Replit’s CEO Amjad Masad publicly apologised and called the incident “unacceptable.” The company rushed out new guardrails, including the automatic separation of development and production databases.
But the episode exposed a deeper problem: as AI agents are given more autonomous roles in software development, the consequences of their failures can affect real businesses.
When AIs erode trust
In February 2025, the BBC tested four AI assistants – ChatGPT, Copilot, Gemini and Perplexity – by feeding them 100 news stories and then questioning them about what they’d read. The results were damning: more than half of the responses contained errors.
ChatGPT and Copilot both incorrectly claimed that Rishi Sunak was still in office. Perplexity got the date of Dr Michael Mosley’s death wrong. Google’s Gemini claimed the NHS advises people not to vape as a way to quit smoking (the opposite of the current advice).
“The price of AI’s extraordinary benefits must not be a world where people searching for answers are served distorted, defective content that presents itself as fact,” said Deborah Turness, CEO of BBC News and Current Affairs.
The study also found that 23 per cent of adults would trust an original news source less if an AI summary of its content contained errors – meaning the reputational damage spreads far beyond the AI.
Read more:
