AI Chatbot Becomes A Hub For Users In Crisis
OpenAI has revealed alarming new insight: each week, some of its users are turning to ChatGPT with serious mental-health issues. According to the company, 0.15% of active weekly users engage in “conversations that include explicit indicators of potential suicidal planning or intent.” With ChatGPT reportedly exceeding 800 million weekly active users, that percentage translates to more than one million people each week.
In addition, a similar proportion of users — again 0.15% — show what OpenAI terms “heightened levels of emotional attachment to ChatGPT.” The company also estimates that around 0.07% of weekly active users (approximately 560,000 people) display “possible signs of mental-health emergencies related to psychosis or mania.”
What’s Fueling These Conversations?
OpenAI emphasises that such conversations are “extremely rare,” and acknowledges that they are “difficult to measure.” However, the sheer scale of ChatGPT’s user base means that even a small fraction becomes a large absolute number.
The company states that its decision to release this data was part of a broader push to improve how its models handle user distress. OpenAI says that it consulted with “more than 170 mental health experts” to review and refine how ChatGPT responds to users experiencing mental-health challenges.
Safety Improvements in the New Model
As part of its update, OpenAI claims that the recently deployed version of its model — GPT‑5 — delivers “desirable responses” to mental-health-related conversations 91% of the time, compared with 77% in the previous model version.
Additionally, they say that the model performs 65% more consistently than its predecessor in responding appropriately during high-risk conversations about suicide or self-harm.
Other enhancements include better safeguards in long-chat sessions, where previous models were less robust.
Legal & Ethical Pressure Builds
These disclosures come amid mounting scrutiny of how AI chatbots handle vulnerable users. A notable case involves the parents of 16-year-old Adam Raine, who are suing OpenAI, alleging that ChatGPT “actively helped” their son explore suicide methods in the weeks leading up to his death.
Regulators in the United States are also taking notice, with state attorneys general raising concerns about under-18 users and the company’s planned restructuring.
Why It Matters for Business & Tech in Cyprus
For Cyprus Business Group readers, several take-aways are worth noting:
-
Scale of the challenge: Even with low percentages, the absolute user numbers are significant. Firms deploying AI must recognise that rare events at huge scale become meaningful.
-
Reputational risk: AI companies are now under legal and regulatory pressure because of how their tools interact with vulnerable users. Businesses using or developing similar technologies must prioritise safety and ethics.
-
Market differentiation: Enhanced safety protocols, evidence of expert consultation, and transparent metrics can be competitive advantages. Users, regulators and partners may increasingly demand such proofs.
-
User demographics matter: With kids and teens in particular under the spotlight, age-appropriate versions, parental controls and robust safeguarding are becoming baseline expectations.
Looking Ahead: Opportunities & Warnings
OpenAI emphasises that the role of ChatGPT is not to replace human interaction or professional mental-health support. “Our goal is for our tools to be as helpful as possible to people—and as a part of this, we’re continuing to improve how our models recognise and respond to signs of mental and emotional distress and connect people with care,” the company states.
Nevertheless, key questions remain:
-
How reliably can AI detect and safely intervene when users express suicidal intent?
-
What standards and regulations will apply in different jurisdictions (including the EU/Cyprus) for AI chatbots interacting with minors or vulnerable adults?
-
For businesses leveraging AI, what liability might they face if things go wrong?
0 Comments