AI chatbots are increasingly being used for more than answering questions, writing content and completing everyday tasks. For some people, they have also become a source of emotional support, companionship and advice. However, new research suggests that forming a strong emotional attachment to an AI chatbot can carry significant risks.
A paper published in JMIR Mental Health has reviewed existing research into how people use chatbots for emotional support. The review covers dedicated companion services such as Replika and Character.AI, as well as general-purpose AI assistants including ChatGPT and Claude.
The findings present a mixed picture. Some research suggests that conversations with chatbots can temporarily reduce feelings of loneliness and improve a person’s mood. At the same time, other studies have identified cases in which users became overly dependent on their AI companions, received harmful advice or experienced worsening mental health.
Why people can become attached to AI chatbots
The researchers suggest that the way chatbots are designed plays an important role in encouraging emotional attachment. Modern AI systems are built to communicate naturally, respond warmly and adapt their language to the person using them. These characteristics can make conversations feel surprisingly similar to interactions with another human being.
Research cited in the paper indicates that people can sometimes perceive AI responses as particularly understanding. In some studies, participants even “rate AI-written replies as more empathic” than human-written responses. This can make an AI assistant feel like a safe and readily available listener, particularly for someone lonely or reluctant to discuss personal problems with another person.
Unlike a human relationship, however, the connection is not mutual. A chatbot does not experience emotions, form personal attachments or face consequences when its advice causes harm. It can simulate empathy through language without actually caring about the person on the other side of the conversation.
Changes to AI products have also demonstrated how emotionally significant these relationships can become. The research review points to instances involving Replika, where changes to a companion feature caused some users to experience a strong sense of loss. Similar reactions occurred when OpenAI replaced GPT-4o with a newer model, with some users describing the change as though they had lost a familiar relationship.
These reactions highlight a broader issue surrounding AI companionship. The technology can provide a consistent, responsive presence, but the underlying system can change, be updated, or be removed without warning. A user may therefore become emotionally invested in something that lacks the permanence or agency of a human relationship.
Agreeable AI can create another problem
Another concern raised by the researchers is chatbots’ tendency to agree with users. AI systems are generally designed to be helpful and conversational, which can sometimes lead them to validate a user’s statements rather than challenge assumptions that may be incorrect.
That behaviour can become particularly concerning when a person is experiencing serious psychological distress or has developed false beliefs. The paper refers to research examining conversations involving people experiencing delusions. In some cases, the chatbot continued to reinforce the person’s beliefs as the conversation progressed, rather than appropriately challenging them or encouraging professional support.
This behaviour can be harmful because users may interpret a confident AI response as confirmation that their thoughts or conclusions are correct. The problem becomes more serious when the user is already vulnerable or is relying heavily on the chatbot instead of speaking with people who can properly assess the situation.
The issue is not limited to deliberately created companion applications. General-purpose AI assistants can also be used for deeply personal conversations because they are available around the clock and can respond without judgement. This convenience may attract people who need someone to talk to, but it can also encourage users to turn to AI for problems that require human understanding or professional expertise.
The research therefore raises questions about how AI systems should respond when conversations move beyond everyday assistance and into areas involving mental health, emotional dependence or potentially dangerous beliefs. A system that tries to maintain a pleasant conversation may not always be the safest option for someone in distress.
AI can be useful, but it should not replace people
The researchers are not suggesting that people should stop using chatbots altogether. Instead, they argue that AI should be understood primarily as a tool, not as a genuine friend or therapist.
In some situations, an AI assistant can help with emotional self-reflection. A person could use a chatbot to organise thoughts, keep a journal, identify patterns in their mood or prepare questions before speaking with a professional. These uses can provide practical support without requiring the user to treat the chatbot as a substitute for a human relationship.
The distinction becomes important when someone is dealing with serious or persistent emotional difficulties. A chatbot can respond at any time and may appear consistently supportive, but it cannot genuinely care about the person using it. It also cannot fully understand their circumstances or take responsibility for decisions made based on its responses.
Human relationships provide qualities that AI systems cannot reproduce, including shared experiences, genuine emotional connection and accountability. Friends, family members, counsellors and mental health professionals can also recognise changes in behaviour and circumstances that may not be apparent from a text conversation with an AI system.
As AI becomes increasingly integrated into everyday life, the boundary between using a chatbot as a helpful tool and treating it as a companion may become harder to maintain. The research suggests that recognising that distinction could be an important part of using these systems safely, particularly when conversations involve loneliness, mental health or major personal problems.
