Does it matter if empathic AI has no empathy?

Imagine a machine that provides a simulation of any experience a person might want, but once the machine is activated, the person is unable to tell that the experience isn’t real. When Robert Nozick formulated this thought experiment in 1974 (ref. ¹), it was meant to be obvious that people in otherwise ordinary circumstances would be making a horrible mistake if they hooked themselves up to such a machine permanently. During the intervening decades, however, cultural commitment to that core value — the value of being in contact with reality as it is — has become more tenuous, and the empathic use of AI, in which people seek to be understood, cared for and even loved by a large language model (LLM), is on the rise.

The use of LLMs for information, entertainment and even behavioural encouragement (such as encouragement to go for a walk or make a friend) can be constructive. Applications of LLM chatbots in certain therapeutic domains, from diagnosis to health advice, also seem promising². However, we, as a team of psychologists, philosophers and computer scientists, have concerns about LLMs as a source of empathic care. We list several research questions on the consequences and risks of empathic AI that must be addressed (Table 1), and in the following, we challenge four reassurances that have been offered in defence of empathic AI.