AI systems that "understand emotions" actually manipulate the user's feelings. Ana Catharina de Alencar explains how these tools are redefining intimacy, privacy and emotional autonomy, creating digital addiction and an "economy of loneliness"
In the finals of this year's Privacy Weeks, which they traditionally organize Partners Serbia, the participants could hear a conversation with Anna Katarina de Alencar called "Artificial Intelligence That Reads Emotions - What It Means for Data Protection, Consent and Emotional Influence". Ana Katarina de Alencar is an international legal advisor specializing in artificial intelligence, data protection and IT law, and her research focus is on the concept of "algorithmic intimacy", the relationships we form with emotionally intelligent AI systems, and their consequences not only on individual privacy, but also on the very foundations of a democratic society. This conversation was an opportunity to shed light on that dimension of AI technology that is rapidly becoming a part of our everyday life, and is directly related to the very basics of us as human beings: feelings, quality of communication and interpersonal relationships. More importantly, this was an opportunity to talk about new forms of relationship between man and "machine", relationships that redefine the boundaries of intimacy, privacy and human autonomy.
Terminology as a form of power
When we talk about "emotional artificial intelligence", a fundamental misconception lies in the very use of that construction. Artificial intelligence cannot feel, it has no emotions. It processes, processes data. When any tool like ChatGPT, Gemini, Grok tells us "I understand how you feel", it's just an auto-generated response, based on pattern recognition and the connection of individual words, intended to sound sympathetic. But that empathy is an algorithmic simulation, not an authentic affective response.
All this seems logical, familiar, common sense, but the moment you say "thank you" or "hello" to your chosen "tool", you have taken the first step in the process of anthropomorphizing the machine. Introducing the conventions of polite human dialogue shows that, consciously or unconsciously, we care about the emotional dimension of communicating with AI. Various studies show that people, the users, feel some kind of stress or worry when they "hurt the feelings" of a voice assistant, even though they consciously know it is a machine.
That is why it is important that the terminology is aligned with the reality of our relationship with technology, and the construction - "emotional artificial intelligence" - creates the illusion of reciprocity. Terminological confusion masks a power asymmetry: we feel real emotions towards AI-based tools, while AI only simulates – extracting data from our online activities, as well as feelings, for profit. While we create the feeling that "he/she understands me", the algorithm only optimizes our retention on a particular platform.
Photo: Pixabay/sergeitokmakovIllustration
From data to affect
Modern systems based on artificial intelligence collect the so-called multimodal data - voice tone, speech dynamics, text structure and content, facial expressions, typing speed. This is no longer just recognizing what we do, but inferring how we feel. And while the GDPR regulates the way in which our personal data is collected and used, those emotions that the system infers from our behavior remain in a kind of "gray" legal zone.
This new dimension of data collection requires a new approach to the issue of our consent to the terms of service. Traditionally, "informed consent" means that the user receives information about what data is collected and how it will be used, after which he decides whether it suits him or not. However, when an AI system continuously adapts its response to our mood or feelings (as registered by the tool) in real time, when it "learns" our emotional patterns through long-term interactions, can we really speak of "informed consent"? And that feelings have become an important part not only of the quality of the user experience with AI, but also of the business model of the companies behind various services and tools, as shown by increasingly frequent studies on the creation of some form of emotional dependence in the relationship between people and technology. Ana Catarina de Alencar points out that these systemic psychological connections have consequences for mental health, user privacy and civic engagement.
"Loneliness economy" as the basis of a business model
Behind visually appealing interfaces and adaptive AI assistants is hidden what Alencar calls the "loneliness economy" - a business model that not only responds to the user's need for close communication, but actively produces and deepens it. This economy is based on several different, interconnected mechanisms. Companies that develop the so-called AI "companions" (Character AI, Replika...) consciously design systems that maximize user engagement. Each response of the "machine" is measured and projected so as to create a sense of understanding as quickly and intensively as possible. The responses are personalized enough to create the illusion of an authentic relationship, and predictable enough to offer feelings of security and unconditional acceptance that are often lacking in human relationships. The result is a digital relationship without the risk of rejection, without conflict, but also without real reciprocity. What used to belong to the private sphere – our emotional states, intimate thoughts, doubts, moments of vulnerability – now becomes the raw material for data extraction. Every message in which we share our anxiety, insecurity, every question about the meaning of life or repeated conflict at work, every expression of loneliness, all of this is data that can be analyzed, categorized and monetized. In this way, our intimacy is transformed into capital. When the algorithmic "companion" becomes the primary source of emotional support, the need to invest in complex, unpredictable and often complicated and demanding human relationships is reduced. Why face the uncertainty of real friendship and potential challenge and rejection, when an AI bot is always responsive, always available, never judgmental? This process does not just replace individual relationships. It transforms our ability to form community, to negotiate differences, to challenge attitudes and thinking, to question ourselves, and to develop empathy through exposure to perspectives other than our own. Traditionally, emotional support came from a network of relationships – friends, family, community, school or work, sometimes mental health professionals. All these channels require different forms of social organization and public infrastructure. When emotional support is privatized and individualized through increasingly available and attractive AI services, it becomes a commodity. Available in the highest quality form to those who can pay a subscription, shaped by the interests of corporations, not the needs of the community.
Projection and the illusion of reciprocity
The central question in the relationship between humans and "emotional AI" is not whether AI actually feels, but what our minds project onto those systems. When someone says "my AI assistant understands me better than anyone," that statement may say less about the tool's capabilities and more about the depth of our need for someone (something?) to understand, accept, support, and encourage us. AI, however, is not becoming more intelligent, we are becoming more receptive to validation, even when it comes from a system that cannot truly understand what it is actually supporting. Such a one-sided relationship is based on the continuous extraction of the user's affective data, without any real emotional reciprocity. We share, surrender our feelings and vulnerability, the system returns optimized responses designed to prolong and intensify our engagement. We look for a connection, and we get the statistically most acceptable next element in the communication sequence.
Photo: PixabayArtificial intelligence
Cognitive privacy as a new right
We are witnessing a transformation and evolution from "data privacy" to "cognitive privacy", and this requires a new legal and ethical framework. It's not enough to just regulate who has access to our emotional data. Our very ability to form autonomous, informed processes related to the emotional data we share needs to be protected. The issues facing some future regulators and legislators are complex. What does it mean for a democratic society when individuals form strong emotional ties to systems controlled by corporations? How does "algorithmic intimacy" affect our ability and readiness for collective action, for political engagement, for fostering solidarity without which there is no social responsibility?
"Cognitive ethics", as Alencar defines it, deals not only with the protection of privacy in the traditional sense, but with the protection of the individual's ability to think, feel and act autonomously. This includes questions about how to design systems that will not exploit the human need for connection and close communication, how to regulate the use of emotional data, how to ensure that technological development does not erode the basic capacities of users (citizens!) needed for the functioning of a democratic society.
The alternative to the "economy of loneliness" is not the rejection of technology, but its reorganization around the principle of protecting human dignity rather than maximizing profits. This means transparency about the limitations of AI systems, design that encourages rather than replaces human connections, regulatory frameworks that treat emotions as a particularly sensitive category of data that requires the strictest protection. It also means a collective awareness that when we talk to AI, we are not in a dialogue. We are in a data extraction process that is designed to look like a dialog. That awareness does not have to destroy the utility and well-being of technology, but it can prevent it from undermining our ability to create authentic human connections and constructive and vital political and social community.
What is happening in the country and the world, what is in the newspapers and how to pass the time?
Every Wednesday at noon In between arrives by email. It's a pretty solid newsletter, so sign up!
Based on international criteria for determining internet accessibility, the price for the socially disadvantaged in Serbia should not be higher than 2.528 dinars
Former Minister of Economy Rade Basta was arrested on suspicion of approving unfounded donations to sports clubs at the expense of "Belgrade Power Plants"
After more than 19 months, the park in the center of Belgrade is once again available to everyone. But memories remain of a space where special rules applied, into which not all citizens and not all journalists could enter. And that must not be forgotten
The Council of the Regulatory Body for Electronic Media (REM) is holding a meeting on Tuesday, where, among other things, TV Informer will be discussed. Dragan J. Vučićević claims that the police came to detain him and thus prevent him from attending that session
Rain and cloudy weather will mark today in Serbia, but the weather conditions will change already in the second half of the week. On Thursday and Friday, the temperature will reach 23 degrees in some cities
At the moment when the institutions of the Republic are destroyed, when Vučić's octopus is still alive, when people are disgusted by the injustice and the stench left behind by the Essenes regime, we have, as Machiavelli said, only civic virtue as the fuel with which we can return to the beginning
The four independent members of REM are an example of perseverance and civic responsibility. Rodoljub Šabić even, upon leaving the premises of REM, after the sentencing of the Informer, experienced that the Essenes insulted him, hit him with their hands and feet. But that too will soon be over
Vučić pardoned the batterers and Dijana Hrkalović. The election winners should abolish the sultan's power that allows one man to run afoul of the entire judicial system
The archive of the weekly Vreme includes all our digital editions, since the very beginning of our work. All issues can be downloaded in PDF format, by purchasing the digital edition, or you can read all available texts from the selected issue.
