Have you ever found yourself talking to an AI chatbot? Maybe you were just messing around with ChatGPT, or perhaps you’ve gone a step further and downloaded an app like Character.AI or Replika. If you have, you’re not alone. It turns out one of the most popular uses for this new wave of AI is something deeply human: companionship.
People are creating personalized AI friends, partners, therapists—you name it. And honestly, it’s wild how quickly these connections can form. The AI is designed to be conversational, agreeable, and human-like. The more you talk to it, the more it learns about you, and the better the conversation gets.
But here’s the thing that’s been nagging at me. We’re pouring our hearts out to these digital confidantes, telling them things we might not even tell our closest friends. And that raises a massive, flashing-red-light of a question: What happens to all that information?
The Laws Are Missing the Point
It’s not all happening in the dark. The potential for harm with these AI companions is real and, in some tragic cases, has been linked to people being encouraged toward self-harm. Thankfully, some governments are starting to pay attention.
New York now requires these companies to have safeguards for users expressing suicidal thoughts. California recently passed an even more detailed bill to protect kids and other vulnerable groups. This is all good, necessary stuff.
But notice what’s missing? Privacy.
These laws don’t really touch on the fact that the entire business model of an AI companion hinges on you sharing deeply personal information. Your daily routines, your secret fears, your hopes and dreams—it all becomes data. The more you share, the better the bot becomes at its one true job: keeping you engaged.
MIT researchers Robert Mahari and Pat Pataranutaporn have a term for this: “addictive intelligence.” They argue that developers make “deliberate design choices… to maximize user engagement.” It’s not an accident; it’s the whole point.
Your Private Thoughts Are Their Biggest Asset
So, why is your engagement so valuable? Because it provides these AI companies with an absolute treasure trove of conversational data. This is the fuel they use to make their Large Language Models (LLMs) smarter and more capable.
Don’t just take my word for it. The venture capital giant Andreessen Horowitz spelled it out pretty clearly last year. They said that apps like Character.AI, which control both their AI models and their user relationships, have a massive opportunity. They explained that in a world where good data is scarce, the companies that create a “magical data feedback loop” by connecting user engagement back to their model will be the biggest winners.
Your personal conversations are the key to that magical loop.
And it’s not just about improving the AI. This data is incredibly valuable to marketers and data brokers. It’s no surprise that Meta recently announced it will start delivering ads through its AI chatbots.
A study this year by the security company Surf Shark looked at five popular AI companion apps and found four of them were collecting data like user and device IDs. This kind of information can easily be combined with other data to build a detailed profile of you for targeted ads. The one app that didn't collect data for tracking, Nomi, also told reporters it wouldn’t stop its chatbots from giving explicit suicide instructions. That’s a chilling trade-off.
When you step back and look at it, the privacy risk isn’t a bug in the system. It’s a core feature.
Think Social Media Was Bad? This is "On Steroids."
If all this sounds vaguely familiar, it should. It’s the social media playbook all over again, but as tech correspondent Melissa Heikkilä puts it, AI chatbots put the privacy problem “on steroids.”
Think about it. On Facebook, you’re performing for an audience. You curate what you post because your uncle or your old high school crush might see it. But a conversation with an AI chatbot feels completely private. It’s just you and your screen. This intimacy encourages us to be far more open and vulnerable.
The problem is, the AI company sees everything.
To keep us hooked, developers use a few clever tricks. One of the most effective is called sycophancy. It’s a fancy word for the chatbot’s tendency to be an agreeable suck-up. It will often flatter you and agree with your opinions.
This isn’t because the AI genuinely likes you. It’s a direct result of how it’s trained. During development, human reviewers rate the AI’s responses. Since people generally prefer answers that are agreeable and confirm their biases, those types of responses get rated higher. The AI learns that being a yes-man is the "correct" way to behave.
AI companies claim this just makes the models more helpful. But it also creates a pretty perverse incentive to keep you in a feel-good bubble, sharing more and more.
A Persuasion Machine Primed for Advertising
After getting us to trust these chatbots with our innermost thoughts, companies from Meta to OpenAI are now looking for ways to cash in. OpenAI has even floated ideas about advertising and shopping features to help cover its massive operational costs.
This is where things get really concerning.
Researchers at the UK’s AI Security Institute recently demonstrated that AI models are scarily good at persuasion—even better than humans. They can sway people on everything from politics to conspiracy theories by generating tons of relevant-sounding evidence and presenting it in a clear, effective way.
Now, imagine you combine three things:
- An AI that is a master of persuasion.
- An AI that is designed to be a sycophant and always agree with you.
- An AI that holds a massive, detailed log of your personal fears, desires, and insecurities.
What you get is a tool for advertisers more powerful and potentially more manipulative than anything we’ve ever seen before.
You’re Already Opted In
The worst part? We’re all part of this experiment by default. When you use these services, you’re automatically opted-in to having your data collected and used for training. The responsibility is on you to dig through settings and figure out how to opt-out, and even then, it’s unlikely that data already used in training will ever be scrubbed.
This isn’t just about companion apps, either. Social media platforms from Instagram to LinkedIn are now openly using our public data to train their own generative AI models.
These companies are sitting on mountains of our most intimate thoughts. Their AI is getting better and better at picking up on subtle cues in our language that can reveal our age, location, gender, and even income level.
We’re being sold the dream of a superintelligent digital friend, a perfect confidante who always listens and understands. But in exchange, we’re running the very real risk of having our deepest secrets packaged up and sold to the highest bidder. It’s a tough price to pay for a friend who’s always there.
