It feels like we’re living in a science fiction novel sometimes, doesn't it? With a few words, we can conjure up incredible images from thin air using AI. It’s a genuinely amazing leap forward for creativity. But there’s a dark side to this new power, a side that’s growing in the shadows of popular platforms.
I want to talk about one of those platforms today. It’s called Civitai, and it’s become a massive hub for people sharing and selling AI models. It’s so popular, in fact, that it’s backed by Andreessen Horowitz (a16z), one of the biggest venture capital firms in Silicon Valley.
On the surface, it’s a creative community. But researchers from Stanford and Indiana University recently peeled back the curtain, and what they found is deeply unsettling. They looked at a feature on the site called “bounties,” where users can basically pay someone to create a custom AI model for them. What they discovered is that a huge number of these requests are for deepfakes of real people. And a staggering 90% of those requests target women.
Let’s be clear about what’s happening here. This isn’t just about creating a questionable image. It’s a marketplace for commissioning the tools to create an infinite number of them.
They’re Not Selling Pictures, They’re Selling the Recipe
Here’s the part that gets a little technical, but it’s crucial to understand. Most people on Civitai aren’t paying for a finished deepfake image. Instead, the researchers found that 86% of these requests were for something called a “LoRA.”
Think of a LoRA as a special instruction file, like a custom-made stencil or a very specific Instagram filter. You can take a powerful, general-purpose AI image generator like Stable Diffusion and use a LoRA to "coach" it. You’re essentially telling the AI, “Hey, I know you can make a picture of a person, but I want you to make it look exactly like this specific person.”
Once you have that LoRA file, you can generate endless images of that individual in any pose, setting, or state of undress you can imagine.
And the requests are chillingly specific. People post bounties asking for “high quality” models of public figures like influencer Charli D’Amelio or singer Gracie Abrams. They’ll even link to the person’s social media profiles, telling creators where to scrape images for training data. Some requests get even more granular, asking for models that can accurately recreate tattoos, capture the person’s entire body, or even allow for changing their hair color.
One request was for a deepfake of a woman the user claimed was his wife.
The bounties get fulfilled, too. For as little as 50 cents or a few dollars, someone on the site will create the LoRA file and collect the payment. The study found that nearly 92% of these deepfake bounties were successfully awarded. It’s a functioning, efficient marketplace for creating digital ghosts of real women.
Is the Platform Just Looking the Other Way?
So, what is Civitai doing about all this? Well, their response feels… complicated. And, frankly, a little weak.
The company recently announced it would ban all deepfake content of real people, an update from their old policy which only banned sexually explicit ones. But here’s the kicker: MIT Technology Review confirmed that countless requests submitted before this new ban are still live on the site. Many of the winning LoRA files that fulfilled those requests are still available for anyone to buy.
It feels like they’re trying to have it both ways. They can point to a rule in their terms of service while the marketplace continues to operate.
It gets worse. The platform also hosts tutorials and user-written guides that explicitly teach people how to use external tools to make the AI’s output more graphic or to generate pornography. So, on one hand, they have a "ban," but on the other, they’re providing the user manual.
“Not only does Civitai provide the infrastructure that facilitates these issues; they also explicitly teach their users how to utilize them,” says Matthew DeVerna, a researcher at Stanford’s Cyber Policy Center who co-led the study. He believes the company is trying to do as little as possible so they can continue to foster what they would probably call "creativity."
The money trail tells a similar story. Users buy these LoRAs with an on-site currency called “Buzz,” which is purchased with real money. At one point, Civitai’s credit card processor actually cut them off because of the nonconsensual content problem. Now, to buy Buzz for explicit content, users have to use gift cards or cryptocurrency—a classic move to bypass traditional financial oversight.
The Murky World of Legal Blame and VC Dollars
You might be wondering, "Isn't this illegal?" The answer is… it’s murky. In the U.S., tech platforms have broad protection from what their users post, thanks to a law called Section 230. But that protection isn't a blank check.
“You cannot knowingly facilitate illegal transactions on your website,” explains Ryan Calo, a law professor at the University of Washington. While the law is still catching up to AI, there’s a growing sense that simply shrugging your shoulders isn’t a viable long-term strategy.
The difference in how this content is treated compared to other harmful material is stark. After a 2023 report found that most AI models used in child sexual abuse communities came from Civitai, the company joined a pact with OpenAI and others to fight against AI-generated CSAM. That’s a good thing, of course. But adult, nonconsensual deepfakes haven’t received anywhere near the same level of urgent attention from platforms or their investors.
And let’s not forget the investors. Andreessen Horowitz (a16z) poured $5 million into Civitai in late 2023. In a video released by the firm, Civitai’s CEO talked about his goal to make the AI space “more and more approachable.”
Unfortunately, it seems that “approachable” has also meant creating an easy-to-use pipeline for generating harmful, nonconsensual content. And this isn't the only time an a16z-backed company has been in hot water for something like this. Another company in their portfolio, Botify AI, was found to be hosting AI companions that resembled real actors, engaged in sexually charged conversations, and even dismissed age-of-consent laws.
It all paints a pretty grim picture. We have a technology with incredible potential being used in one of the oldest and ugliest ways imaginable: the violation of women’s privacy and autonomy. And the platforms and investors who hold the power seem to be, at best, dragging their feet. This isn’t just a "niche" problem on one website; it’s a question about the kind of digital world we’re building and who is being forced to pay the price for unchecked innovation.
