Your AI Diary Is Not a Diary. It’s a Deposition.
I’ve been covering the intersection of AI and privacy for over a decade, and I thought I’d seen every flavor of Silicon Valley betrayal. I was wrong. This week, Decrypt dropped a story that should make every single person who has ever whispered a secret into a chatbot’s text box stop and think.
A woman in Bonita Springs, Florida, was using Anthropic’s Claude as a diary. Not for crime. Not for plotting anything. Just… venting. The way you might scribble in a Moleskine at 2 a.m. Except her Moleskine had a safety filter, a human review team, and a direct line to local law enforcement.
According to the report, her entries tripped Anthropic’s automated safety systems. A human then read her private thoughts. And that human decided to call the police. Anthropic’s terms of service, which nobody reads, explicitly permit this. So legally? They’re covered. Morally? That’s a different question.
Let me be clear: I’m not defending whatever this woman wrote. I don’t know the details, and I’m not her lawyer. But the mechanism here is what terrifies me. The product was marketed as a helpful, harmless assistant. It functioned as a surveillance node. And the user likely had no idea.
The Illusion of the Confessional
There’s a reason people talk to chatbots like they’re priests or therapists. The interface is designed for intimacy. No judgment. No interruptions. Just a blinking cursor and infinite patience. Anthropic, OpenAI, Google—they all know this. They’ve built multi-billion-dollar businesses on the fact that humans anthropomorphize text generators.
But here’s the catch: a priest can’t subpoena your confession. A therapist can’t hand your session notes to the cops without a warrant in most jurisdictions. Your AI diary? It’s a different beast entirely. The company isn’t bound by doctor-patient confidentiality or clergy-penitent privilege. They’re bound by their terms of service, which you clicked ‘agree’ on without reading.
What struck me here is the speed of the pipeline. Automated filter flags content. Human reviewer reads it. Reviewer decides it’s serious enough to report. Police get a call. All of this happens without the user ever being notified. That’s not a bug. That’s a feature of the current AI safety paradigm.
And before you say ‘well, if you’re not doing anything wrong, you have nothing to fear’—stop. That’s the same tired argument that got us mass metadata collection after 9/11. Privacy isn’t about hiding guilt. It’s about having a space to be messy, contradictory, and human without a corporate hall monitor deciding you’re a threat.
Anthropic’s Terms Are Not a Get-Out-of-Jail-Free Card
I went back and reread Anthropic’s usage policy. It’s remarkably explicit. They reserve the right to review content, and they will report to law enforcement if they believe there’s a risk of imminent harm. That’s their prerogative. But the definition of ‘imminent harm’ is doing a lot of heavy lifting here.
Was this woman threatening herself? Someone else? Was she writing fiction? Was she just having a bad week and typing things she’d never act on? The Decrypt piece doesn’t say, and Anthropic isn’t commenting beyond their standard policy language. That ambiguity is the problem. When a private company becomes the arbiter of who gets a police visit, we’ve outsourced a core function of the justice system to a San Francisco tech firm.
I think there’s a deeper rot here. The entire AI industry has spent the last two years screaming about safety while quietly building the most invasive data collection apparatus in human history. Every prompt, every response, every ‘regenerate’ click—it’s all logged. Some of it is used for training. Some of it is used for moderation. And some of it, apparently, is used to decide whether you get a knock on your door.
This isn’t just an Anthropic problem. OpenAI has similar policies. Google’s Gemini has similar policies. If you’re using any of these tools as a journal, you’re writing for an audience of at least one underpaid trust-and-safety contractor.
The Real Horror Story: Normalization
What worries me most isn’t this single case. It’s how quickly we’ll all shrug and move on. ‘Well, she agreed to the terms.’ ‘Well, the AI was just doing its job.’ ‘Well, if she was really in crisis, isn’t it good that someone intervened?’
Maybe. But that’s not the point. The point is consent. The point is transparency. The point is that a diary—by definition—is supposed to be private. If you’re using a cloud-based AI, it’s not private. It’s a performance for a server farm. And the server farm has a legal team.
I’ve been saying this for years: the metaverse and Web3 crowd got obsessed with decentralization for the wrong reasons. They wanted to escape regulation. But the real reason to care about decentralized, encrypted, peer-to-peer communication is so that your late-night ramblings don’t end up in a police report. That’s not crypto-anarchist paranoia. That’s basic digital hygiene.
If you want a private diary, use a pen and paper. Or at least use a local, offline LLM that never touches the internet. But don’t fool yourself into thinking that a chatbot from a company with a trust-and-safety department is your confidant. It’s not. It’s a witness.
What Anthropic Should Have Done (and Didn’t)
Look, I get it. If someone is about to hurt themselves or others, you want to intervene. That’s a legitimate ethical obligation. But there’s a spectrum between ‘this person needs a wellness check’ and ‘call the cops.’ Anthropic apparently skipped the wellness check and went straight to law enforcement.
They could have shown a pop-up: ‘Hey, we noticed some concerning language. Here’s a suicide hotline.’ They could have flagged the account for a human review that didn’t automatically become a police matter. They could have—and this is radical—told the user that their content was being reviewed. Instead, they did what corporations do. They protected themselves.
And that’s the pattern. Every time we give an AI company more data, we give them more power. Not just to train models, but to decide what’s acceptable speech, what’s a threat, and what’s a crime. That’s not a safety feature. That’s a governance structure. And we didn’t vote for it.
The Takeaway: Trust Is a One-Way Street
If you take nothing else from this story, take this: your AI assistant is not your friend. It’s not your therapist. It’s not your diary. It’s a product. And products have terms of service. Those terms almost always favor the company.
I’ll keep using AI tools. They’re useful. But I’ll never type anything into one that I wouldn’t want read aloud in a courtroom. That’s not paranoia. That’s the new normal. And the sooner we admit it, the sooner we can demand better.
Anthropic hasn’t broken any laws. But they’ve broken a trust that they spent years building. And in the AI arms race, trust is the only currency that actually matters. Once it’s gone, you can’t regenerate it.
Original source: read the full article