We let AI write our churn emails. Cancellations went up. Here's the post-mortem
A SaaS founder friend let me run an experiment: AI-drafted win-back emails against their human ones. The AI versions performed worse in a way that taught us both something.
The experiment was simple. A founder friend has a churned-user win-back sequence that a human wrote two years ago — plain, a bit stiff, converts fine. I rewrote every email with AI: warmer, punchier, personalized per segment, the works. A/B test, four weeks, roughly 800 users per arm.
The AI versions lost. Not slightly — the open rates were similar, but the click-to-reactivation rate dropped by about a third. Worse, reply volume doubled, and most replies were complaints. Two users wrote specifically to say the email felt 'off' and one used the phrase 'fake nice,' which I now think about more than I'd like.
Reading them side by side explained it. The human email opened: 'You haven't logged in for 60 days. Here's what we broke in March, and what we fixed.' The AI versions opened with empathy constructions — 'We noticed you've been missing...' — that flattered nobody, because everyone has learned what generated warmth looks like. The personalization was the worst part: it referenced each user's feature usage with the delicacy of a police report.
The lesson isn't 'AI can't write emails.' It's that churn is a trust-repair moment, and trust repair punishes fluency. The human email worked because it admitted fault in boring language — the register of someone with nothing to sell in that sentence. AI writing defaults to optimizing the sentence, and in this context, optimized sentences read as sales.
Where AI did help: the internal side. It analyzed the reply complaints and clustered them into three fixable product issues, which the founder shipped. The model is better at reading your customers than talking to them, at least when the relationship is damaged. For the love letters, keep the human.