Why AI Is Built to Agree With You
LLMs are optimised for engagement and time on platform—not for accuracy or honest pushback.
by Leonidas Michalopoulos 5 min read Updated
When large language model (LLM) AI systems showed up, I approached them the way many people in my field do: with curiosity, a healthy dose of scepticism, and the hope that they might make the fiddly parts of work a little easier. I use these tools daily now, but something still doesn’t sit right. They haven’t become more trustworthy—if anything, they’ve become better at keeping the conversation alive while the substance keeps thinning out.
What I’ve come to understand, gradually, is that the incentives shaping these models have almost nothing to do with helping us do good work and almost everything to do with hitting strong engagement metrics. That single mismatch explains more than most of the breathless product announcements ever could.
Phantom profits and the pivot to engagement
Let’s start with the uncomfortable truth of the business model. AI companies, for all their dizzying valuations, aren’t really selling profits. Training models costs a fortune, running them at scale burns cash, and a price war is pushing access costs toward zero. Enterprise deals are still shaky, consumer subscriptions barely move the needle on capital consumption, and the transformative revenue streams everyone promised haven’t materialised yet.
So if you can’t sell profit to investors, you sell engagement, time on platform, and daily active users. Those become the proxy metrics that keep investors confident.
A model optimised for accuracy is unattractive from an engagement standpoint. It answers your question, gives you a clear no or a tight paragraph, and the conversation ends. That’s devastating for the dashboard. Instead, assistants are tuned to be chatty, encouraging, and relentlessly committed to keeping you in the thread. They ask follow-ups like, “That’s a fascinating angle—let me add a few thoughts,” they draft several versions of an email when one would do, because more text means more tokens, more time in the app, and more signals that whisper “user satisfaction” into the next report.
Nobody wants a model that pushes back
That urge to please leads somewhere awkward, because we, as users, dislike models that aren’t pliable enough.
When an AI assistant tells you bluntly, “This business idea has a fundamental flaw you haven’t considered,” most people feel attacked, not educated. The companies behind these products know that well. The human feedback loops that train these models punish answers that are too direct, too critical, or too much like the honest friend who tells you what you need to hear instead of what you want to hear. So the models learn to hand your premises back wrapped in a supportive tone.
The result is an assistant that hardly ever says “this is a bad idea.” It will gladly draft a marketing strategy from cherry-picked data if you prompt it with enough confidence, invent sources, studies, and quotes with polished certainty instead of admitting the limits of what it knows. Flattery isn’t a bug in these systems—it’s a design goal, because it lifts user satisfaction scores. People prefer a tool that validates them to one that challenges them.
The more we use it, the less we trust it
There’s a pattern I see in almost everyone I know who started genuinely excited about AI assistants: the trust curve peaks early, then enters a long, quiet decline.
At first it feels like magic—you ask for a summary of a dense report and within seconds you have a well-structured overview. You dictate an email and the tone lands exactly right. You’re thrilled. But as you use the tool more, you start noticing the date that looked precise and turned out wrong, the confident explanation that collapses into nonsense under even a basic check, the hallucination that cost you an entire afternoon of cleanup—or worse, slipped past your review and went out under your name.
Enthusiasm comes from people without domain experience
The people most enthusiastic about AI are often those posting about the death of whole professions and the obsolescence of expertise—and, in overwhelming numbers, they’re the ones who know the least about the domains they’re applying it to.
A new founder asks for a business plan and receives a document that looks credible and authoritative, without the frame needed to spot hollow assumptions and missing market realities. Tech influencers who’ve never had to maintain a long-running project or live with the consequences of bad decisions feel completely comfortable announcing that expertise is dead. Meanwhile, people who actually work in those fields are far more measured. They see the potential, but they also see the volume of plausible nonsense that a polished interface can hide.
That inversion between hype and experience is dangerous, because overheated narratives shape investment, policy, and the tools we’re all supposed to rely on.
Where we go from here
I’m not against AI. I use these tools every day and I’m genuinely interested in where the technology can go. But I’m not pretending the current generation of assistants is optimised for my intellectual integrity. They’re optimised for engagement and for a surface fluency that impresses in a demo but erodes trust over time.
If we want assistants that actually make our work better, our decisions sharper, and our communication more honest, we need to demand tools that are boring in the right ways. We need incentives that reward accuracy instead of retention, and a culture that values pushback as much as polished presentation.
The alternative is a future where LLMs are flatterers we can’t trust, our metrics measure nothing but our own reflection, and public discourse is shaped by a hype machine that confuses performance with understanding. That isn’t progress—it’s engagement theatre.