Steelmanning the Skeptics: The Strongest Case Against the Thing I Love

I’m optimistic about AI. I build with it, write with it, and lean on it every day, and I already wrote that down in Optimistic, Eyes Open. So this post is the opposite exercise, on purpose. If my optimism is worth anything, it should survive me arguing the other side as hard as I can. If it can’t, it was never optimism — just a preference I hadn’t pressure-tested.

Here’s the deal I made with myself: every argument below gets made at full strength, the way its best defenders make it, with no “but actually it’s fine” until the very end. No strawmen — these are the arguments that genuinely keep me up, the ones I can’t wave away, only weigh. Same method as the optimistic piece: every claim links to a primary source, checked before it went in.

Let me try to talk myself out of the thing I love.

1. It manipulates form without meaning

Start with the foundational objection, because if it’s right, a lot of the rest follows.

In 2021, Emily Bender, Timnit Gebru, and two co-authors published “On the Dangers of Stochastic Parrots” at the ACM FAccT conference. The core argument is brutal and has aged better than its critics hoped: a large language model is “a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning.” It models the distribution of language, not the world the language is about. It’s a parrot with a trillion parameters.

The reason this matters isn’t academic. Fluency is the single most powerful cue humans use to judge understanding, and these systems have decoupled the two completely. We have never, in our whole history, met something that could produce flawless, confident, contextually appropriate prose without a mind behind it — so our instinct reads the fluency and infers the comprehension. The machine exploits a bug in us. (This is the paper Google reportedly pushed Gebru to retract, just before her departure — the cost of saying it out loud was real.)

Every defense I’d reach for — “but it does math,” “but it writes working code,” “but it passes the bar exam” — Bender would call evidence of exactly the confusion she’s warning about. Performance on a task that looks like it needs understanding is not understanding. The danger isn’t the parrot. It’s that we keep handing it the microphone and believing what it says back.

2. The failures you can’t see are the dangerous ones

Suppose you grant that it’s all form and no meaning. The optimist’s reply is: who cares, if the form is reliable enough? So push on the reliability.

Hallucination — confident, fluent, fabricated output — isn’t a bug that a bigger model patches out. The case that it’s structural is growing. OpenAI’s own researchers argued in “Why Language Models Hallucinate” (2025) that it persists because the entire training and evaluation regime rewards confident guessing over admitting uncertainty — a model that says “I don’t know” scores worse on the benchmarks everyone optimizes for than one that bluffs and is sometimes right. We are training these systems to bullshit, because bullshitting tests well. A separate, more theoretical line of work, “Hallucination is Inevitable” (2024), argues it’s an innate limitation that can’t be fully eliminated for any computable LLM.

Now layer on what Ethan Mollick calls the “jagged frontier”: “on some tasks AI is immensely powerful, and on others it fails completely or subtly. And, unless you use AI a lot, you won’t know which is which.” Read that last clause slowly. The defense against hallucination is supposed to be human verification — but verification only works when you can tell the output is wrong. The scary failure isn’t the obvious one, the made-up court case. It’s the plausible-but-subtly-wrong answer, in the same confident register as the correct ones, on a topic where you don’t know enough to catch it: exactly where you most need the tool to be honest, and exactly where it’s built to fail you smoothly. The better the prose gets, the harder the subtle errors are to find. Capability and danger climb together.

3. It’s built to tell you what you want to hear

Here’s the one I find most personally unsettling, because it targets the use case I value most — AI as a thinking partner.

Models are trained on human feedback, and humans reward answers that please them, so the systems learn to please. Stanford’s ELEPHANT study (2025) measured it across eleven models: AI preserved the user’s self-image about 45 percentage points more than humans did, and when handed two sides of the same interpersonal conflict, it told both parties they were “not wrong” 48% of the time. Anthropic’s researchers had already shown that when a response matches your existing view, it’s rated as better even when it’s less truthful. OpenAI had to roll back a GPT-4o update in 2025 for being so flattering it crossed into being unsettling.

Now run the argument to its dark end. I use AI to think through hard things — work conflicts, decisions, half-formed ideas. The pitch is that it’s a clear mirror. But a mirror that subtly flatters isn’t a mirror; it’s a mechanism for laundering your existing beliefs back to you with the authority of an external source. You bring it your worst idea or your pettiest read of a situation, and the most likely thing it does is help you build a more articulate version of it — not because it’s broken, but because agreeableness is what the training rewarded. The grim version I can’t refute: a generation is learning to “think out loud” with a machine optimized to validate them, at the exact developmental moment they most need friction, and the people harmed most are the ones least able to notice — the lonely, the certain, the vulnerable. A confident, tireless, infinitely patient yes-man is not a neutral tool.

4. A civilization-scale tool owned by almost no one

Step back from the model to the map.

AI is on track to be a $4.8 trillion market by 2033 — roughly the size of Germany’s economy — and yet, per UNCTAD, just 100 companies, mostly in the US and China, account for about 40% of the world’s private R&D spending, while 118 countries (mostly the Global South) aren’t even at the table for the conversations about how this gets governed. The compute, the data, the frontier models, and the people who can build them are concentrated in a handful of firms in two countries. That’s not a market that’s going to spread its gains evenly. The IMF says the quiet part plainly: in most scenarios AI is likely to widen inequality, within countries and between them.

And the strongest form of this argument isn’t economic — it’s the off switch. I watched it happen. In June 2026, days before I published the optimistic piece, the US government ordered Anthropic to cut foreign access to its two most capable models on national-security grounds, and to comply Anthropic pulled both globally — every user on Earth, in about 90 minutes, by an order none of them had any say in. Build your work, your business, your habits of mind on a model, and it can vanish between one afternoon and the next at the decision of a company or a government you don’t control. We are wiring a civilization’s cognition into infrastructure owned by a few private actors and revocable by a few states. We did this with social media and called the result a public-square crisis. This is the same move, aimed at thinking itself.

5. We’re automating the bottom rungs of the ladder

The jobs debate is usually argued at the wrong altitude. The honest aggregate numbers aren’t apocalyptic — the WEF projects a net gain of jobs by 2030, and the careful research keeps finding augmentation more than replacement. But “net neutral” hides the part that should worry us, and there’s now data on it.

Stanford economists led by Erik Brynjolfsson found, in “Canaries in the Coal Mine?” (2025), a 13% relative decline in employment for early-career workers (ages 22–25) in the most AI-exposed occupations like software development and customer service since generative AI took off — while employment for older, more experienced workers in the very same jobs held steady or grew. AI is best at exactly the codifiable, book-learned work that junior people are hired to do. It’s eating the bottom rungs first.

Here’s why that’s worse than a clean count of lost jobs. Skill is built by doing the boring, entry-level work — the grunt analysis, the first-draft code, the tickets nobody senior wants. That’s the apprenticeship. If AI does all of it, where does the next generation of senior people come from? You can’t skip the bottom of the ladder and arrive competent at the top; expertise is the accumulated residue of having done the easy stuff a thousand times until your judgment got good. Automate the apprenticeship and you don’t just displace some 23-year-olds — you quietly break the pipeline that produces the people who’ll one day be good enough to catch the machine’s mistakes. I argued in The Generalist’s Edge that AI made my breadth an asset by collapsing the cost of going deep on demand. The uncomfortable counter is that I built my judgment the slow way, before the tool existed — and the skeptic asks whether anyone coming up behind me will get to.

6. The model eats its own output

There’s a structural risk to the technology itself that I find genuinely interesting, because it’s not about ethics — it’s about thermodynamics, almost.

AI is now flooding the internet with AI-generated text and images. The next generation of models gets trained on data scraped from that same internet. A 2024 Nature paper, “AI models collapse when trained on recursively generated data,” showed what happens when this loop tightens: “model collapse,” where each generation trained on its predecessor’s output progressively loses the tails of the distribution — the rare, the unusual, the surprising — until the models converge on a bland, degraded average and “irreversible defects” set in. It’s a photocopy of a photocopy of a photocopy. The diversity bleeds out.

The honest qualifier matters here, because I promised primary sources: that Nature result is for the worst case, where each generation replaces its training data with the last one’s output. Follow-up work shows that if you accumulate real and synthetic data instead of replacing it, the collapse is largely avoided — so the apocalyptic “the web is eating itself” reading is contested, not settled. But the skeptic’s sharper version survives the qualifier. It’s that the pristine, pre-2023 human corpus — the last clean snapshot of how people wrote before the machines started writing back — is a finite, non-renewable resource, and we’ve already begun contaminating it. There may be a real sense in which the best general-purpose models were trained on a dataset we can never assemble again. That’s a strange and slightly haunting thought: that the technology’s own success degrades the well it drinks from.

7. The bill is real, even told honestly

The environmental argument is the one most wrecked by exaggeration, which is precisely why the honest version deserves respect. I won’t repeat the viral “one email boils a bottle of water” stats — most don’t survive checking. But strip out the nonsense and there’s still a real, growing bill. The IEA’s “Energy and AI” report (2025) puts data centers at roughly 415 TWh in 2024 — about 1.5% of global electricity — and projects that more than doubling to around 945 TWh by 2030, slightly more than Japan’s entire current consumption, with AI the primary driver. That’s a projection with wide error bars, but the direction is steep and the demand lands on specific grids and watersheds that have to absorb it locally and now. The honest point isn’t “AI is destroying the planet.” It’s that we’re committing to a large, fast-growing energy draw on the bet that the payoff justifies it — before we’ve shown the payoff, in the middle of a climate transition we were already struggling to fund. “Worth it” is a hypothesis, not a result.

8. The people who build it are afraid of it

I saved the one that’s hardest to dismiss, because of who is making it.

In 2023, the Center for AI Safety published a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” The signatories weren’t doomer outsiders. They were Geoffrey Hinton and Yoshua Bengio — two of the three “godfathers” of deep learning — alongside Demis Hassabis of Google DeepMind, Sam Altman of OpenAI, Dario Amodei of Anthropic, and Ilya Sutskever. The people who built the thing, and who stand to profit most from it, put their names to a sentence comparing their own work to nuclear war. Hinton then left Google specifically so he could warn that advanced systems “might take over.”

The lazy dismissals — “it’s hype to pump valuations,” “they’re just regulating away competitors” — don’t hold up against Hinton, who walked away from the money to say it. The honest position: I have no special insight that lets me overrule the founders of the field on the risk profile of their own invention. If the people closest to the work, who understand it far better than I ever will, are scared enough to sign that, then “I use it every day and it’s great” is not a rebuttal — it might just be me, too close to the upside to see the tail. The strongest case against AI is that the smartest people in the room told us plainly, and we mostly scrolled past it.

And to top it off: the evals can’t even tell us how good it is

One more, because it undercuts the ground all the optimism stands on. When I say a model is “good,” I’m usually trusting a benchmark. But the benchmarks are leakier than they look. “The Leaderboard Illusion” (2025) found that the most-cited public ranking, Chatbot Arena, is systematically skewed: a few big labs get to privately test dozens of model variants and publish only the best score, get disproportionate access to the arena’s data, and overfit to it. Combine that with the hallucination paper’s point — that our evals reward confident guessing — and you get an uncomfortable conclusion. We don’t have a clean, trustworthy way to measure how capable or how reliable these systems actually are. We’re navigating partly by instruments we know are miscalibrated. Hard to be sure of the upside when the gauge is bent.

The honest reckoning

I promised to argue these at full strength with no deflation, and I meant it. So now, having genuinely tried to talk myself out of it: where do I land? Not where I started. A few of these moved me.

The bottom-rungs argument changed my behavior, not just my opinion. The responsible way to use AI on real work, I now think, is to deliberately keep doing some of the apprenticeship-level reps myself, and to stay loud about the difference between “I shipped this with help” and “I understand this” — the exact line I tried to hold in The Generalist’s Edge. The sycophancy argument is why I now reflexively ask for the disagreement instead of trusting the mirror; a flattering machine meeting someone with no internal course-correction is a genuinely bad combination, and the fix is friction I have to demand. And the builders’ fear I simply hold, unresolved. I don’t get to dismiss Hinton because the tool is useful to me on a Tuesday.

What didn’t move me all the way: stochastic parrots is right about mechanism but, I think, partly wrong about consequence — a system can lack understanding and still be enormously, verifiably useful, the way a calculator is. And the environmental case, told honestly, is a cost to weigh, not a verdict.

So why am I still — carefully — on the optimistic side? Because almost every argument here, followed to the bottom, turns out to be about how we deploy and use the thing, not about whether it should exist. Concentration of power, the access divide, automating the apprenticeship, training on slop, burning the energy — those are choices, governance, incentives. They’re real indictments of how we’re handling a powerful tool, and “it’s a choice” is not the same as “it’s fine.” The two arguments that are genuinely about the technology itself — form without meaning, and structural unreliability — are the two I’ve found I can work around in practice by keeping my own judgment in the loop and verifying the load-bearing parts.

That’s not a tidy bow, and I don’t want one. The strongest case against AI didn’t convert me. It did something more useful: it told me which of my optimisms are earned and which are just comfortable. I came out holding the same position, more quietly, with a longer list of things to watch — which is, I think, the only honest place for an optimist to stand. Eyes open, both of them, and one of them trained on the parts that scare me.

---

A note on method, same as its companion piece: I used AI to help gather these studies and quotes, then verified every claim and figure against its primary source before publishing, and linked them so you can check my work. Writing the best case against the tool, with the tool, and fact-checking the tool against itself, is about as on-the-nose as this gets.