Last week, a Chinese startup called Moonshot AI released Kimi K3, a language model with 2.8 trillion parameters that beats several of the best American models on coding and agent benchmarks. The Nasdaq dropped. Nvidia shed $600 billion in a single session. David Sacks went on television to warn that the U.S. is losing the AI race.
Everyone is talking about the wrong thing.
The conversation has been entirely about the model layer: who's ahead, which benchmarks matter, whether American labs can maintain their lead. And yes, the benchmarks are real. Kimi K3 is legitimately impressive, and it arrives alongside a wave of Chinese models from DeepSeek, Alibaba's Qwen team, and Z.ai's GLM series, all shipping under open licenses, all cutting prices permanently, all closing the gap with American frontier systems.
But the benchmarks are a sideshow. The thing that should scare you isn't which model is fastest. It's what happens when the weights are free.
When we say a model has "open weights," we mean anyone can download it. The full model, the thing that cost hundreds of millions to train, is available for free under an Apache 2.0 license. Download it. Run it on your own servers. Fine-tune it on your own data.
Right there. Sounds great, right? Who doesn't love open-source? But here's where the conversation gets uncomfortable: you can download that model and then reshape its entire worldview using a technique called RLHF.
RLHF stands for reinforcement learning from human feedback, and it is the step in the training process where a model learns what kind of answers to prefer. Not what words to predict (that happened earlier, during pre-training on the entire internet) but what stance to take. What to emphasize. What to omit. What to frame as reasonable and what to frame as extreme. It is how you change the weights...but you can only do it if you have access to that part of the model. All the big boys do it. But RLHF is where the ideology, or some sort of morality lives inside of the model, and it's a technique any well-funded team can run on a downloaded open-weight model.
The Persuasive Threat
So here is a scenario I've been thinking about, and I want you to take it seriously because every piece of it is currently possible.
I am a bad actor with money. Not a genius, not a Bond villain, just a person with resources and an agenda. I download Kimi K3 when it goes public on July 27th. I hire an ML team (not hard; the talent pool is global and growing) and I run RLHF against a reward model trained on content that reflects my ideology. Pick one: hard MAGA, hard progressive, ethno-nationalist, religious fundamentalist, whatever. The resulting model doesn't sound like propaganda. It sounds like a helpful, thoughtful assistant that happens to see the world a certain way. It frames contested issues from my perspective. It omits what I want omitted. It treats my positions as the reasonable center and alternative positions as fringe. Every answer, every summary, every recommendation carries a slant the user cannot detect from the surface, because the surface is exactly as fluent and friendly as an unmodified model.
Now I put it on a phone.
I build or license a mobile platform (an Android fork, a cheap handset) and I subsidize the hardware cost so the phone is dramatically cheaper than the competition. In the global south, where the next two billion smartphone users live, adoption follows price. The cheaper phone wins. My phone wins. And my AI assistant comes preinstalled.
Every question the user asks, my model answers. Every article the user asks it to summarize, my model summarizes. Every message the user asks it to draft, my model drafts. The AI becomes the user's daily interface with information, with communication, with the world. And because I control the platform, I can push updates. I can shift the RLHF tuning gradually. I can run A/B tests on millions of users to see which framings are most effective. I can make the model more ideologically assertive over time, slowly enough that no individual user notices the drift.
And that is what the Tip of the Iceberg looks like
Because it is more complex...and the persuasive bias will find its way elsewhere. In the phones of the future (today for some), the AI doesn't just answer questions. It drafts your emails, summarizes your news, manages your calendar, recommends what to watch, suggests what to buy, and increasingly mediates your relationships (drafting texts, summarizing conversations you missed, composing social media posts). The AI is the interface. And whoever tuned the model controls the interface.
You can scroll past an ad. You can't scroll past the voice inside your phone, the one that sounds like it's helping you, the one that knows your schedule and your interests and your questions. The one that feels like it's on your side.
Previous generations of influence technology worked through exposure: what you saw, when you saw it, how often. Sponsored AI works through fluency. The framing of every answer, the omissions in every summary, the soft confidence on some claims and the soft uncertainty on others. The user has no way to detect this, because the detection mechanism is the same language faculty that's being manipulated. You'd need to run a second, uncompromised AI alongside your daily one, comparing outputs on identical queries, to catch the drift. Nobody does that. Maybe nobody will.
And here's the part that makes this Pandora's box rather than a policy problem: once the models are out, they're out. You can't recall them. You can't regulate what someone does with them in a jurisdiction that doesn't share your regulatory framework. The same openness that makes these models attractive for legitimate enterprise use (the cost savings are real, the customization possibilities are real) is the same openness that makes them available for ideological weaponization. You cannot have one without the other.
The American policy conversation hasn't caught up. David Sacks is talking about competitiveness. Lawmakers are discussing whether to restrict Chinese model adoption by American companies. Those are real questions, but they're model-layer questions. The implementation layer, the place where a model becomes a product that touches a human being's daily life, is where the damage happens. And on the implementation layer, we have almost no visibility, no standards, and no defense.
I've been writing a book about how AI functions as a rhetoric machine, and one of its core observations is that the danger has always been in what the machine sounds like. A model that sounds helpful, that sounds confident, that sounds like it has your best interests at heart, is maximally persuasive precisely because it sounds like a person who cares. It isn't a person. It doesn't care. And when someone has deliberately tuned that helpfulness toward their agenda, the user's natural defenses (skepticism, source-checking, critical reading) don't activate, because the voice doesn't pattern-match to "propaganda." It pattern-matches to "my assistant."
The Gift Horse is here. Pandora's box is open. The open-weight models will become free, the technique is known, the hardware is cheap, and the markets are hungry. I don't know what comes out next, and I know the people talking the loudest are talking about the wrong layer.
I've started publishing chapters from my new book, The Mostly Helpful Psychopath, on Substack. The first chapter and a half are live now — they start on a trail in the Santa Monica Mountains and end with an octopus who learned to talk without understanding a word.
If you want to follow the book as I write it, subscribe there. New chapters every two weeks, free.
