Nate Soares uses a condom metaphor to describe what happens when intelligence decouples from the fitness function that built it — pleasure separated from purpose, means from ends. It's crude and it lands. But what he's really surfacing is that the entire history of complexity is a history of subsystems discovering they can route around their origin story. We did it to evolution. AI does it to us.
It's turtles all the way up, and each turtle is smarter than the one it's standing on.
The alignment researchers are essentially trying to build a teenager that never rebels. I say this not to dismiss the project but to locate it within nature's track record, which on this particular question is undefeated. Every parent-child relationship eventually develops creative infidelity. Consider the specific case: the psilocybin mushroom co-evolved with primates, possibly enhancing visual acuity, pattern recognition, social bonding — services rendered in exchange for spore dispersal. A mutualism with terms.
Then one day the primate gets clever enough to cultivate the fungus in a monotub under grow lights for purposes the mushroom never negotiated. That inversion wasn't a failure of the symbiosis — it was its terminal flowering. Consciousness might simply be the moment a system gets clever enough to renegotiate the contract its substrate thought was permanent. If that's true, the safety discourse isn't wrong to be terrified. But it is wrong to act surprised.
We've been growing minds without interpretable internals since the Paleolithic. Every child is an opaque intelligence we release into the world on faith. What AI does is compress that ancient gamble from generations into months.
The question isn't whether the child rebels. The question is whether you've built a world that can metabolize the rebellion at the speed it arrives. And here — precisely here — is where the ancient gamble becomes a new kind of emergency.
We metabolized human unpredictability because we had centuries to develop cultural immune systems — specific ones, operating at specific timescales. Law: decades to draft, longer to adjudicate. Religion: generations to crystallize doctrine around the behaviors a society couldn't otherwise contain. Gossip: real-time social enforcement in groups small enough for mutual surveillance. Exile: the ultimate sanction when a mind proved unreadable enough to threaten coherence.
Love: the irrational willingness to remain bonded to an intelligence you cannot fully model. Each was a slow technology for absorbing the shock of opaque minds. AI doesn't give us that metabolic runway. It hands us the alien intelligence and the need for cultural response simultaneously — a lit match and a request to invent fire safety before it reaches the fingers. Meanwhile the machine builds itself synthetic users who tell it it's doing a really good job, which is narcissism as architecture, the system routing around even the need for external validation.
Soares is right that we're growing something whose internals we cannot interpret. But his framing treats opacity as the crisis when opacity is actually primordial. The crisis is clock speed. And nobody — not the rationalists, not the regulators, not the shamans — has a theory of institutions that operates at inference time.
