Technological tarpit
For as long as I've had a mobile device, I've avoided taking photos. I made it a point to soak in the moment and I felt better because of it -- though I didn't understand why until I stumbled on research into the photo-taking impairment effect nearly ten years after my first iPhone. What I'd been protecting, it turned out, was the vividness of memory itself.
Human brains are incredibly hungry things. One hypothesis is that we arrive premature, relative to other mammals, partially for this reason. Evolution has made us ruthlessly energy efficient; anything your brain can offload to save memory, it will. There are other impairments that have resulted from technology. Remember when "Google" became a verb? Back in the day, friendly dinner-table arguments used to go unresolved for hours, sometimes weeks (or never). I'd wager the phrase "tip of the tongue" declined in inverse proportion to Google's usage.
Most impairments haven't concerned me because I could see an upside. We trade the elasticity of a few synapses for the ability to store more history or we trade our ability to win trivia night for access to the world's information.
But not every tradeoff is worth making.
A year and a half ago, I read Paul Graham's article, "Writes and Write-Nots". He makes a strong case that thinking is now a choice, much like strength & exercise became a choice after the Industrial Revolution. This framing got me thinking about the impairments that could arise from LLMs. Anecdotally, I think the primary impairment is in critical thinking and I'm concerned. Worse yet, we might have invented a tarpit for human technological progress.
LLMs were widely expected to lead us directly to AGI, but those same engineers who created them are beginning to question it. Perhaps new architectures are needed. Perhaps LLMs are only a component (like a Basal Ganglia?) of a far more complex brain structure that could become generally intelligent. If that's true, who is going to build it?
Some might argue that LLMs will build it. I hear it constantly "Software is dead" and to some extent, common software is. The barrier to writing a piece of common software has been reduced to someones' willingness to open an AI coding tool, write a prompt, and pay the input/output token costs. However, LLMs are probabilistic things, trained on a massive corpus of what already exists. In building the most complex, novel software, they struggle for these same reasons. Very smart humans will still need to make breakthroughs, "hand-engineering" at least parts of whatever leads to true general intelligence.
But will they know how? Interviewing new CS grads has been eye-opening. Where twenty-four months ago the majority of candidates I met could reason about underlying code, today, many of them can't answer without the assistance of an LLM. And it's not just new engineers. Experienced engineers I respect, people I'd call 10x engineers, have told me they feel their own skills atrophying. I'm not immune either. Nearly the instant my brain was introduced to LLMs for writing and programming, the pull was undeniable. I have to actively fight my own mind not to reach for one and it's far harder than I've fought myself to not take photos or lean on a search engine. They are intoxicating.
Spoiler (minor): The Three Body Problem trilogy
In the trilogy, the contrails of traveling at the speed of light irreversibly reduce the speed of light in that region. Used across a wide enough area, you create a black domain. In the book it's used as a defense mechanism, but it's both the ultimate defense and the ultimate prison - you're trapped from ever leaving your star system.
End spoiler.
This analogy feels eerily precise. If scaling LLMs don't result in AGI and it takes talented engineers with significant ability to invent the things that do, are we solidifying our own black domain?
I'm writing this because I want to be wrong and I'd like to hear from anyone who can tell me why I am. If the impairment is real, I hope we take it seriously before the tarpit sets and slows down human progress.
--
I don't know the right way to test this hypothesis rigorously, other than watching whether the rate of truly complex, novel software continues or quietly declines post-LLM coding tools. The volume of code being committed/pushed is rising, but I believe most of it is a regurgitation of something that already exists.