I keep running up against this argument about LLMs and generative AI:
It's just like a calculator; it makes work faster, but doesn't change anything about the work you're doing. This isn't anything new.
I get the instinct to normalize this invention, to place it in a box with others that came before. But every so often, humans manage to make something that defies analogy. Agriculture. The printing press. The atomic bomb. Pizza.
Of course this is hyperbole—generative models are no pizza. Nor are they an existential threat to humanity like nuclear weapons. I tend to think of them in the category of irredeemable creations like the cigarette, high fructose corn syrup, and cable news.
When you use a calculator to solve a math problem, a transaction occurs between you and the machine. You hand over the labor of computation, or arithmetic, to the device, in exchange for computation speed. This exchange is about the same in any useful technology you might imagine: the machine reduces manual labor and increases rate of output. The calculator's value is a higher solution rate for mathematical formulas.
The requisite conceptual understanding to achieve a useful outcome is neither surrendered by the user, nor of value to the calculator. You can tap on a calculator all day long, but without understanding your objective, understanding how to apply mathematical concepts to a given problem, the output of the machine is useless. And no matter how long you tap on a calculator, no matter how sophisticated the device, it cannot grasp the nature of the problem for you.
Quick aside. If you're thinking that it isn't useless in a math class where computing problem sets is the whole point, congratulations: you've just identified the core issue with the most common way of teaching math, but that's like a whole other book I don't wanna write.
And this is where LLMs fall into a different category for me. With absolutely zero domain knowledge, you can ask a generative model for output that may or may not be directly useful to the problem at hand. The result is both dangerous in its uncertain value, and because of what it takes from the user in an imbalanced exchange.
Generative models do not automate the grunt work. They steal knowledge work and replace it with a variably convincing facsimile. The output may not be accurate, but at least it was fast—a characteristic that deep, serious knowledge work never shares.
The material difference in output between calculators and generative models is its applicability. I can't take 4.8 from my calculator and turn it in as my homework by itself. But I can present the output of generative models as complete works of human thought, including legal briefs, academic research, and legislation. The artifact stands (barring moderate scrutiny) on its own, independent of any capability or understanding from the user that prompted its generation. This disconnect not only separates generative models from other assistive technology, it makes them fundamentally more dangerous to society.
Right now, model creators are releasing "deep research" products that purport to check their references, cite their work, and generally make less up than the bog-standard models. Reviews are mixed, but generally positive, insofar as the accuracy seems "better" than normal model output. Conclusions and interpretation of facts is an open question. But here's the thing: in order to disagree meaningfully with model output, including "deep research" output, you must be an expert in the material. I hope it's clear that you cannot become an expert in any field by relying on the output of generative models. In case it isn't though: imagine thinking you were a professional football player because you watched—not even played—some streamer playing "Madden." Your knowledge of the material is so many layers abstracted from praxis, it can barely be called knowledge at all.
Research is not a thing to be cut short. You know, if all these models and agents did was effectively scour the wasteland of modern search (a wasteland they helped to create) to provide high-quality sources for further reading, that'd be rather lovely. But that isn't what they do. They provide answers (not facts or truth, as I've written previously) quickly. Yes, they cite their sources (sometimes, sometimes erroneously), but the shape of the thing matters here. If you do my history essay for me and also provide the sources cited, am I going to go read the sources?
We now have evidence to suggest that the very use of generative models impairs our critical thinking instincts. So it isn't just that the research process is short-circuited; the new exchange takes even more from the user than time and effort—it robs them of ability.
Sorry to say, but humans are rather terrible at seeing the value in process when the product is readily available. But the process matters deeply. That's something folks like Mark Cuban, who claim generative models can replace educators, can never understand.
Courtney Milan puts it beautifully.
The answer was not the point. The answer was never the point. The process of searching is the process of learning.
— Courtney Milan (@courtneymilan.com) February 17, 2025 at 5:48 PM
Process is key. Some processes are banal, like driving a car to get from A to B. Some are beneficial, like taking the time to learn a subject deeply. And some are deleterious, like succumbing to addiction.
In exchanging the hard work of human knowledge construction for the quick answers of generative models, what process do we enter, knowingly or otherwise?