The Debate Over Understanding in AI's Large Language Models

We survey a current, heated debate in the AI research community on whether large pre-trained language models can be said to understand language—and the physical and social situations language encodes—in any humanlike sense. We describe arguments that have been made for and against such understanding, and key questions for the broader sciences of intelligence that have arisen in light of these arguments. We contend that an extended science of intelligence can be developed that will provide insight into distinct modes of understanding, their strengths and limitations, and the challenge of integrating diverse forms of cognition.

Introduction. What does it mean to understand something? This question has long engaged philosophers, cognitive scientists, and educators, nearly always with reference to humans and other animals. However, with the recent rise of large-scale AI systems—especially so-called large language models—a heated debate has arisen in the AI community on whether machines can now be said to understand natural language, and thus understand the physical and social situations that language can describe. This debate is not just academic; the extent and manner in which machines understand our world has real stakes for how much we can trust them to drive cars, diagnose diseases, care for the elderly, educate children, and more generally act robustly and transparently in tasks that impact humans.

Moreover, the current debate suggests a fascinating divergence in how to think about understanding in intelligent systems, in particular the contrast between mental models that rely on statistical correlations and those that rely on causal mechanisms.

Until quite recently there was general agreement in the AI research community about machine understanding: while AI systems exhibit seemingly intelligent behavior in many specific tasks, they do not understand the data they process in the way humans do. Facial recognition software does not understand that faces are parts of bodies, or the role of facial expressions in social interactions, or what it means to “face” an unpleasant situation, or any of the other uncountable ways in which humans conceptualize faces. Similarly, speech-to-text and machine translation programs do not understand the language they process, and autonomous driving systems do not understand the meaning of the subtle eye contact or body language drivers and pedestrians use to avoid accidents. Indeed, the oft-noted brittleness of these AI systems—their unpredictable errors and lack of robust generalization abilities—are key indicators of their lack of understanding [59]. However, over the last several years, a new kind of AI system has soared in popularity and influence in the research community, one that has changed the views of some people about the prospects of machines that understand language. Variously called Large Language Models (LLMs), Large Pre- Trained Models, or Foundation Models [11], these systems are deep neural networks with billions to trillions of parameters (weights) that are “pre-trained” on enormous natural-language corpora, including large swathes of the Web, online book collections, and other collections amounting to terabytes of data. The task of these networks during training is to predict a hidden part of an input sentence—a method called “self-supervised learning.” The resulting network is a complex statistical model of how the words and phrases in its training data correlate. Such models can be used to generate natural language, be fine-tuned for specific language tasks [58], or be further trained to better match “user intent” [65]. LLMs such as OpenAI’s well-known GPT-3 [12] (and more recent ChatGPT [69]) and Google’s PaLM [16] can produce astonishingly humanlike text, conversation, and, in some cases, what seems like human reasoning abilities [83], even though the models were not explicitly trained to reason. How LLMs perform these feats remains mysterious for lay people and scientists alike. The inner workings of these networks are largely opaque; even the researchers building them have limited intuitions about systems of such scale. The neuroscientist Terrence Sejnowski described the emergence of LLMs this way: “A threshold was reached, as if a space alien suddenly appeared that could communicate with us in an eerily human way. Only one thing is clear—LLMs are not human...Some aspects of their behavior appear to be intelligent, but if not human intelligence, what is the nature of their intelligence?” [70].

Related work. One faction argues that these networks truly understand language and can perform reasoning in a general way (though “not yet” at the level of humans). For example, Google’s LaMDA system, which was pre-trained on text and then fine-tuned on dialogue [77], is sufficiently convincing as a conversationalist that it convinced one AI researcher that such systems “in a very real sense understand a wide range of concepts” [1] and are even “making strides towards consciousness” [3].

Another machine language expert sees LLMs as a canary in the coal mine of general human-level AI: “There is a sense of optimism that we are starting to see the emergence of knowledge-imbued systems that have a degree of general intelligence” [54]. Another group argues that LLMs “likely capture important aspects of meaning, and moreover work in a way that approximates a compelling account of human cognition in which meaning arises from conceptual role” [67]. Those who reject such claims are criticized for promoting “AI denialism” [2].

Those on the other side of this debate argue that large pretrained models such as GPT-3 or LaMDA—however fluent their linguistic output—cannot possess understanding because they have no experience or mental models of the world; their training in predicting words in vast collections of text has taught them the form of language but not the meaning [8, 9, 55]. A recent opinion piece put it this way: “A system trained on language alone will never approximate human intelligence, even if trained from now until the heat death of the universe” and “it is clear that these systems are doomed to a shallow understanding that will never approximate the full-bodied thinking we see in humans” [13]. Another scholar argued that intelligence, agency, and by extension, understanding “are the wrong categories” for talking about these systems; instead LLMs are compressed repositories of human knowledge more akin to libraries or encyclopedias than to intelligent agents [33]. For example, humans know what is meant by a “tickle” making us laugh, because we have bodies. An LLM could use the word ”tickle”, but it has obviously never had the sensation. Understanding a tickle is to map a word to a sensation, not to another word.

Those on the “LLMs do not understand” side of the debate argue that while the fluency of large language models is surprising, our surprise reflects our lack of intuition of what statistical correlations can produce at the scales of these models. Anyone who attributes understanding or consciousness to LLMs is a victim of the Eliza effect [37]—named after the 1960s chatbot created by Joseph Weizenbaum that, simple as it was, still fooled people into believing it understood them [84].

Method. While “humanlike understanding” does not have a rigorous definition, it does not seem to be based on the kind of massive statistical models that today’s LLM’s learn; instead it is based on concepts—internal mental models of external categories, situations, and events, and of ones own internal state and “self”. In humans, understanding language (as well as nonlinguistic information) requires having the concepts that language (or other information) describes, beyond the statistical properties of linguistic symbols. Indeed, much of the long history of research in cognitive science has been a quest to understand the nature of concepts, and how understanding arises from coherent, hierarchical sets of relations among concepts that include underlying causal knowledge [6, 43]. These models enable people to abstract their knowledge and experiences in order to make robust predictions, generalizations, and analogies, to reason compositionally and counterfactually, to actively intervene on the world in order to test hypotheses, and to explain one’s understanding to others [29, 32, 38, 41, 45, 73, 74]. Indeed, these are precisely the abilities lacking in current AI systems, including state-of-the-art LLMs, though ever-larger LLMs have exhibited limited sparks of these general abilities. It has been argued that understanding of this kind may enable abilities not possible for purely statistical models [19, 27, 44, 66, 76]. While LLMs exhibit extraordinary formal linguistic competence—the ability to generate grammatically fluent, humanlike language—they still lack the conceptual understanding needed for humanlike functional language abilities—the ability to robustly understand and use language in the real world [52]. An interesting parallel can be made

Discussion. For example, a long-standing criticism of quantum mechanics is that it provides an effective means of calculation without providing conceptual understanding.

The detailed nature of human concepts has been the subject of active debate for many years.

Researchers disagree on the extent to which concepts are domain-specific and innate versus more general-purpose and learned [14, 30, 31, 53, 75, 85], the degree to which concepts are grounded via embodied metaphors [28, 46, 61], are represented in the brain via dynamic, situation-based simulations [5], and the conditions under which concepts are underpinned by language [20, 23, 51], by social learning [4, 81, 26] and by culture [7, 60, 63]. In spite of these ongoing debates, concepts, in the form of causal mental models as described above, have long been considered to be the units of understanding in human cognition. Indeed, the trajectory of human understanding—both individual and collective—is the development of highly compressed, causally based models of the world, analogous to the progression from Ptolemy’s epicycles to Kepler’s elliptical orbits, and to Newton’s concise and causal account of planetary motion in terms of gravity. Humans, unlike machines, seem to have a strong innate drive for this form of understanding, both in science and in everyday life [34]. We might characterize this form of understanding as requiring little data, minimal or parsimonious models, clear causal dependencies, and strong mechanistic intuition.

The key questions of the debate about understanding in LLMs are the following: (1) Is talking of understanding in such systems simply a category error, mistaking associations between language tokens for associations between tokens and physical, social, or mental experience? In short, is it the case that these models are not, and will never be, the kind of things that can understand? Or conversely, (2) do these systems (or will their near-term successors) actually, even in the absence of physical experience, create something like the rich concept-based mental models that are central to human understanding, and, if so, does scaling these models create ever better concepts? Or (3) If these systems do not create such concepts, can their unimaginably large systems of statistical correlations produce abilities that are functionally equivalent to human understanding? Or, indeed, that enable new forms of higher-order logic that humans are incapable of accessing? And at this point will it still make sense to call such correlations “spurious” or the resulting solutions

Conclusion. The debate over understanding in LLMs, as ever larger and seemingly more capable systems are developed, underscores the need for a extending our sciences of intelligence in order to make sense of broader conceptions of understanding, for both humans and machines. As neuroscientist Terrence Sejnowski points out, “The diverging opinions of experts on the intelligence of LLMs suggests that our old ideas based on natural intelligence are inadequate” [70]. If LLMs and related models succeed by exploiting statistical correlations at a heretofore unthinkable scale, perhaps this could be considered a novel form of “understanding”, one that enables extraordinary, superhuman predictive ability, such as in the case of the AlphaZero and AlphaFold systems from DeepMind [40, 72], which respectively seem to bring an “alien” form of intuition to the domains of chess playing and proteinstructure prediction [39, 68].

It could thus be argued that in recent years the field of AI has created machines with new modes of understanding, most likely new species in a larger zoo of related concepts, that will continue to be enriched as we make progress in our pursuit of the elusive nature of intelligence. And just as different species are better adapted to different environments, our intelligent systems will be better adapted to different problems. Problems that require enormous quantities of historically encoded knowledge where performance is at a premium will continue to favor large-scale statistical models like LLMs, and those for which we have limited knowledge and strong causal mechanisms will favor human intelligence. The challenge for the future is to develop new scientific methods that can reveal the detailed mechanisms of understanding in distinct forms of intelligence, discern their strengths and limitations, and learn how to integrate such truly diverse modes of cognition.