Language Models are Pragmatic Speakers

How do language models “think”? This paper formulates a probabilistic cognitive model called bounded pragmatic speaker, which can characterize the operation of different variants of language models. In particular, we show that large language models fine-tuned with reinforcement learning from human feedback (Ouyang et al., 2022) implements a model of thought that conceptually resembles a fast-and-slow model (Kahneman, 2011). We discuss the limitations of reinforcement learning from human feedback as a fast-and-slow model of thought and propose directions for extending this framework. Overall, our work demonstrates that viewing language models through the lens of cognitive probabilistic modeling can offer valuable insights for understanding, evaluating, and developing them.

Introduction. Large language models (Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; Zhang et al., 2022a; Scao et al., 2022; Touvron et al., 2023) have emerged as a new artificial intelligence powerhouse. These models exhibit many traits of human and superhuman intelligence: they can hold natural conversations with humans (OpenAI, 2022), learn from few examples (Dong et al., 2022), solve complex reasoning problems (Wei et al., 2022b), generate programs (Chen et al., 2021), and pass exams written for human professionals (OpenAI, 2023). While the capabilities of large language models have been well-documented, little is known about the underlying cognitive mechanisms that power these capabilities. By consuming a ginormous amount of records of human behavior and knowledge, have these models learned to think and reason like humans? Or are they merely copycats? If neither, what exactly is their “model of thought”? Giving a scientific answer to this question is important to dispel ungrounded speculations about large language models and direct their future developments.

Discussion / Conclusion. In this work, we show that Bayesian models of human cognition can be used to effectively explain the operation of large language models. Our proposed framework represents only a simple version of the models computational cognitive scientists have developed. More advanced proposals like hierarchical Bayesian models (Tenenbaum et al., 2011) can potentially accommodate more complex reasoning and offer better explainability. It has been challenging to scale up these models to realworld problems because of their expensive inference cost. However, as we have shown, large language models and its learning techniques like RLHF can offer themselves as useful tools for developing more scalable Bayesian probabilistic models. To develop more effective inference algorithms, we suggest taking inspiration from human pragmatic communication. Current learning paradigms like imitation and reinforcement learning emulate very primitive forms of communication that are far inferior to human communication. New paradigms like in-context learning (Wei et al., 2022a) allow for learning from rich language instructions, but the pragmatic elements of human communication are still missing (Fried et al., 2022).