A Primer on the Inner Workings of Transformer-based Language Models

ABSTRACT The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive overview of the known internal mechanisms implemented by these models, uncovering connections across popular approaches and active research directions in this area.

Introduction. The development of powerful Transformers-based language models (LMs; Radford et al., 2019; Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2023) and their widespread utilization underscores the significance of research devoted to understanding their inner mechanisms. Gaining a deeper understanding of these mechanisms in highly capable AI systems holds important implications in ensuring the safety and fairness of such systems, mitigating their biases and errors in critical settings, and ultimately driving model improvements (Wei et al., 2022; Costa-jussà et al., 2023). As a result, the natural language processing (NLP) community has witnessed a notable increase in research focused on interpretability in language models, leading to new insights into their internal functioning. Existing surveys present a wide variety of techniques adopted by Explainable AI analyses (Räuker et al., 2023) and their applications in NLP (Madsen et al., 2022; Lyu et al., 2024).

Discussion / Conclusion. In this paper, we have offered an overview of the existing interpretability methods useful for understanding Transformer-based language models, and have presented the insights they have led to. Although the focus of this work is on practical methods and findings, we acknowledge theoretical studies related to the interpretability of Transformers, such as investigations explaining in-context learning (Akyürek et al., 2023; Von Oswald et al., 2023; Xie et al., 2022), explorations of Transformers through the lens of data compression and representation learning (Yu et al., 2023b; Voita et al., 2019a), the study of Transformers’ learning dynamics (Tian et al., 2024; 2023; Tarzanagh et al., 2024), or the analyses on their generalization properties on algorithmic tasks (Nogueira et al., 2021; Anil et al., 2022; Zhou et al., 2024). Looking forward, we believe that the ultimate test for insights collected in years of interpretability work remains their applicability in debugging and improving the safety and reliability of future models, providing developers and users with better tools to interact with them and understand the factors influencing their predictions (Longo et al., 2024).