How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding

The utility of AI in multi-coder qualitative coding has been widely discussed, yet little empirical evidence exists to delineate the contexts in which it performs reliably. We address this gap by quantifying the effectiveness of multi-agent LLM coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes. We developed a literature-informed baseline pipeline that enables AI agents to independently code, debate, and reconcile disagreements. Results revealed that coding accuracy depends on factors such as codebook length, qualitative data similarity, and agent disagreement. Notably, intense and unresolved debates between agents led to higher accuracy. Our analysis showed that while LLMs emulate many human discussion behaviors, they lack adaptive responsiveness to context. From these findings, we offer design recommendations for building automated coding systems. Our open-source AI discussion dataset and methodological framework lay the groundwork for advancing the design of AI-mediated automated thematic analysis.

Introduction. Deductive qualitative coding is the process of applying predefined frameworks to categorize and interpret qualitative data. It is a time-consuming and labor-intensive task even for experts, demanding meticulous examination and contextual understanding of nuanced data [77, 80]. The central role of qualitative coding in social science has led to a growing body of recent work on automating qualitative coding through artificial intelligence (AI) and natural language processing (NLP) techniques. Motivated by our own interest in qualitative research with large amounts of data, we investigate AI-based coding methods modeled on human practices, such as multi-coder comparison and consensus. Our study quantifies the effectiveness of multi-agent Large Language Model (LLM) coding across varied qualitative datasets, revealing key contextual and structural factors that mediate coding outcomes.

Discussion / Conclusion. 6.1 Summary of Key Findings and Implications Our study reveals that LLM “coders” can engage in a structured dialogue to perform qualitative coding, with performance influenced by specific dataset characteristics and discussion dynamics. First, we found that coding accuracy is shaped by dataset’s length and semantic similarity, and the gap of degree of specialization between codebook and data for coding. These findings highlighted that preprocessing codebooks and qualitative data for coding can boost AI coding performance. Second, our results show that the frequency of “Undecidable” label and conflict between AI agents can be utilized as a performance indicator. These signals assist users in calibrating their expectations of automated systems. We discovered that instances of disagreement and debate between the AI coders can be a positive signal for high accuracy of coding. Rather than viewing disagreement as a failure, our study indicates that when the two AI coders disagreed on how to code a segment, the conversation often increased interpretive depth. The AI’s convergence is a double-edged sword: on one hand, it led to consistent consensus in most cases, but on the other hand, we suspect it might sometimes mask uncertainty.