Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration

Large Language Model (LLM)-powered coding agents are increasingly used in professional settings to write software more rapidly, via human-artificial intelligence (AI) collaboration. In this new era of work, it is important to understand what kinds of human behavior traits contribute to task success. Moreover, we need to uncover key skills required for modern professionals, which can inform educators on how to foster the development of these skills among students. Existing guidelines for human-AI collaboration are built from either top-down theory or empirical, context-specific observations of human-AI interactions. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, while empirical guidelines may become obsolete very quickly. In this work, we explore an automated, data-driven approach to uncover patterns, which we term traits, of effective human-AI interaction that are aligned with collaborative coding task outcomes. We propose Principal Trait Analysis (PTA), a Principal Component Analysisinspired algorithm for deriving common traits from patterns in LLM conversations.

Introduction. LLM coding assistants are increasingly used in software development and integrated into computer science education, yet it is still unclear what distinguishes effective from ineffective use in either setting. The same model, given to two software developers, can produce vastly different code depending on how each developer interacts with the model, and the productivity gains from coding assistants vary substantially across developers [15]. Likewise, two students can have vastly different usage patterns which result in different learning experiences and subsequent outcomes [16, 17]. Uncovering these usage patterns and relating them to outcomes would give i) developers guidelines on how to collaborate with coding agents and ii) instructors guidelines on how to teach “AI use skills” to computer science students. However, uncovering these patterns and establishing universal guidelines is difficult, since the space of relevant behavioral traits is large and rapidly changing as LLM capabilities improve.

Discussion / Conclusion. In this paper, we introduced Principal Trait Analysis, a data-driven, principal component analysis-inspired approach that derives interpretable, traits of LLM usage directly from human-AI interaction session trace datasets. We evaluated our approach on human-AI collborative conversations in a software development setting, demonstrating our approach in both a learning and professional context. Our approach recovers traits that explain outcomes beyond prior achievement heuristics and seem readily interpretable. However, our analysis reveals that the traits are inconsistent with the qualities required to interpret these traits as human-AI collaboration skills. Our work outlines avenues for future research into the derived traits and overall methodology. A possible next step for future work would be to test this methodology across a larger educational dataset, beyond classroom-scale, and not isolated to computer science.