Exploring the Role of Prior Beliefs for Argument Persuasion

Public debate forums provide a common platform for exchanging opinions on a topic of interest. While recent studies in natural language processing (NLP) have provided empirical evidence that the language of the debaters and their patterns of interaction play a key role in changing the mind of a reader, research in psychology has shown that prior beliefs can affect our interpretation of an argument and could therefore constitute a competing alternative explanation for resistance to changing one's stance. To study the actual effect of language use vs. prior beliefs on persuasion, we provide a new dataset and propose a controlled setting that takes into consideration two reader-level factors: political and religious ideology. We find that prior beliefs affected by these reader-level factors play a more important role than language use effects and argue that it is important to account for them in NLP studies of persuasion.

We hypothesize that studying the actual effect of language on persuasion will require a more controlled experimental setting — one that takes into account any potentially confounding user-level (i.e., reader-level) factors that could cause a person to change, or keep a person from changing, his opinion. In this paper we study one such type of factor: the prior beliefs of the reader as impacted by their political or religious ideology. We adopt this focus since it has been shown that ideologies play an important role for an individual when they form beliefs about controversial topics, and potentially affect how open the individual is to being persuaded.

Our main finding is that prior beliefs associated with the selected user-level factors play a larger role than linguistic features when predicting the successful debater in a debate. In addition, the effect of these factors varies according to the topic of the debate topic. The best performance, however, is achieved when we rely on features extracted from user-level factors in conjunction with linguistic features derived from the debate text. Finally, we find that the set of linguistic features that emerges as the most predictive changes when we control for user-level factors (political and religious ideology) vs. when we do not, showing the importance of accounting for these factors when studying the effect of language on persuasion.

The prior work most relevant to ours is Lukin et al. (2017), who studied the effect of an individual's personality features (open, agreeable, extrovert, neurotic, etc.) on the type of argument (factual vs. emotional) they find more persuasive. Our work differs from this work since we study debates and in our setting the voters can see the debaters' profiles as well as all the interactions between the two sides of the debate rather than only being exposed to a monologue. Finally, we look at different types of user profile information such as a user's religious and ideological beliefs and their opinions on various topics.

Moving on to comparing our results with other studies on LLM persuasiveness, previous research has shown that rational, alternative explanations, or counterevidence is more effective for persuasion than psychological approaches like cognitive effort or moral-emotional language (Costello et al., 2024). However, there is a difference between the experimental setting of Costello et al. (2024) and the dataset we have analyzed (Durmus et al., 2024). While Costello et al. (2024) disclosed to the participants when they were interacting with a LLM, this was not the case for Durmus et al. (2024), which may explain the diverging results that we obtained in this paper. Since previous research has shown that people attribute more impartiality to AI compared to humans (Claudy et al., 2022; Logg et al., 2019), future research should investigate if knowingly interacting with a LLM increases the effect of rational arguments, while the opposite is true when the participants do not know who they are interacting with.