The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review

Source: Smart Learning Environments (Zhai, Wibowo, Li) · 2024-06-18

The growing integration of artificial intelligence (AI) dialogue systems within educational and research settings highlights the importance of learning aids. Despite examination of the ethical concerns associated with these technologies, there is a noticeable gap in investigations on how these ethical issues of AI contribute to students’ over-reliance on AI dialogue systems, and how such over-reliance affects students’ cognitive abilities. Overreliance on AI occurs when users accept AI-generated recommendations without question, leading to errors in task performance in the context of decision-making. This typically arises when individuals struggle to assess the reliability of AI or how much trust to place in its suggestions. This systematic review investigates how students’ over-reliance on AI dialogue systems, particularly those embedded with generative models for academic research and learning, affects their critical cognitive capabilities including decision-making, critical thinking, and analytical reasoning. By using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, our systematic review evaluated a body of literature addressing the contributing factors and effects of such over-reliance within educational and research contexts. The comprehensive literature review spanned 14 articles retrieved from four distinguished databases: ProQuest, IEEE Xplore, ScienceDirect, and Web of Science. Our findings indicate that over-reliance stemming from ethical issues of AI impacts cognitive abilities, as individuals increasingly favor fast and optimal solutions over slow ones constrained by practicality. This tendency explains why users prefer efficient cognitive shortcuts, or heuristics, even amidst the ethical issues presented by AI technologies.

Artificial intelligence (AI) dialogue systems, often known as conversational agents, are complex software mechanisms that emulate human dialogue, leveraging the prowess of AI, natural language processing, and machine learning technologies (Zhai & Wibowo, 2023a). Integrating generative dialogue systems in research and education has drawn considerable interest in recent years. This is because these technologies promise to revolutionize research and education by streamlining repetitive tasks, aiding in data interpretation, and pioneering new learning and assessment methods (George & Wooden, 2023; Song & Xiong, 2021; Zhai & Wibowo, 2023). However, there are concerns about the potential negative impact of its widespread use on cognitive abilities, particularly in academic writing (Liu et al., 2023). Research and education fundamentally rely on evidence, decision-making, critical thinking, and analytical thinking are crucial for thoroughly analyzing and evaluating the quality of information found in existing literature studies (Hanim et al., 2020). The importance of cultivating a mindset of cognitive abilities cannot be overstated, especially for students who are tasked with synthesizing, evaluating, and forming arguments (Kaeppel, 2021).

A few studies have been conducted on the issues concerning over-reliance on AI dialogue systems. Gao et al. (2022) found a concerning trend where users exhibit an over-reliance on AI dialogue systems, often accepting their generated outputs, AI hallucination, without validation. This overdependence is exacerbated by cognitive biases where judgments deviate from rationality and heuristics or the use of mental shortcuts, leading to uncritical acceptance of AI-generated information. Grassini (2023) identified that algorithmic biases are frequently a result of AI systems being trained on datasets with inherent prejudices, causing users to regard these biased outputs as objective mistakenly. This misplaced trust can skew analysis and interpretation, further entrenching the issue. Xie et al. (2021) found that over-reliance on unverified AI outputs can cause misclassification and misinterpretation. The generation of such unvalidated content by AI systems poses a significant risk, potentially culminating in research misconduct, including plagiarism, fabrication, and falsification. Dempere et al. (2023) highlighted the risks associated with embedding AI dialogue systems in higher education, such as privacy violations and illegal data use. These authors caution against the normalization of intrusive data practices that might emerge from an over-reliance on AI, where the collection and analysis of student data do not fully honor privacy rights. Meanwhile, Dergaa et al. (2023) argued the importance of data transparency in AI systems within research and education.

Other studies revealed that regular utilization of dialogue systems is linked to a decline in abilities of cognitive abilities, a diminished capacity for information retention, and an increased reliance on these systems for information (Dergaa et al., 2023; Marzuki et al., 2023). This over-reliance often occurs without verifying the validity and authenticity of the provided data, especially when such information lacks proper references (Krullaars et al., 2023). Krullaars et al. (2023) argue that the over-reliance on AI dialogue systems might diminish students’ drive and commitment to learning, as they might lean too heavily on these systems for answers instead of actively participating in the learning experience. The adoption and over-reliance on AI dialogue systems have overshadowed critical ethical concerns whereby issues such as the generation of inaccurate or misleading content, algorithmic biases, plagiarism, privacy breaches, and transparency concerns have not been adequately addressed (Hua et al., 2023).

To address this research gap, we conducted a systematic literature review to investigate the contributing factors and effects of over-reliance on AI dialogue systems in research and education. The study specifically examines the contributing factors of over-reliance, such as AI hallucination, algorithmic bias, plagiarism, privacy concerns, and transparency issues, and how over-reliance impacts cognitive abilities, including decision-making, critical thinking, and analytical thinking. Additionally, this review outlines potential strategies and technological interventions to alleviate these challenges, aiming to foster responsible use of AI dialogue systems. Several research questions are formulated to address the concerns stated above:

How does over-reliance on AI dialogue systems affect critical and analytical thinking abilities in different educational subjects and levels?

What are the primary ethical concerns causing over-reliance on AI dialogue systems in research and education?

Given that these AI systems can handle vast data and yield accurate forecasts, there is a looming danger of humans becoming excessively reliant on AI when making choices. This over-reliance might stifle creativity and innovative thinking in both educators and learners, possibly degrading educational quality (Ahmad et al., 2023). Krullaars et al. (2023) posit that an over-reliance on dialogue systems hinders students from developing their critical thinking and problem-solving abilities.

As AI dialogue systems offer pre-formulated answers, this practice can curtail students’ freedom to convey their unique thoughts and viewpoints (Krullaars et al., 2023). Similarly, Ahmad et al. (2023) argue that one consequence of over-reliance on dialogue systems is a potential decline in user prowess of cognitive abilities. Furthermore, Gao et al. (2022) claim that students often overly rely on the source of information, leading to challenges in differentiating whether the content produced by an AI dialogue system was referenced to a credible source. This scenario of AI hallucination involves the AI creating plausible but untrue statements or assertions, which can mislead users and obscure the line between fact and fiction. This phenomenon raises important questions about users’ ability to critically evaluate and discern the accuracy of AI-generated content. Moreover, Hatem et al. (2023) discuss the issue of AI systems referencing non-existent sources, a form of confabulation that presents false information within a seemingly credible framework. This misrepresentation can deceive users and undermine the trustworthiness of the information provided by AI systems. Athaluri et al. (2023) argue that the issue of AI confabulations can potentially have adverse effects on decision-making and may lead to ethical and legal dilemmas. The authors found that among 178 reference sources produced by AI dialogue systems, 69 lacked a Digital Object Identifier (DOI), and 28 could not be found through a Google search, nor did they possess an existing DOI.

The utilization of AI tools to compose a paper in any academic or professional context constitutes plagiarism (Gao et al., 2022).