"It makes you think": Provocations Help Restore Critical Thinking to AI-Assisted Knowledge Work

Recent research suggests that the use of Generative AI tools may result in diminished critical thinking during knowledge work. We study the effect on knowledge work of provocations: brief textual prompts that offer critiques for and propose alternatives to AI suggestions. We conduct a between-subjects study (n= 24) in which participants completed AI-assisted shortlisting tasks with and without provocations. We find that provocations can induce critical and metacognitive thinking. We derive five dimensions that impact the user experience of provocations: task urgency, task importance, user expertise, provocation actionability, and user responsibility.

We connect our findings to related work on design frictions, microboundaries, and distributed cognition. We draw design implications for critical thinking interventions in AI-assisted knowledge work.

Introduction. Generative AI (GenAI), defined as any “end-user tool [...] whose technical implementation includes a generative model based on deep learning”1, is achieving wide penetration in knowledge workflows. As a result, knowledge work is shifting “from material production to critical integration” [100]. This means that the physical activities of producing text, images, video, audio, etc. such as typing and drawing are being increasingly delegated by users to GenAI tools, and the user focus shifts to deciding when, where and how to apply GenAI to their work, and evaluating and editing its output to integrate it back into the ecosystem of their knowledge artefacts [13, 113, 117].

But this shift comes at a cost to human critical thinking (defined in Section 2.1). One obvious risk is overreliance, defined as “as users accepting incorrect AI recommendations – i.e., making errors of commission.” [88]. But beyond errors and hallucinations, there is a subtler and more pervasive tendency that affects knowledge work even when the output is “correct” in some objective sense. Recent research has identified a tendency toward “mechanised convergence” [100, 104], that the introduction of GenAI to a knowledge workflow increases the homogeneity of work produced.

There is substantial evidence across multiple studies and knowledge workflows for the phenomenon of mechanised convergence. Predictive text encourages predictable writing [5]. Programmers using GitHub Copilot produce fewer unique identifier names [70]. Exposure to GenAI “ideas” leads to less diverse content produced by short story writers [33]. Strategy consultants using ChatGPT retain a high proportion of its response, and produce less conceptually varied ideas than those without access [31]. Participants produce less distinct ideas in a creative ideation task when using ChatGPT than without it [3]. Visual artists using GenAI experience a decline in average visual and content novelty [136]. Conversational search can produce a “generative echo chamber” [111]. Co-writing with “opinionated” language models shifts writers’ opinions [53].

People tend to offload effortful cognitive reasoning tasks to technology [9]. Deliberate thinking requires mental resources that people are unwilling to devote [128], and media with fewer opportunities for interaction contribute less to the development of critical thinking [97]. Analytic methods from economics calculate that a significant proportion of jobs have significant exposure to (i.e., possibility of automation with) GenAI [37], a trend borne out in surveys of GenAI adoption [11]. The complementary risks of overreliance and mechanised convergence build toward a picture where GenAI has a negative effect on knowledge work by compromising the ability of human workers to form deeply personalised, critical, contextual, and subjective responses to AI output in the process of integrating it into their workflow.

In this paper, we evaluate the concept of provocations: short text prompts whose aim is to induce critical thinking in the user in AI-assisted tasks. Provocations highlight risks, biases, limitations, and alternatives of GenAI suggestions.

Provocations draw on two lineages of research: explorations of integrating critical thinking aids in education (detailed in Section 2.1), and design research for promoting reflective thinking (Section 2.2). While much research has been conducted in improving critical thinking in education and certain professional disciplines, interventions for critical thinking in contemporary, AI-assisted knowledge workflows have not been studied. Similarly, design research in reflective thinking focuses on applications such as promoting mental health and wellbeing, misinformation detection, and media consumption habits, but critical thinking interventions for common knowledge workflows remain under-explored.

2025-01-30 01:02. Page 2 of 1–38.

Unpublished working draft.

“It makes you think”: Provocations Help Restore Critical Thinking to AI-Assisted Knowledge Work 3 We focus on one such workflow: shortlisting (detailed in Section 3.1). For example, shortlisting candidates to interview from a list of job applicants, choosing a set of houses to view from a list of search results, or shortlisting papers to read. Shortlisting is a common yet important and diverse type of knowledge work where GenAI can be applied to suggest factors for shortlisting and automatically evaluate candidate items according to these factors. Yet shortlisting encompasses many detailed, conscious, subjective and critical judgements, which makes it important to explore how AI can be used to support the task without diminishing – and preferably, enhancing – the user’s ability to think critically about their problem.

Related work. 2.1 Critical Thinking Frameworks. “Critical thinking” is a contested term. As a starting point, we adopt the influential framework developed by Bloom et al. [12, 52], a hierarchical taxonomy that characterises student learning objectives into six types: knowledge (recall of ideas), comprehension (demonstrating understanding of ideas), application (putting ideas into practice), analysis (contrasting and relating ideas), synthesis (combining ideas), and evaluation (judging ideas through criteria).

Subsequent scholarship has developed similar frameworks and also identified psychological dispositions (not unlike personality traits) such as inquisitiveness, that mediate critical thinking [40–42, 89]. Though not all scholars use the terms interchangeably, critical thinking is often also referred to as reflective thinking [32]. Research in reflective thinking often conceptualises it as a skill that is developed in sequential stages [64, 85, 89]. There have been multiple proposals to integrate these disparate frameworks for critical thinking [36, 68, 83]. However, in our research we adopt the Bloom et al. framework for its relative simplicity, as well as its strong support in the research literature and wide operationalisation in educational systems.

Despite concerns about whether critical thinking can be taught [129], research in education has developed a number 2.2 Designing for Critical Thinking and Reflection 2.2.1 Design Dimensions. One design dimension for critical or reflective thinking interventions is whether the system should be proactive, i.e., introduce critical thinking prompts without an explicit user request. There is mixed evidence that suggests the appropriate solution is heavily context dependent. For example, in a stock investment workflow, the system might only intervene when the user is about to make an important decision [92]. On the other hand, in a language micro-learning workflow, the system might interrupt the user at random occasions [63].

Research has demonstrated the importance of user participation in critical thinking interventions. For instance, Danry et al. [29] find that AI explanations presented as questions, rather than statements, improves logical discernment.

Surfacing contextual questions and encouraging the self-generation of questions improves critical engagement with reading [93, 134]. Attention checks during surveys promote systematic thinking on tricky tasks [48]. Engagement with artistic artefacts and conflict-filled peer-discussion fosters critical thinking about AI in youth [71]. Increased participation and engagement in the reasoning process, including with other human discussants, is more likely to result in attitudinal and behavioural changes [74, 80].

Method. 3 System Design 3.1 Shortlisting While our work is motivated by the tendency towards overreliance and mechanised convergence in knowledge work broadly, to focus our investigation it is necessary to narrow our scope to specific tasks. We chose shortlisting – selecting a small set of items from a larger set – as the user task on which to focus our investigation. Shortlisting tasks are a ubiquitous type of knowledge work: examples include shortlisting interview candidates from a list of job applicants, shortlisting papers to accept for a conference from a list of submissions, shortlisting software features to implement from a list of user requests, etc.

As a type of sensemaking through data analysis [91], shortlisting is relatively small in scope: it involves choosing relevant factors and their weights for evaluating candidates, then applying those factors to individual candidates, sorting and ranking them. While small in scope, shortlisting activities are nonetheless diverse and ecologically valid [107].

They present a suitable test bed for evaluating provocations because they require users to enact a range of critical thinking behaviours: choosing qualitative and quantitative factors, applying them (often subjectively) to candidates, and iterative decision-making and reflection as users refine their factors and shortlist selection over the course of the task.

Shortlisting is not the only such task; several types of data-driven sensemaking tasks are diverse yet well-constrained, such as personal budgeting and financial planning, online shopping and comparison, etc. [34]. Suitable tasks are also available in other domains, such as design or writing. As a research team, our collective prior experience in qualitative analysis has been focused on studies of end-users performing data analysis tasks in spreadsheets, including shortlisting, and our relative strength in qualitative interpretation of participant experiences in such tasks informed our choice.

3.2 Design Decisions and Research Goals Our system takes two roles during shortlisting. The first is the conventional role of AI as assistant. The system assists by suggesting potential factors and weights for those factors, that the user may wish to use to evaluate the candidate items in their dataset. Moreover, the system evaluates each item according to the factors selected by the user, sorts the list of items, and creates a shortlist of the highest ranking items. This process is illustrated in Figure 1.

The second role is that of AI as provocateur [103] or critical thinking tool [132] – presenting provocations that aim to induce critical thinking about the use of AI suggestions in the user’s workflow. This is the role that we are interested in evaluating.

The design of provocations in our system can be seen in Figure 2, and their role within the overall shortlisting workflow in Figure 1. Our design is informed by the following principles from prior literature:

• From Salomon [99] we apply the key technique of presenting short texts to prompt user reflection. We limit provocations to two sentences to avoid information overload, according to guidelines and best practices for human-AI interfaces [2].

• Due to the complexity and range of phenomena associated with agentised or anthropomimetic presentation (e.g., [86, 118, 133]) we present provocations in the language of plain factual warnings, to avoid any inadvertent confounding effects of agentic presentation.

• Previous research on the timing and quantity of interventions has shown that the solution depends on the context [63, 92]; we attach a single provocation directly to each AI-generated factor suggestion, presented at the same time and location as the suggested factor. This allows provocations to apply to individual suggestions and thus be more specific, while only slightly increasing the cognitive effort of consuming the information related to each 2025-01-30 01:02. Page 6 of 1–38.

Unpublished working draft.

Discussion. Regarding RQ1: Do provocations affect users’ critical thinking during shortlisting tasks?, our qualitative analysis finds impacts on metacognition and critical thinking at multiple levels of Bloom’s taxonomy (Section 5.6).

2025-01-30 01:02. Page 12 of 1–38.

Unpublished working draft.

“It makes you think”: Provocations Help Restore Critical Thinking to AI-Assisted Knowledge Work 13 0.5 0.0 0.5 1.0 1.5 Similarity Rating (RBO) 0.0 0.5 1.0 1.5 2.0 Density Charity Dataset 0.5 0.0 0.5 1.0 1.5 Similarity Rating (RBO) Movies Dataset Median for Provocations Median for No Provocations Fig. 3. Distribution of final shortlist similarity for the four groups of conditions and tasks within the study (Charity task on the left, Movies task on the right). This kernel density estimate (KDE) plot [126] visualises the Rank-Biased Overlap (RBO) similarity metric, where a higher RBO means more similar lists. Median RBO values for each dataset in Condition A (Provocations) (in orange) are lower than the respective values for Condition B (No Provocations) (in blue).

We did not find statistically significant effects of provocations on shortlist diversity (Section 5.1), or the rate of thinkaloud reflections and user interface actions (Section 5.2). Though not statistically significant, Condition A (Provocations) had a higher median shortlist diversity and higher rate of think-aloud reflections.

We did not find significant effects of provocations on aggregate measures of self-reflection (Section 5.3) or trust (Section 5.4).

Regarding RQ2: What important dimensions of the system design, task, or user interact with the effectiveness of provocations?, we identify task urgency, task importance, user expertise, provocation actionability, and user responsibility as key elements contributing to critical thinking and the effectiveness of provocations (Section 5.7).

5.6 Qualitative Effects of Provocations on Critical Thinking With some exceptions, our quantitative comparisons are not statistically significant. This is likely due to a combination of the inadequacy of our quantitative measures, insufficient statistical power, and the relative infrequency of conscious critical thinking “events” (discussed in Section 6). However, this obscures the qualitative effects on critical thinking observable in the think aloud data.

In this section, we describe how provocations resulted in consciously articulated critical thinking events. We coded participant utterances according to the “level” of critical thinking they corresponded to in Bloom’s taxonomy [12, 52]: analysis, synthesis, evaluation, comprehension, and application. Additionally, we observed a number of utterances corresponding to increased metacognition. Research has uncovered a range of metacognitive demands associated with GenAI [117]. Metacognition and critical thinking are related; some critical thinking processes are primarily metacognitive in nature (e.g., reflecting on the limits of one’s own knowledge) and we find it more useful to characterise them as such.

We note that in participant utterances, there is not always a neat separation between the levels of Bloom’s taxonomy, or between Bloom’s taxonomy and metacognition. Many participant utterances reflected multiple levels simultaneously, or reflected the fluid and instantaneous transition from one type of thinking to another. For simplicity, we focus on one level at a time, but recognise that this entails some degree of artificial separation.