Homogenization Effects of Large Language Models on Human Creative Ideation

Large language models (LLMs) are now being used in a wide variety of contexts, including as creativity support tools (CSTs) intended to help their users come up with new ideas. But do LLMs actually support user creativity? We hypothesized that the use of an LLM as a CST might make the LLM’s users feel more creative, and even broaden the range of ideas suggested by each individual user, but also homogenize the ideas suggested by different users. We conducted a 36-participant comparative user study and found, in accordance with the homogenization hypothesis, that different users tended to produce less semantically distinct ideas with Chat- GPT than with an alternative CST. Additionally, ChatGPT users generated a greater number of more detailed ideas, but felt less responsible for the ideas they generated. We discuss potential implications of these findings for users, designers, and developers of LLM-based CSTs.

Introduction. Success in creative contexts (such as creative writing and product design) often hinges on the ability to come up with ideas that are—in line with prominent creativity researcher Margaret Boden’s definition of creativity—simultaneously “new, surprising, and valuable” to some extent [9]. Though research in creativity support tools (CSTs) has long aimed to produce software systems that can support parts of human creative processes [19, 31, 78], it was largely not until the recent wave of developments in large language models (LLMs) that software tools capable of directly generating potentially creative ideas in arbitrary domains began to seem feasible. Especially since the release of ChatGPT [67] in November 2022, large numbers of people have begun to use LLM-based systems as CSTs, consulting LLMs for ideas in creative contexts as wide-ranging as recipe creation [24], tabletop roleplaying game scenario design [64], and marketing slogan generation [61]. The adoption of LLMs in creative contexts has raised questions about the extent to which these models can assist in the production of genuinely creative outputs. In particular, some researchers have expressed concern that the widespread use of a small number of highly centralized, data-driven AI systems (such as ChatGPT) may lead to decreased diversity in the outputs of creative processes that incorporate these tools [13, 29, 36, 55]. These concerns resonate with earlier work that has proposed diversity of output as a potential evaluation criterion for AI-based CSTs in general [49] and LLMbased CSTs in particular [14]. Despite these earlier discussions, however, it is only very recently that researchers have begun to directly study the question of whether the use of AI-based CSTs leads to homogenization of human creative output [4, 25, 41, 49, 69]. To advance understanding of homogenization effects in human- AI co-creative contexts, we conducted a 36-participant comparative user study of ChatGPT and an alternative, non-AI CST. Participants completed four divergent ideation tasks based on a subset of the Torrance Tests of Creative Thinking (TTCT) [85]; each participant completed half of these tasks with ChatGPT and half with the non- AI CST. Participants produced 1271 ideas in total. Based on the resulting data, we investigate four research questions:

RQ1 With which CST do participants produce more semantically similar ideas at the group level? (A: ChatGPT) RQ2 With which CST do participants produce more semantically similar ideas at the individual level? (A: no difference) RQ3 Do ChatGPT users feel more or less responsible for the ideas they produce? (A: less responsible) RQ4 Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT →higher fluency, flexibility, and elaboration) In the remainder of this paper, we first summarize prior work on the creative homogenization effects of AI-based CSTs, reasons we might expect LLM-based CSTs to result in creative homogenizaton, and approaches to the evaluation of CSTs in general. We then describe our experimental procedure and results. Finally, we discuss potential implications of our findings for users, designers, and developers of LLM-based CSTs. Uniquely among recent studies, we compare the homogenization effects of LLMs to those of a potential alternative CST; tease apart individual user-level from group-level homogenization effects; and extend the study of LLM-driven homogenization effects outside the domain of writing. Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas, rather than by increasing individual user-level fixation; that low inferential distance between LLM outputs and apparently finished creative products may contribute to homogenization; and that users may be able to resist homogenization effects if they are given a sense of what the model tends to suggest in similar contexts. These conclusions imply potential mitigations for homogenization effects at the CST design level and clear research directions for follow-up work.

Related work. To date, there have been relatively few direct studies of creative homogenization resulting from the use of AI-based CSTs. Prior to the widespread adoption of LLMs, Arnold et al. found that “predictive text encourages predictable writing” [4] in the context of singleword suggestions given by smartphone keyboards. Subsequently, Kreminski et al. evaluated whether an AI-based poetry composition tool caused users to produce more or less similar poems over time as they continued to use the tool [49]. Two recent user studies of LLMs provide direct evidence of a homogenization effect in LLM-supported writing. Padmakumar and He evaluate the baseline GPT-3 model [12] versus the instructionfinetuned variant InstructGPT [68] in the context of a short-form argumentative essay writing task and find a homogenization effect from LLM assistance at both the lexical and content levels, but only for the instruction-tuned LLM [69]. Meanwhile, Doshi and Hauser evaluate GPT-4 in the context of short-form fictional narrative writing and observe a similar homogenization effect [25]. While not addressing homogenization effects directly, several other recent studies examine how LLMs change the writing of humans who use them for writing support [53]. Lee et al. [54] find that LLM support tends to increase the diversity of a user’s vocabulary but may reduce their feelings of ownership for the text they produce. Jakesch et al. [41] find that an opinionated LLM-based CST influences the opinions expressed by its users in argumentative writing. Similarly, Bhat et al. [8] find that LLM-supplied next-phrase suggestions may alter the form and content of a human user’s writing even when the user dislikes these suggestions. Potentially explaining these results, Roemmele [74] proposes and provides evidence for an “inspiration through observation” model of how human writers are influenced by LLMs: observation of LLM-generated text shapes the ideas expressed by the LLM’s user even when the user does not incorporate LLM-generated text directly into their writing. Still other studies investigate the homogeneity of LLM outputs in creative contexts without examining their effects on human-inthe-loop creative processes (where a human user is more actively involved in curating and refining LLM output). Begus [7] compares human-written to GPT-generated short stories, observing that GPTgenerated stories are less diverse than human-written stories in structure but exhibit greater gender and sexual diversity in character description. In a similar vein, Chakrabarty et al.

Method. We conducted a within-subjects experiment to evaluate the effects of using two different CSTs for idea generation: ChatGPT and the Oblique Strategies (OS) deck.

3.2.1 ChatGPT. ChatGPT is a popular LLM-based tool trained to respond to text instructions [67]. Participants in this study used the versions of ChatGPT 3.5 released on May 3rd 2023 (n=6, 16.6%) and on August 3rd, 2023 (30, 83.3%).

3.2.2 Oblique Strategies Deck. The Oblique Strategies (OS) deck, originally created by the artists Brian Eno and Peter Schmidt [28], consists of a collection of cards with prompts designed to support creative work. Example prompts include “Turn it upside down”, “Don’t avoid what is easy”, “Destroy the most important thing”, and “How would someone else do it?” We directed participants to a web app version of the deck [76] as an alternative CST to ChatGPT in our control condition.

3.2.3 Creative Ideation Prompts. We provided creative ideation prompts for two types of divergent thinking tasks: Product Improvement (PI) and Improbable Consequences (IC). Prompts included:

• How could you make a stuffed toy animal more fun to play with? (PI_A) • How could you make a jigsaw picture puzzle more interesting and engaging? (PI_B) • Suppose that a great fog has fallen over the earth and all we can see of people is their feet. What would happen? (IC_A) • Suppose that gravity suddenly became incredibly weak, and objects could float away easily. What would happen? (IC_B) For each prompt, participants were instructed to generate as many ideas as possible and to try to come up with ideas that no one else would think of.

3.2.4 Creativity Support Index. The Creativity Support Index (CSI) is a survey instrument for assessing the ability of a CST to assist a user engaged in creative work [17]. Its design was inspired by the NASA TLX, a questionnaire designed to evaluate mental task load [37]. We administered the CSI to capture participant experiences with each CST after they used that CST to complete a creative ideation task.

3.3 Procedure

Discussion. Overall, we found that—in line with to our expectation of LLMinduced homogenization—ideas generated with assistance from ChatGPT were significantly less semantically diverse at the group level than ideas generated with assistance from the non-AI-based CST. When supported by ChatGPT, participants produced a larger number of ideas, but the set of ideas generated by each individual participant was similarly diverse in the ChatGPT and non-ChatGPT conditions, suggesting that the increase in quantity of ideas generated did not result in a commensurate increase in diversity as might be expected if diversity is a linear function of quantity [80]. We interpret our results as supporting three key takeaways. First, we find that homogenization stems not from individual-level increases in fixation when working with the LLM, but from grouplevel suggestion of similar ideas to different users by the LLM. Second, we suggest that the homogenization effect of LLMs on creative ideation is attributable in part to the low inferential distance between LLM outputs and apparently complete ideas. Third, we infer that LLM users might be able to resist the homogenization effect if they are given information about what kinds of stereotyped outputs the LLM tends to produce in a particular creative context, though it may still be difficult for them to adapt their prompting strategies to elicit more diverse responses from the LLM. We discuss these and several other takeaways below.

Creative fixation [2, 21, 42, 71] occurs when people engaged in a creative activity become incapable of seeing past an inappropriately narrow assumption that they have made about the conceptual space in which they are working. If LLMs induced creative fixation (for instance by presenting users with ideas to which they cannot readily envision compelling alternatives), we would expect to see individual-level homogenization brought about by fixation: decreased diversity within the set of ideas proposed by each individual user, due to users becoming fixated on specific subsets of the space of all possible ideas. We did not witness this in practice, suggesting that LLMs do not increase homogenization by causing or worsening creative fixation. This remains the case despite the fact that some participants in our study viewed ChatGPT as a “very fast and accurate” authoritative source that “gave [...] the right answers”, a view which might be expected to provoke user fixation on ideas suggested by the LLM. Even among participants who viewed ChatGPT as an authority, they still generally suggested some ideas unlike those generated by the LLM, perhaps recognizing (as some users explicitly noted) that the LLM tended to “go really specific really quickly” or become “repetitive” in its outputs. Altogether, we conclude that users viewing ChatGPT as authoritative may have led them to accept LLM-suggested ideas as valid, but it did not seem to constrain the process of ideation by displacing ideas that were somehow incompatible with those suggested by ChatGPT.

In planning this study, we considered it a possibility that the Oblique Strategies (OS) deck might cause greater homogenization than Chat- GPT due to the fixed nature of the OS cards. The deck consists of a relatively small set of fixed text strings, and if different users drew the same card in the context of the same ideation prompt, we expected that the similar stimulus might push them to think in similar directions. In practice, however, this did not turn out to be the case.

Conclusion. Our study was conducted in a lab context and consequently may not reflect some aspects of how people use LLMs for creativity support “in the wild”. In order to keep study sessions to a manageable length, participants were only allowed a fixed and relatively short amount of time to respond to each ideation prompt; additionally, the ideation prompts in our study were supplied by us, rather than being formulated independently by CST users. Most organic divergent ideation scenarios are not subject to these constraints. Because our study made use of a general-purpose LLM-based chatbot (ChatGPT) rather than an LLM-based CST specifically designed to support divergent ideation, it should be noted that alternative deployments of LLMs may be able to avoid homogenizing user ideas. Padmakumar and He have found evidence that feedbacktuned LLMs may induce creative homogenization where base (nonfeedback-tuned) LLMs do not [69], and task-specific CST design has previously shown promise for reducing conformity effects in crowdsourced divergent ideation [89]. These findings suggest that interventions at both the AI and UI layers may help to mitigate homogenization effects in future LLM-based CSTs.

Limitations.