Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search

However, one of the main challenges for this beautiful vision is that the users’ queries may contain some linguistic problems (e.g., omissions and coreference) and it becomes much harder to capture their real search intent under the multi-turn conversation context (Dalton et al., 2021; Mao et al., 2022a).

To achieve conversational search, an intuitive method known as Conversational Query Rewriting (CQR) involves using a rewriting model to transform the current query into a de-contextualized form. Subsequently, any ad-hoc search models can be seamlessly applied for retrieval purposes.

In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLM as a search intent interpreter to facilitate conversational search. Specifically, we first prompt LLM to generate both short query rewrites and longer hypothetical responses in multiple perspectives and then aggregate these generated contents into an integrated representation that robustly represents the user’s real search intent. Under our framework, we propose three specific prompting methods and aggregation methods, and conduct extensive evaluations on three widely used conversational search benchmarks,

In general, our framework has two main advantages. First, by leveraging the powerful contextual understanding and generation abilities of large language models, we show that additionally generating hypothetical responses to explicitly supplement more plausible search intents underlying the short rewrite can significantly improve the search performance. Second, we show that properly aggregating multiple rewrites and hypothetical responses can effectively filter out incorrect search intents and enhance the reasonable ones, leading to better search performance and robustness.

3.2.1 Rewriting Prompt (REW) In this prompting method, we directly treat LLM as a well-trained conversational query rewriter and prompt it to generate rewrites. Only the red part of Figure 4 is enabled. Although straightforward, we show in Section 4.5 that this simple prompting method has been able to achieve quite a strong search performance compared to existing baselines.

3.2.2 Rewriting-Then-Response (RTR) Recently, a few studies (Mao et al., 2021; Gao et al., 2022; Yu et al., 2023; Mackie et al., 2023) have shown that generating hypothetical responses for search queries can often bring positive improvements in retrieval performance. Inspired by them, in addition to prompting LLM to generate rewrites, we continue to utilize the generated rewrites to further prompt LLM to generate hypothetical responses that may contain relevant information to answer the current question.