FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

Modern Large Language Models (LLMs) are capable of following long and complex instructions that enable a diverse amount of user tasks. However, despite Information Retrieval (IR) models using LLMs as the backbone of their architectures, nearly all of them still only take queries as input, with no instructions. For the handful of recent models that do take instructions, it’s unclear how they use them. We introduce our dataset FOLLOWIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions. FOLLOWIR builds off the long history of the TREC conferences: as TREC provides human annotators with instructions (also known as narratives) to determine document relevance, so should IR models be able to understand and decide relevance based on these detailed instructions. Our evaluation benchmark starts with three deeply judged TREC collections and alters the annotator instructions, re-annotating relevant documents. Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework.

Introduction. Large language models (LLMs) have shown impressive results in understanding long complex instructions and have enabled a broad array of tasks (Brown et al., 2020; Achiam et al., 2023). The Information Retrieval (IR) community has adopted the use of LLMs as the main backbone of their neural retrieval architectures and has seen an explosion of progress in semantic search (Karpukhin et al., 2020; Khattab & Zaharia, 2020; Reimers & Gurevych, 2019). However, these existing systems rely on under-specified keywords and short sentences, despite the growing capabilities of LLMs. And even though the retrieval community is increasingly adopting larger IR models for their backbones (such as LLaMA 7B in Pradeep et al. (2023); Ma et al. (2023); Weller et al. (2023)), they are only used to estimate document relevance without any instructions. Moving past this ad-hoc search to search with instructions would enable experts to narrow in on complex information needs using only flexible natural language, adapting to new tasks on the fly.

Discussion / Conclusion. Despite the use of LLMs as the backbone of neural retrieval models, most existing IR models do not take instructions that define document relevance. Further, there is no existing resource that measures how well retrieval models can follow instructions. We build a new benchmark that explicitly measures the instruction following ability of retrieval models and find that nearly all retrieval models do not follow instructions, with the exception of very large models (3B+ parameters) or instruction-tuned LLMs that typically are not used for retrieval. However, we show that it is possible to improve their instruction following ability, and build and release a training corpus for teaching retrieval models to follow instructions. Our new model, FOLLOWIR-7B, shows improvement on both standard retrieval metrics as well as in instruction following. We hope that these resources will allow the community to develop more capable instruction-following retrieval models that can quickly adapt to a relevance definition given flexible natural language text.