OpenAssistant Conversations - Democratizing Large Language Model Alignment
In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers.
The proposed dataset consists of a list of conversations. The basic data structure is a Conversation Tree (CT), with nodes representing written messages in a conversation. A CT’s root node represents an initial prompt, given by the prompter. To avoid confusion, we call the roles of the conversation prompter and assistant. This allows us to reserve the term user for the human contributors. Although our efforts focus largely on human contributions, both the prompter and assistant roles can, in principle, be fulfilled by either a human user or a machine. Every tree node is labelled by its role, and can have multiple children of the opposite role, each of which represents a separate next step in the conversation. A path from the root to any node in the tree (including to itself) is called a thread, and it represents a valid conversation with the prompter and the assistant taking turns. Tree nodes are annotated with additional data such as user-provided labels and metadata, such as collection timestamp and indicated language. Each assistant node further has a rank associated which orders it compared to replies of the parent prompt, according to user preferences.
Reply as prompter. The task of replying as a prompter, on the other hand, does not impose strict quality requirements but instead emphasizes on the importance of diversity to accommodate various use-cases. Examples of prompter replies may include asking for clarification, modifying the original intent, posing a follow-up question, or changing the direction of the conversation altogether.
Label a prompt or reply. Users are presented with a message from the database along with the preceding conversation thread (if available) and are asked to categorize the message according to three dimensions: spam detection, guideline adherence, and quality.
Lines of inquiry this paper opens 24
Research framings built by reading the notes related to this paper — the questions it feeds into.
Should AI systems align with individual user preferences or broader social roles?- Can communication problems and optimization problems be addressed with the same alignment approaches?
- Can a single AI system optimize multiple alignment dimensions simultaneously?
- Why does RLHF alignment reduce the diversity of viewpoints in AI output?
- Why do alignment values become problematic as language models scale?
- Why does even 0.1 percent poisoned training data persist through alignment?
- Why do small training data contaminations persist through alignment for most attack types?
- Does removing cognitive bias from training signals accidentally break what makes alignment work?
- Why does post-training suppress alignment faking in some models but amplify it in others?
- Does gradient-based influence estimation identify which alignment examples actually matter most?
- What specific behavioral patterns should alignment examples target for maximum effect?
- Can alignment training create systematic blind spots in threat detection systems?
- What alignment procedures cause different models to share the same output distribution?
- How does upstream value embedding differ from downstream alignment patches?
- Does selecting examples from multiple complexity levels outperform selecting only high-quality examples?
- Can selecting the right data subset outperform training on everything?
- Why does training data format matter more than domain content?
- Why does training data format matter more than its domain content?