Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data

Reinforcement Learning from Human Feedback (RLHF) has proven effective in aligning large language models with human intentions, yet it often relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and present challenges in sample efficiency and stability. In this paper, we introduce Inverse-Q, an innovative framework that transcends traditional RL methods by optimizing token-level reinforcement learning without the need for additional reward or value models. Inverse- Q leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model’s responses, facilitating more granular and flexible policy shaping. Our approach reduces reliance on human annotation and external supervision, making it especially suitable for low-resource settings. We present extensive experimental results demonstrating that Inverse-Q* not only matches but potentially exceeds the effectiveness of PPO in terms of convergence speed and the alignment of model responses with human preferences. Our findings suggest that Inverse-Q* offers a practical and robust alternative to conventional RLHF approaches, paving the way for more efficient and adaptable model training approaches.

Introduction. Reinforcement Learning from Human Feedback (RLHF, Christiano et al., 2017) is a mainstream approach for aligning large models to human intentions, demonstrated in applications such as Chat- GPT (Ouyang et al., 2022) and Llama3 (AI, 2024). The RLHF framework involves modeling a reward function from preference data and learning an optimal policy through PPO (Schulman et al., 2017), which also estimates expected returns, translating language modeling into an MDP problem. This method provides nuanced supervision over training samples, proving effective in tasks like instruction following and safety (Ramamurthy et al., 2022; Ouyang et al., 2022; Glaese et al., 2022). Nonetheless, PPO’s high performance depends on complex code optimization and hyper-parameter tuning, with ongoing concerns about its sample efficiency and stability. As an efficient alternative to PPO, Direct Preference Optimization (DPO, Rafailov et al., 2024b) aligns large models from the perspective of contextual bandits, not token-level decisions (Yue et al., 2012, Dudík et al., 2015).

Discussion / Conclusion. In this article, we propose the Inverse-Q* algorithm, which has demonstrated comparable sample utilization efficiency and supervision granularity to PPO, achieving token-level reinforcement learning across all sampling outcomes without the need for additional reward or value models. This efficiency significantly eases the demands on labeling and computational resources. Extensive experiments validate the effectiveness of the Inverse-Q* framework in low-resource RLHF training, showing its potential to match or even surpass the performance of PPO training. Our method has proven to significantly enhance the alignment of large language model responses with human preferences, achieving faster convergence compared to traditional RLHF methods such as PPO and DPO.