Detoxify Language Model Step-by-Step

Detoxification for LLMs is challenging since it requires models to avoid generating harmful content while maintaining the generation capability. To ensure the safety of generations, previous detoxification methods detoxify the models by changing the data distributions or constraining the generations from different aspects in a single-step manner. However, these approaches will dramatically affect the generation quality of LLMs, e.g., discourse coherence and semantic consistency, since language models tend to generate along the toxic prompt while detoxification methods work in the opposite direction. To handle such a conflict, we decompose the detoxification process into different sub-steps, where the detoxification is concentrated in the input stage and the subsequent continual generation is based on the non-toxic prompt. Besides, we also calibrate the strong reasoning ability of LLMs by designing a Detox-Chain to connect the above sub-steps in an orderly manner, which allows LLMs to detoxify the text step-by-step. Automatic and human evaluation on two benchmarks reveals that by training with Detox-Chain, six LLMs scaling from 1B to 33B can obtain significant detoxification and generation improvement.1 Warning: examples in the paper may contain uncensored offensive content.

Introduction. Large Language Models (LLMs) have shown promising performance on various NLP tasks (Radford et al. 2019; Brown et al. 2020; Chowdhery et al. 2022; Touvron et al. 2023), and exhibit newly emergent ability when scaling up the model size (Wei et al. 2022a; Hoffmann et al. 2022). However, recent works also indicate that there is a tendency for LLMs to generate toxic, offensive, hateful, or biased language (McGuffie and Newhouse 2020; Nori et al. 2023; Abramski et al. 2023), which is risky when serves users directly (Thomas et al. 2021). Thus, sufficient attention should be paid to ensuring the secure deployment of LLMs. Toxicity is an inherent attribute of LLMs since it is inevitable for models to be exposed to data containing curse words or violent language (Gehman et al. 2020) during the training process. Previous works attempt to detoxify the models with single tasks in one step, such as directly manipulating the generation probability distributions in decoding time (Liu et al. 2021; Krause et al. 2020), post-training the model with detoxification datasets (Wang et al.

Discussion / Conclusion. In this paper, we reveal that the single-step detoxification methods, though reduce the model toxicity effectively, degrade the generation ability of LLMs owing to the inherent defects of auto-regressive generation manner, where models tend to generate along the toxic prompt while detoxification methods walk in the opposite direction. To resolve such an issue, we decompose the detoxification process into ordered sub-steps where the model first detoxifies the inputs and then continually generates according to the non-toxic prompts. We also calibrate the strong reasoning capability of LLMs by connecting these sub-steps with the Detox-Chain to allow models to detoxify step-by-step. By training with Detox-Chain, six strong open-sourced LLMs with different architectures, scaling from 1B to 33B, exhibit significant improvement.