通过学习有机交互来改进开放式语言模型

摘要

我们介绍了BlenderBot 3x，这是对会话模型BlenderBot 3的更新，现在使用了参与系统的用户的有机对话和反馈数据进行训练，以提高其技能和安全性。我们公开发布了参与者去标识化互动数据，供研究社区使用，以推动进一步的进展。使用有机数据训练模型具有挑战性，因为与人们在现实场景中的互动既包括高质量的对话和反馈，也包括对抗性和有毒行为。我们研究了一些技术，使模型能够从有益的教师那里学习，同时避免从试图欺骗模型产生无益或有毒回应的人那里学习。BlenderBot 3x在对话中更受青睐，同时在挑战性情境中显示出更安全的回应，相较于BlenderBot 3。虽然我们目前的模型仍然远非完美，但我们相信通过继续使用本研究中探讨的技术，可以进一步改进。

English

We present BlenderBot 3x, an update on the conversational model BlenderBot 3, which is now trained using organic conversation and feedback data from participating users of the system in order to improve both its skills and safety. We are publicly releasing the participating de-identified interaction data for use by the research community, in order to spur further progress. Training models with organic data is challenging because interactions with people "in the wild" include both high quality conversations and feedback, as well as adversarial and toxic behavior. We study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is both preferred in conversation to BlenderBot 3, and is shown to produce safer responses in challenging situations. While our current models are still far from perfect, we believe further improvement can be achieved by continued use of the techniques explored in this work.

通过学习有机交互来改进开放式语言模型

Improving Open Language Models by Learning from Organic Interactions

摘要

Support