PokemonChat：审计 ChatGPT 对于 Pokémon 宇宙知识

摘要

最近发布的ChatGPT模型展示了在零样本问答方面前所未有的能力。在这项工作中，我们探究了ChatGPT的对话理解能力，并引入了一个可在未来研究中采用的对话框架（协议）。由于其封闭世界假设，Pokémon宇宙是审核ChatGPT推理能力的理想测试场所。在揭示ChatGPT对Pokémon宇宙的背景知识后，我们测试了其在战斗场景中使用这些概念的推理过程。然后，我们评估其获取新知识并将其纳入推理过程的能力。我们的最终目标是评估ChatGPT的泛化能力、特征组合能力，以及从人类反馈中获取和推理新引入知识的能力。我们发现，ChatGPT具有Pokémon宇宙的先验知识，在战斗场景中能够在很大程度上进行推理，即使引入新信息。该模型在有协作反馈和初始信息检索阶段时表现更好，但有时会产生幻觉，并容易受到对抗性攻击的影响。

English

The recently released ChatGPT model demonstrates unprecedented capabilities in zero-shot question-answering. In this work, we probe ChatGPT for its conversational understanding and introduce a conversational framework (protocol) that can be adopted in future studies. The Pok\'emon universe serves as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to its closed world assumption. After bringing ChatGPT's background knowledge (on the Pok\'emon universe) to light, we test its reasoning process when using these concepts in battle scenarios. We then evaluate its ability to acquire new knowledge and include it in its reasoning process. Our ultimate goal is to assess ChatGPT's ability to generalize, combine features, and to acquire and reason over newly introduced knowledge from human feedback. We find that ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in battle scenarios to a great extent, even when new information is introduced. The model performs better with collaborative feedback and if there is an initial phase of information retrieval, but also hallucinates occasionally and is susceptible to adversarial attacks.

PokemonChat：审计 ChatGPT 对于 Pokémon 宇宙知识

PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge

摘要

Support