PokemonChat：審計 ChatGPT 對於寶可夢宇宙知識

摘要

最近釋出的ChatGPT模型展示了在零樣本問答方面前所未有的能力。在這項研究中，我們探究了ChatGPT的對話理解能力，並引入了一個未來研究中可採用的對話框架（協議）。Pokémon宇宙作為ChatGPT推理能力的審計理想測試場所，因為其具有封閉世界假設。在揭示ChatGPT對Pokémon宇宙的背景知識後，我們測試了其在戰鬥情境中應用這些概念的推理過程。然後評估其獲取新知識並將其納入推理過程的能力。我們的最終目標是評估ChatGPT的泛化能力，結合特徵，並從人類反饋中獲取和推理新引入的知識。我們發現ChatGPT對Pokemon宇宙有先驗知識，在戰鬥情境中可以相當程度地進行推理，即使引入新信息。該模型在合作反饋下表現更好，如果有信息檢索的初始階段，但有時也會出現幻覺，並容易受到對抗性攻擊的影響。

English

The recently released ChatGPT model demonstrates unprecedented capabilities in zero-shot question-answering. In this work, we probe ChatGPT for its conversational understanding and introduce a conversational framework (protocol) that can be adopted in future studies. The Pok\'emon universe serves as an ideal testing ground for auditing ChatGPT's reasoning capabilities due to its closed world assumption. After bringing ChatGPT's background knowledge (on the Pok\'emon universe) to light, we test its reasoning process when using these concepts in battle scenarios. We then evaluate its ability to acquire new knowledge and include it in its reasoning process. Our ultimate goal is to assess ChatGPT's ability to generalize, combine features, and to acquire and reason over newly introduced knowledge from human feedback. We find that ChatGPT has prior knowledge of the Pokemon universe, which can reason upon in battle scenarios to a great extent, even when new information is introduced. The model performs better with collaborative feedback and if there is an initial phase of information retrieval, but also hallucinates occasionally and is susceptible to adversarial attacks.

PokemonChat：審計 ChatGPT 對於寶可夢宇宙知識

PokemonChat: Auditing ChatGPT for Pokémon Universe Knowledge

摘要

Support