MindZero：在零註釋下學習線上心智推理

摘要

有效的現實世界輔助需要具備穩健心智理論（ToM）的人工智慧代理，能從人類行為推斷其心理狀態。儘管近年有所進展，仍有若干關鍵挑戰尚待解決，包括：（1）需對多重假設進行穩健不確定性更新之線上推論；（2）適合即時輔助的高效推理；以及（3）真實世界領域缺乏真實心智狀態標註。我們提出 MindZero 以應對這些挑戰，該框架為一種自監督強化學習方法，訓練多模態大型語言模型（MLLM）以實現高效且穩健的線上心智推理。在訓練過程中，模型因生成心智狀態假設而獲得獎勵，這些假設能最大化由規劃器估計的觀察行為之似然，類似於基於模型的 ToM 推理。此方法因而無需明確的心智狀態標註。訓練完成後，MindZero 將基於模型的推理內化為快速的單次推論。我們在網格世界與家居領域中，針對具挑戰性的心智推理及 AI 輔助任務，將 MindZero 與基準方法進行比較。結果發現，僅靠大型語言模型（LLM）有所不足；基於模型的方法雖能提升準確度，但速度慢、成本高，且受骨幹 MLLM 的能力所限。相比之下，MindZero 增強了 MLLM 的內在 ToM 能力，且在準確度與效率上均顯著優於基於模型的方法，顯示心智推理可有效學習為一種自監督技能。

English

Effective real-world assistance requires AI agents with robust Theory of Mind (ToM): inferring human mental states from their behavior. Despite recent advances, several key challenges remain, including (1) online inference with robust uncertainty updates over multiple hypotheses; (2) efficient reasoning suitable for real-time assistance; and (3) the lack of ground-truth mental state annotations in real-world domains. We address these challenges by introducing MindZero, a self-supervised reinforcement learning framework that trains multimodal large language models (MLLMs) for efficient and robust online mental reasoning. During training, the model is rewarded for generating mental state hypotheses that maximize the likelihood of observed actions estimated by a planner, similar to model-based ToM reasoning. This method thus eliminates the need for explicit mental state annotations. After training, MindZero internalizes model-based reasoning into fast single-pass inference. We evaluate MindZero against baselines across challenging mental reasoning and AI assistance tasks in gridworld and household domains. We found that LLMs alone are insufficient; model-based methods improve accuracy but are slow, costly, and limited by backbone MLLM capacity. In contrast, MindZero enhances MLLMs' intrinsic ToM ability and significantly outperforms model-based methods in both accuracy and efficiency, showing that mental reasoning can be effectively learned as a self-supervised skill.