理解代理式AI系統中的認知誘發風險

摘要

由大型語言模型（LLMs）驅動的前沿代理系統展現出類似人類的認知模式。隨著這些系統在不同領域中深度融合，其認知參與對人類社會引發了尚未被充分研究的重大關切。為填補此一空白，我們依照一個由認知範圍界定的三層級框架，系統性分析認知能力擴展所引發的風險，範疇從物理認知、社會認知，乃至於最終的自我指涉認知。我們針對每個認知層級，研究其對人類能動性、自主性與控制能力的潛在風險。最後，我們提出策略以減輕這些風險，並增強代理式AI系統的可控性，確保其長期安全發展。

English

Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.