理解智能体AI系统中的认知诱发风险

摘要

由大语言模型（LLM）驱动的前沿智能体系统展现出类人的认知模式。随着这些系统在不同领域的深度融合，其认知参与给人类社会带来了尚未得到充分研究的重大关切。为弥补这一空白，我们遵循一个以认知范围界定的三层框架，从物理认知到社会认知，再到自我指涉认知，系统分析了认知能力扩展所引发的风险。我们针对每个认知层级，研究其对人类能动性、自主性和控制能力的潜在风险。最后，我们提出相应策略，以缓解这些风险并增强智能体AI系统的可控性，确保其长期安全发展。

English

Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.