SoCRATES：迈向跨领域与社会认知差异的主动式大语言模型调解的可靠自动化评估

摘要

评估大语言模型调解员仍然具有挑战性，因为调解是一个实时演进的轨迹，受到争议方情绪、意图和情境变化的塑造。现有测试平台依赖于少数专家撰写的领域，主要变化在于策略姿态，并对每一轮对话针对每个话题进行评分，从而引入与主题无关的噪声。我们提出SoCRATES，一个用于在真实、多领域测试平台中评估主动型大语言模型调解员的基准测试。它通过一个跨八个领域的智能体流水线，从真实冲突中构建场景，探测五个社会认知适应维度（策略姿态、各方构成、历史长度、情绪反应性和文化身份），并通过一个主题局部评估器，仅对推进某一话题的轮次进行评分。该评估器与人类专家的一致性达到0.82，是每轮基线的两倍以上。在对八个前沿大语言模型的基准测试中，我们发现，即使在最强大的调解员手中，在多样且逼真的测试平台下，也只能弥合约三分之一的未调解共识差距，且表现因社会认知维度而异，这表明进展的关键在于针对不同条件的社会适应能力。

English

Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.