大型語言模型在用戶意圖演變中迷失

摘要

隨著大型語言模型（LLM）的能力日益增強，它們越來越常被部署為協作代理，透過反覆互動來執行使用者委派的任務。然而，真正的互動本質上是動態的：使用者鮮少在一開始就明確表達其意圖，反而會在對話過程中逐步揭露、修正與重塑目標。儘管如此，目前LLM仍大多在單輪、完全指定的情境下進行評估或訓練，這留下了一個根本問題：當使用者的意圖在對話中持續演變時，LLM能多好地追蹤並據此行動？為此，我們提出一個框架，將靜態的單輪任務轉化為動態的多輪對話——在對話過程中，使用者的意圖會逐步揭露、修正，甚至中途轉向——同時保留每項任務原有的評估協議，使現有基準可作為受控測試平台重複使用，無需新增標註。在多項任務中，我們觀察到一致的現象：在靜態設定中表現優異的模型，並未將此優勢延伸到意圖演變的情境，且各模型家族的效能均大幅下滑。我們的發現指向一個根本缺口：當今的LLM尚無法確實追蹤並依據使用者演變中的意圖採取行動，這項能力在靜態評估中隱而未見，但對於未來的協作代理卻至關重要。

English

As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents.