當記憶說謊：視覺語言模型（VLM）代理中空間記憶過時性的實證研究

摘要

記憶增強型視覺語言模型代理依賴持久性空間知識運作，然而隨著環境變化，這些知識會悄然過時。我們探討當代理必須調和確信的記憶主張與相互矛盾的觀察時會發生什麼，以及當前模型能否在衝突演變為安全相關錯誤之前加以察覺。我們利用動態 FrozenLake 測試平台，將過時檢測任務與下游導航任務配對，涵蓋三種閉源模型與三種開放權重視覺語言模型，並在文本與圖像輸入兩種條件下進行（1,800 次檢測運行，以及四個大語言模型導航器在共享 50 種隨機種子規模下的 12,000 次文本模式導航回合）。研究得出三項發現。第一，文本可解性並不意味著視覺基礎能力：能夠可靠地從文本中標記過時條目的模型，在相同的網格上視覺 F1 分數卻從 0.887 降至 0.067，最弱的模型持續做出流暢而自信但忽略圖像的決策。第二，未經審計地使用過時記憶是一種安全隱患：在我們主要的 GPT-4o 設定中，信任原始記憶的代理死亡頻率是完全没有記憶的同一個代理的兩倍以上。第三，審計有所幫助但未能彌合差距：透明的讀取時過濾器在文本模式下消除了大部分安全成本，然而即使在當前網格規模下使用神諭級過時標籤也未帶來進一步的顯著提升，而當視覺審計不可靠時，過濾並未產生一致的效益。綜合來看，這些結果將空間記憶過時界定為一種安全失效模式，並將記憶─觀察衝突下的可靠視覺基礎與動作選擇，確立為記憶增強代理的核心開放性挑戰。

English

Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an agent must reconcile a confident memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake. Using a dynamic FrozenLake testbed, we pair a staleness-detection task with a downstream navigation task across three closed-source models and three open-weight VLMs under both text and image inputs (1,800 detection runs, and 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale). Three findings emerge. First, text solvability does not imply visual grounding: models that flag stale entries reliably from text nonetheless span vision F1 from 0.887 down to 0.067 on the identical grids, and the weakest keeps making fluent, confident decisions that ignore the image. Second, consuming stale memory without an audit is a safety liability: in our primary GPT-4o setting, an agent that trusts raw memory dies more than twice as often as the same agent given no memory at all. Third, auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, yet even oracle stale labels bring no further significant gain on the current grid size, and when visual auditing is unreliable, filtering yields no consistent benefit. Together these results frame spatial-memory staleness as a safety failure mode and isolate reliable visual grounding and action selection under memory--observation conflict as the central open challenges for memory-augmented agents.