LaTIM：測量Mamba模型中的潛在令牌間交互作用

LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models

February 21, 2025

作者: Hugo Pitorro, Marcos Treviso

cs.AI

摘要

狀態空間模型（SSMs），如Mamba，已成為長上下文序列建模中變換器（transformers）的高效替代方案。然而，儘管其應用日益廣泛，SSMs仍缺乏對於理解和改進基於注意力架構至關重要的可解釋性工具。雖然近期的研究提供了對Mamba內部機制的洞察，但這些研究並未明確分解各個詞元的貢獻，導致在理解Mamba如何跨層選擇性處理序列方面存在空白。在本研究中，我們提出了LaTIM，一種針對Mamba-1和Mamba-2的新穎詞元級分解方法，實現了細粒度的可解釋性。我們在多樣化的任務上廣泛評估了該方法，包括機器翻譯、複製以及基於檢索的生成，證明了其在揭示Mamba詞元間交互模式方面的有效性。

English

State space models (SSMs), such as Mamba, have emerged as an efficient alternative to transformers for long-context sequence modeling. However, despite their growing adoption, SSMs lack the interpretability tools that have been crucial for understanding and improving attention-based architectures. While recent efforts provide insights into Mamba's internal mechanisms, they do not explicitly decompose token-wise contributions, leaving gaps in understanding how Mamba selectively processes sequences across layers. In this work, we introduce LaTIM, a novel token-level decomposition method for both Mamba-1 and Mamba-2 that enables fine-grained interpretability. We extensively evaluate our method across diverse tasks, including machine translation, copying, and retrieval-based generation, demonstrating its effectiveness in revealing Mamba's token-to-token interaction patterns.