長上下文大型語言模型的自引導測試時訓練

摘要

長上下文處理對大型語言模型（LLMs）日益重要，但單純擴展上下文窗口並不能保證有效利用長輸入。隨著輸入長度增長，準確率往往下降，顯示模型仍難以辨識並運用與問題最相關的證據。提升長上下文利用率的一個有前景的方法是測試時訓練（TTT），將測試上下文視為訓練樣本進行實例特定的參數調整。然而，對整個長上下文進行TTT在計算上過於昂貴，而在隨機取樣的片段上進行適應則會引入嚴重噪音。由於長上下文中大部分片段與具體問題無關，對這些片段進行訓練甚至可能降低基礎模型的性能。我們的初步研究顯示，TTT對訓練片段的品質極為敏感：在LongBench-v2中，對隨機取樣片段使用TTT會損害性能，而對最佳證據片段（oracle spans）使用TTT則能顯著提升性能。受此啟發，我們提出一個簡單方法：自引導測試時訓練（S-TTT）。在適應前，模型自行識別應學習的證據片段，並僅對這些選定的片段應用標準語言模型訓練目標。在兩個具挑戰性的長上下文推理基準——LongBench-v2與LongBench-Pro上，S-TTT分別提升了Qwen3-4B-Thinking-2507與Llama-3.1-8B-Instruct的準確率，最高達到15%的相對提升。

English

Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.