面向长上下文大语言模型的自引导测试时训练

摘要

长上下文处理对于大语言模型（LLMs）正变得日益重要，但简单地扩展上下文窗口并不能保证有效利用长输入。随着输入长度增长，准确率往往会下降，这表明模型在识别和使用与问题最相关的证据方面仍存在困难。提升长上下文利用率的一种有前景的方法是测试时训练（TTT），该方法将测试上下文视为训练样本，进行实例特定的参数自适应。然而，将TTT应用于整个长上下文的计算成本过高，而在随机采样段落上进行自适应则会产生严重的噪声。由于长上下文中的大多数段落与特定问题无关，基于这些段落的训练甚至可能降低基础模型的性能。我们的前期研究表明，TTT对训练段落的质量高度敏感：在LongBench-v2上，对随机采样段落进行TTT会损害性能，而对理想段落进行TTT则能显著提升性能。受此启发，我们提出一种简单方法——自引导TTT（S-TTT）：在自适应之前，模型先识别出应学习的证据段落，并仅对这些选定段落应用标准语言建模训练目标。在两个具有挑战性的长上下文推理基准（LongBench-v2和LongBench-Pro）上，S-TTT同时提升了Qwen3-4B-Thinking-2507和Llama-3.1-8B-Instruct的准确率，最高实现了15%的相对提升。

English

Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.