从泛化到特化：探索基础模型中的测试时训练机制

摘要

近期实证研究探讨了在测试阶段继续针对特定任务训练模型的想法，即测试时训练（TTT），并发现其能显著提升性能。然而，对于TTT为何及何时有效，目前理解尚浅。早期的解释多集中于观察到TTT在应用于分布外适应或使用特权数据时可能有所帮助。但随着基础模型规模的扩大，大多数测试数据属于分布内，这些解释受到质疑。我们提出，基础模型在全局上仍处于欠参数化状态，TTT提供了一种在泛化后实现专门化的机制，将模型能力集中于与测试任务相关的概念上。具体而言，在线性表示假设下，我们构建了一个模型，其中TTT实现的分布内测试误差远小于全局训练。通过在ImageNet上训练稀疏自编码器，我们实证验证了模型的关键假设，表明语义相关的数据点仅由少数共享概念解释。最后，我们在图像和语言任务上进行了扩展研究，证实了模型的实际意义，并识别出专门化最为有效的场景。

English

Recent empirical studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT), and have found it to yield significant performance improvements. However, there is limited understanding of why and when TTT is effective. Earlier explanations mostly focused on the observation that TTT may help when applied to out-of-distribution adaptation or used with privileged data. However, the growing scale of foundation models with most test data being in-distribution questions these explanations. We instead posit that foundation models remain globally underparameterized, with TTT providing a mechanism for specialization after generalization, focusing capacity on concepts relevant to the test task. Specifically, under the linear representation hypothesis, we propose a model in which TTT achieves a substantially smaller in-distribution test error than global training. We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet, showing that semantically related data points are explained by only a few shared concepts. Finally, we perform scaling studies across image and language tasks that confirm the practical implications of our model, identifying the regimes where specialization is most effective.

从泛化到特化：探索基础模型中的测试时训练机制

Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models

摘要

Support