TESSERA v2：扩展逐像素地球基础模型

摘要

逐像素地球观测（EO）基础模型如今通过生成的空间嵌入实现了最先进的性能。然而，这些模型如何扩展以及如何最优地分配预训练预算仍未被充分理解。我们展示了迄今为止最大规模的地球观测受控扩展研究：在固定的逐像素Barlow Twins族内，于1024个GH200超级芯片上进行了395次训练，每次训练均在15个下游任务上进行了评估。我们发现预训练损失几乎无法预测下游性能（|皮尔逊相关系数| < 0.2），因此根据损失选择模型会浪费大量计算资源。我们还发现，随着训练预算增加，编码器和数据应同步增长，而投影器保持不变，这提供了一个简单的计算分配规则。利用这一规则，我们训练了一系列逐像素模型（0.5B和1B，另有2B模型正在训练中），并将其蒸馏为紧凑的学生模型，用于嵌入即数据部署。2100万参数的蒸馏版TESSERA v2-1B-M在整体性能上超越了所有经测试的开源和专有模型，其中部分模型的规模高出数个数量级。这些学生模型生成的服务成本低廉的Matryoshka表示：一个16维前缀在仅使用1/8存储的情况下保留了完整128维性能的92%。训练完成后，我们计划发布覆盖2017-2025年的v2全球嵌入。综合来看，这些结果为扩展逐像素地球观测基础模型提供了基于经验的具体方案：训练大型编码器，根据下游性能进行选择，并蒸馏为灵活的学生模型。所有代码将在 https://github.com/ucam-eo/tessera 发布。

English

Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs on 1,024 GH200 superchips within a fixed pixel-wise Barlow Twins family, each evaluated on 15 downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise models (0.5B and 1B, with a 2B model in training) and distill them into compact students for embeddings-as-data deployment. The 21-million-parameter distilled TESSERA v2-1B-M in aggregate outperforms all open and proprietary models tested, some of which are orders of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Upon completion of training we plan to release v2 global embeddings covering 2017-2025. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. All code will be released at https://github.com/ucam-eo/tessera.