TESSERA v2：擴展像素級地球基礎模型

摘要

像素級地球觀測（EO）基礎模型現可透過生成的空間嵌入達到最先進的效能。然而，此類模型的縮放行為及如何最佳分配預訓練預算，至今仍未被充分理解。我們提出了迄今規模最大的EO受控縮放研究：在固定像素級Barlow Twins架構下，於1,024個GH200超級晶片進行395次訓練，每次訓練均在15項下游任務上評估。我們發現，預訓練損失幾乎無法預測下游任務效能（皮爾森相關係數|r| < 0.2），因此根據損失選擇模型會浪費大量運算資源。此外，我們還發現，隨著訓練預算增加，編碼器與資料量應同步成長，而投影器則保持固定，這為運算資源分配提供了一個簡單規則。依據此規則，我們訓練了一系列像素級模型（0.5B與1B，另有2B模型正在訓練中），並將其蒸餾為緊湊的學生模型，以利於嵌入即資料的部署。參數量僅2,100萬的蒸餾模型TESSERA v2-1B-M，在總體表現上優於所有受測的開源與商業模型，其中部分模型的參數量高出數個數量級。這些學生模型可產生套娃式表征，且部署成本低廉：僅使用16維前綴即可保留完整128維92%的效能，而儲存量僅為1/8。訓練完成後，我們計畫釋出覆蓋2017至2025年的v2全域嵌入。綜合上述結果，我們提出了一個具體且經實證支持的縮放像素級EO基礎模型策略：訓練大型編碼器、根據下游效能進行篩選，並蒸餾為靈活的學生模型。所有程式碼將於 https://github.com/ucam-eo/tessera 公開釋出。

English

Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs on 1,024 GH200 superchips within a fixed pixel-wise Barlow Twins family, each evaluated on 15 downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise models (0.5B and 1B, with a 2B model in training) and distill them into compact students for embeddings-as-data deployment. The 21-million-parameter distilled TESSERA v2-1B-M in aggregate outperforms all open and proprietary models tested, some of which are orders of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Upon completion of training we plan to release v2 global embeddings covering 2017-2025. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. All code will be released at https://github.com/ucam-eo/tessera.