Orion-MSP：面向表格上下文學習的多尺度稀疏注意力機制

Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning

November 4, 2025

作者: Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu

cs.AI

摘要

表格資料仍是現實應用的主流格式。然而由於異質特徵類型和多尺度複雜交互作用的存在，開發有效的表格資料神經模型仍具挑戰。近期表格上下文學習技術（如TabPFN和TabICL）的進展，已能在無需任務特定微調的情況下達到與梯度提升樹相當的頂尖性能。但現有架構存在三大侷限：(1) 單尺度特徵處理忽視層級依賴關係；(2) 稠密注意力機制存在表格寬度的二次方複雜度；(3) 嚴格順序的組件處理阻礙迭代表徵優化與跨組件通信。為解決這些難題，我們提出Orion-MSP表格上下文學習架構，具備三大創新：(1) 多尺度處理機制捕捉層級特徵交互；(2) 融合窗口化、全局與隨機模式的塊稀疏注意力，實現可擴展效率與長程關聯；(3) 感知器風格記憶模組確保組件間安全的雙向信息流。在多樣化基準測試中，Orion-MSP在有效擴展至高維表格的同時，達到或超越現有頂尖性能，為高效表格上下文學習樹立新標杆。模型已開源於 https://github.com/Lexsi-Labs/Orion-MSP 。

English

Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL, have achieved state-of-the-art performance comparable to gradient-boosted trees (GBTs) without task-specific fine-tuning. However, current architectures exhibit key limitations: (1) single-scale feature processing that overlooks hierarchical dependencies, (2) dense attention with quadratic scaling in table width, and (3) strictly sequential component processing that prevents iterative representation refinement and cross-component communication. To address these challenges, we introduce Orion-MSP, a tabular ICL architecture featuring three key innovations: (1) multi-scale processing to capture hierarchical feature interactions; (2) block-sparse attention combining windowed, global, and random patterns for scalable efficiency and long-range connectivity; and (3) a Perceiver-style memory enabling safe bidirectional information flow across components. Across diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables, establishing a new standard for efficient tabular in-context learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-MSP .