Orion-MSP：面向表格上下文学习的多尺度稀疏注意力方法

Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning

November 4, 2025

作者: Mohamed Bouadi, Pratinav Seth, Aditya Tanna, Vinay Kumar Sankarapu

cs.AI

摘要

表格数据仍是现实应用中最主要的数据形式。然而，由于特征类型异构且存在多尺度复杂交互，开发适用于表格数据的有效神经网络模型仍具挑战。近年来表格上下文学习（ICL）技术取得突破，如TabPFN和TabICL无需任务特定微调即可达到与梯度提升树（GBT）相媲美的顶尖性能。但现有架构存在明显局限：（1）单尺度特征处理忽视层次化依赖关系；（2）稠密注意力机制随表格宽度呈二次方复杂度增长；（3）严格顺序的组件处理阻碍迭代表示优化与跨组件通信。为解决这些问题，我们提出Orion-MSP表格ICL架构，其三大创新包括：（1）多尺度处理机制捕捉层次化特征交互；（2）融合窗口化、全局化与随机模式的块稀疏注意力，实现可扩展效率与长程关联；（3）感知器风格记忆模块确保组件间安全的双向信息流。在多样化基准测试中，Orion-MSP在有效扩展至高维表格的同时达到或超越现有顶尖性能，为高效表格上下文学习树立了新标准。模型已开源：https://github.com/Lexsi-Labs/Orion-MSP。

English

Tabular data remain the predominant format for real-world applications. Yet, developing effective neural models for tabular data remains challenging due to heterogeneous feature types and complex interactions occurring at multiple scales. Recent advances in tabular in-context learning (ICL), such as TabPFN and TabICL, have achieved state-of-the-art performance comparable to gradient-boosted trees (GBTs) without task-specific fine-tuning. However, current architectures exhibit key limitations: (1) single-scale feature processing that overlooks hierarchical dependencies, (2) dense attention with quadratic scaling in table width, and (3) strictly sequential component processing that prevents iterative representation refinement and cross-component communication. To address these challenges, we introduce Orion-MSP, a tabular ICL architecture featuring three key innovations: (1) multi-scale processing to capture hierarchical feature interactions; (2) block-sparse attention combining windowed, global, and random patterns for scalable efficiency and long-range connectivity; and (3) a Perceiver-style memory enabling safe bidirectional information flow across components. Across diverse benchmarks, Orion-MSP matches or surpasses state-of-the-art performance while scaling effectively to high-dimensional tables, establishing a new standard for efficient tabular in-context learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-MSP .