FVAttn：用于视频生成的自适应稀疏注意力与运行时负载均衡

摘要

视频扩散Transformer需要处理长时空序列，使得自注意力成为高分辨率视频生成的主要瓶颈。免训练稀疏注意力虽能降低计算成本，但自适应Top-p路由在多GPU序列并行场景下会导致各注意力头负载不均。这种负载异构性使稀疏注意力演变为层级"掉队者"问题。本文提出免训练稀疏注意力系统，通过多GPU序列并行提升自适应稀疏注意力的分布式执行效率。该系统采用Top-p路由、Top-k安全下限及视频感知块组织作为稀疏路由前端，并在运行时修复生成的注意力掩码。运行时负载均衡通过点对点通信迁移少量重负载头，缩短当前关键路径；感知松弛稀疏增强用额外高价值块填充非关键层级的剩余松弛空间；计算重叠机制将调度与迁移开销隐藏于现有计算中。在分步精炼的Wan2.2 I2V模型上，该系统将平均负载不均衡度从1.34降至1.08，相较于FlashAttention实现4.41倍注意力加速，同时获得2.02–2.11倍DiT推理加速，并保持具有竞争力的视频质量。

English

Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present , a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41times attention speedup over FlashAttention, while achieving a 2.02--2.11times DiT inference speedup with competitive video quality.