选择、压缩、再投入：长视频多模态大语言模型中视觉Token分配的受控研究

摘要

长视频语言模型无法观看每一帧：以每秒一帧采样一小时视频将产生3,600张图像，而系统仅从该图像池中保留一小块固定切片。哪些帧能留存于该切片通常被视为预处理细节；我们检验这一做法是否应当如此。已发表的帧选择器使比较变得困难，因为它们同时改变了帧评分器、提示边界、分辨率策略和回答模型。我们将每一项固定不变，每次仅改变一个决策：选择、空间压缩以及节约资源的再投入，涵盖六种无需训练的选择规则、三个长视频基准测试和两个回答模型。选择是最大的单一杠杆：在LongVideoBench的一小时区间上，八个查询选中的帧比十六个均匀间隔的帧高出6.9分；而正交匹配追踪——一种未经修改的、已有数十年历史的稀疏逼近算法——在所有三个基准测试中，与我们所对比的每个专用选择器表现相当或仅差一分以内。压缩几乎免费：在固定时间戳下将每帧的空间预算减半，最多仅损失0.44分。再投入才是将预算转化为准确率的关键环节：将释放出的词元用于两倍数量的压缩帧，在实测代价不高于原始八帧的情况下，可额外提升两到三分；压缩只有在以这种方式花费其节约的资源时才有回报。在此过程中，我们自身AKS基线中的一个实现错误，以及两个测试框架在相同预算下运行相同已发表规则时出现的0.07至3.74分的差距，说明了为何这些比较必须在单一受控框架内进行，而非跨论文比对。

English

Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.