Shengqiong Wu, Weicai Ye, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Shuicheng Yan, Hao Fei, Tat-Seng Chua
774
To address the bottleneck of accurate user intent interpretation within the
current video generation community, we present Any2Caption, a novel framework
for controllable video generation under any condition. The key idea is to
decouple various condition interpretation steps from the video synthesis step.
By leveraging modern multimodal large language models (MLLMs), Any2Caption
interprets diverse inputs--text, images, videos, and specialized cues such as
region, motion, and camera poses--into dense, structured captions that offer
backbone video generators with better guidance. We also introduce Any2CapIns, a
large-scale dataset with 337K instances and 407K conditions for
any-condition-to-caption instruction tuning. Comprehensive evaluations
demonstrate significant improvements of our system in controllability and video
quality across various aspects of existing video generation models. Project
Page: https://sqwu.top/Any2Cap/
Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, Bingsheng He
616
The rise of Large Language Models (LLMs) as evaluators offers a scalable
alternative to human annotation, yet existing Supervised Fine-Tuning (SFT) for
judges approaches often fall short in domains requiring complex reasoning. In
this work, we investigate whether LLM judges truly benefit from enhanced
reasoning capabilities. Through a detailed analysis of reasoning requirements
across evaluation tasks, we reveal a negative correlation between SFT
performance gains and the proportion of reasoning-demanding samples -
highlighting the limitations of SFT in such scenarios. To address this, we
introduce JudgeLRM, a family of judgment-oriented LLMs trained using
reinforcement learning (RL) with judge-wise, outcome-driven rewards. JudgeLRM
models consistently outperform both SFT-tuned and state-of-the-art reasoning
models. Notably, JudgeLRM-3B surpasses GPT-4, and JudgeLRM-7B outperforms
DeepSeek-R1 by 2.79% in F1 score, particularly excelling in judge tasks
requiring deep reasoning.
Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar
522
Soft attention is a critical mechanism powering LLMs to locate relevant parts
within a given context. However, individual attention weights are determined by
the similarity of only a single query and key token vector. This "single token
attention" bottlenecks the amount of information used in distinguishing a
relevant part from the rest of the context. To address this issue, we propose a
new attention method, Multi-Token Attention (MTA), which allows LLMs to
condition their attention weights on multiple query and key vectors
simultaneously. This is achieved by applying convolution operations over
queries, keys and heads, allowing nearby queries and keys to affect each
other's attention weights for more precise attention. As a result, our method
can locate relevant context using richer, more nuanced information that can
exceed a single vector's capacity. Through extensive evaluations, we
demonstrate that MTA achieves enhanced performance on a range of popular
benchmarks. Notably, it outperforms Transformer baseline models on standard
language modeling tasks, and on tasks that require searching for information
within long contexts, where our method's ability to leverage richer information
proves particularly beneficial.
Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, Xihui Liu
383
Recent advancements in Chain of Thought (COT) generation have significantly
improved the reasoning capabilities of Large Language Models (LLMs), with
reinforcement learning (RL) emerging as an effective post-training approach.
Multimodal Large Language Models (MLLMs) inherit this reasoning potential but
remain underexplored in tasks requiring both perception and logical reasoning.
To address this, we introduce SEED-Bench-R1, a benchmark designed to
systematically evaluate post-training methods for MLLMs in video understanding.
It includes intricate real-world videos and complex everyday planning tasks in
the format of multiple-choice questions, requiring sophisticated perception and
reasoning. SEED-Bench-R1 assesses generalization through a three-level
hierarchy: in-distribution, cross-environment, and cross-environment-task
scenarios, equipped with a large-scale training dataset with easily verifiable
ground-truth answers. Using Qwen2-VL-Instruct-7B as a base model, we compare RL
with supervised fine-tuning (SFT), demonstrating RL's data efficiency and
superior performance on both in-distribution and out-of-distribution tasks,
even outperforming SFT on general video understanding benchmarks like
LongVideoBench. Our detailed analysis reveals that RL enhances visual
perception but often produces less logically coherent reasoning chains. We
identify key limitations such as inconsistent reasoning and overlooked visual
cues, and suggest future improvements in base model reasoning, reward modeling,
and RL robustness against noisy signals.
Weizhi Wang, Yu Tian, Linjie Yang, Heng Wang, Xifeng Yan
367
The reproduction of state-of-the-art multimodal LLM pre-training faces
barriers at every stage of the pipeline, including high-quality data filtering,
multimodal data mixture strategies, sequence packing techniques, and training
frameworks. We introduce Open-Qwen2VL, a fully open-source 2B-parameter
Multimodal Large Language Model pre-trained efficiently on 29M image-text pairs
using only 442 A100-40G GPU hours. Our approach employs low-to-high dynamic
image resolution and multimodal sequence packing to significantly enhance
pre-training efficiency. The training dataset was carefully curated using both
MLLM-based filtering techniques (e.g., MLM-Filter) and conventional CLIP-based
filtering methods, substantially improving data quality and training
efficiency. The Open-Qwen2VL pre-training is conducted on academic level
8xA100-40G GPUs at UCSB on 5B packed multimodal tokens, which is 0.36\% of 1.4T
multimodal pre-training tokens of Qwen2-VL. The final instruction-tuned
Open-Qwen2VL outperforms partially-open state-of-the-art MLLM Qwen2-VL-2B on
various multimodal benchmarks of MMBench, SEEDBench, MMstar, and MathVista,
indicating the remarkable training efficiency of Open-Qwen2VL. We open-source
all aspects of our work, including compute-efficient and data-efficient
training details, data filtering methods, sequence packing scripts,
pre-training data in WebDataset format, FSDP-based training codebase, and both
base and instruction-tuned model checkpoints. We redefine "fully open" for
multimodal LLMs as the complete release of: 1) the training codebase, 2)
detailed data filtering techniques, and 3) all pre-training and supervised
fine-tuning data used to develop the model.
Anjiang Wei, Tarun Suresh, Jiannan Cao, Naveen Kannan, Yuheng Wu, Kai Yan, Thiago S. F. X. Teixeira, Ke Wang, Alex Aiken
342
Inductive program synthesis, or programming by example, requires synthesizing
functions from input-output examples that generalize to unseen inputs. While
large language model agents have shown promise in programming tasks guided by
natural language, their ability to perform inductive program synthesis is
underexplored. Existing evaluation protocols rely on static sets of examples
and held-out tests, offering no feedback when synthesized functions are
incorrect and failing to reflect real-world scenarios such as reverse
engineering. We propose CodeARC, the Code Abstraction and Reasoning Challenge,
a new evaluation framework where agents interact with a hidden target function
by querying it with new inputs, synthesizing candidate functions, and
iteratively refining their solutions using a differential testing oracle. This
interactive setting encourages agents to perform function calls and
self-correction based on feedback. We construct the first large-scale benchmark
for general-purpose inductive program synthesis, featuring 1114 functions.
Among 18 models evaluated, o3-mini performs best with a success rate of 52.7%,
highlighting the difficulty of this task. Fine-tuning LLaMA-3.1-8B-Instruct on
curated synthesis traces yields up to a 31% relative performance gain. CodeARC
provides a more realistic and challenging testbed for evaluating LLM-based
program synthesis and inductive reasoning.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, Saining Xie
304
Visual Self-Supervised Learning (SSL) currently underperforms Contrastive
Language-Image Pretraining (CLIP) in multimodal settings such as Visual
Question Answering (VQA). This multimodal gap is often attributed to the
semantics introduced by language supervision, even though visual SSL and CLIP
models are often trained on different data. In this work, we ask the question:
"Do visual self-supervised approaches lag behind CLIP due to the lack of
language supervision, or differences in the training data?" We study this
question by training both visual SSL and CLIP models on the same MetaCLIP data,
and leveraging VQA as a diverse testbed for vision encoders. In this controlled
setup, visual SSL models scale better than CLIP models in terms of data and
model capacity, and visual SSL performance does not saturate even after scaling
up to 7B parameters. Consequently, we observe visual SSL methods achieve
CLIP-level performance on a wide range of VQA and classic vision benchmarks.
These findings demonstrate that pure visual SSL can match language-supervised
visual pretraining at scale, opening new opportunities for vision-centric
representation learning.
Despite remarkable advancements in video depth estimation, existing methods
exhibit inherent limitations in achieving geometric fidelity through the
affine-invariant predictions, limiting their applicability in reconstruction
and other metrically grounded downstream tasks. We propose GeometryCrafter, a
novel framework that recovers high-fidelity point map sequences with temporal
coherence from open-world videos, enabling accurate 3D/4D reconstruction,
camera parameter estimation, and other depth-based applications. At the core of
our approach lies a point map Variational Autoencoder (VAE) that learns a
latent space agnostic to video latent distributions for effective point map
encoding and decoding. Leveraging the VAE, we train a video diffusion model to
model the distribution of point map sequences conditioned on the input videos.
Extensive evaluations on diverse datasets demonstrate that GeometryCrafter
achieves state-of-the-art 3D accuracy, temporal consistency, and generalization
capability.
Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, Bo Han
282
Numerous applications of large language models (LLMs) rely on their ability
to perform step-by-step reasoning. However, the reasoning behavior of LLMs
remains poorly understood, posing challenges to research, development, and
safety. To address this gap, we introduce landscape of thoughts-the first
visualization tool for users to inspect the reasoning paths of chain-of-thought
and its derivatives on any multi-choice dataset. Specifically, we represent the
states in a reasoning path as feature vectors that quantify their distances to
all answer choices. These features are then visualized in two-dimensional plots
using t-SNE. Qualitative and quantitative analysis with the landscape of
thoughts effectively distinguishes between strong and weak models, correct and
incorrect answers, as well as different reasoning tasks. It also uncovers
undesirable reasoning patterns, such as low consistency and high uncertainty.
Additionally, users can adapt our tool to a model that predicts the property
they observe. We showcase this advantage by adapting our tool to a lightweight
verifier that evaluates the correctness of reasoning paths. The code is
publicly available at: https://github.com/tmlr-group/landscape-of-thoughts.
Large Language Models (LLMs) can achieve enhanced complex problem-solving
through test-time computing scaling, yet this often entails longer contexts and
numerous reasoning token costs. In this paper, we propose an efficient
test-time scaling method that trains LLMs on code-related reasoning
trajectories, facilitating their reduction of excess thinking tokens while
maintaining performance. First, we create Z1-Code-Reasoning-107K, a curated
dataset of simple and complex coding problems paired with their short and long
solution trajectories. Second, we present a novel Shifted Thinking Window to
mitigate overthinking overhead by removing context-delimiting tags (e.g.,
<think>. . . </think>) and capping reasoning tokens. Trained with long and
short trajectory data and equipped with Shifted Thinking Window, our model,
Z1-7B, demonstrates the ability to adjust its reasoning level as the complexity
of problems and exhibits efficient test-time scaling across different reasoning
tasks that matches R1-Distill-Qwen-7B performance with about 30% of its average
thinking tokens. Notably, fine-tuned with only code trajectories, Z1-7B
demonstrates generalization to broader reasoning tasks (47.5% on GPQA Diamond).
Our analysis of efficient reasoning elicitation also provides valuable insights
for future research.
Team Cohere, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, Walter Beller-Morales, Alexandre Bérard, Andrew Berneshawi, Anna Bialas, Phil Blunsom, Matt Bobkin, Adi Bongale, Sam Braun, Maxime Brunet, Samuel Cahyawijaya, David Cairuz, Jon Ander Campos, Cassie Cao, Kris Cao, Roman Castagné, Julián Cendrero, Leila Chan Currie, Yash Chandak, Diane Chang, Giannis Chatziveroglou, Hongyu Chen, Claire Cheng, Alexis Chevalier, Justin T. Chiu, Eugene Cho, Eugene Choi, Eujeong Choi, Tim Chung, Volkan Cirik, Ana Cismaru, Pierre Clavier, Henry Conklin, Lucas Crawhall-Stein, Devon Crouse, Andres Felipe Cruz-Salinas, Ben Cyrus, Daniel D'souza, Hugo Dalla-Torre, John Dang, William Darling, Omar Darwiche Domingues, Saurabh Dash, Antoine Debugne, Théo Dehaze, Shaan Desai, Joan Devassy, Rishit Dholakia, Kyle Duffy, Ali Edalati, Ace Eldeib, Abdullah Elkady, Sarah Elsharkawy, Irem Ergün, Beyza Ermis, Marzieh Fadaee, Boyu Fan, Lucas Fayoux, Yannis Flet-Berliac, Nick Frosst, Matthias Gallé, Wojciech Galuba, Utsav Garg, Matthieu Geist, Mohammad Gheshlaghi Azar, Seraphina Goldfarb-Tarrant, Tomas Goldsack, Aidan Gomez, Victor Machado Gonzaga, Nithya Govindarajan, Manoj Govindassamy, Nathan Grinsztajn, Nikolas Gritsch, Patrick Gu, Shangmin Guo, Kilian Haefeli, Rod Hajjar, Tim Hawes, Jingyi He, Sebastian Hofstätter, Sungjin Hong, Sara Hooker, Tom Hosking, Stephanie Howe, Eric Hu, Renjie Huang, Hemant Jain, Ritika Jain, Nick Jakobi, Madeline Jenkins, JJ Jordan, Dhruti Joshi, Jason Jung, Trushant Kalyanpur, Siddhartha Rao Kamalakara, Julia Kedrzycki, Gokce Keskin, Edward Kim, Joon Kim, Wei-Yin Ko, Tom Kocmi, Michael Kozakov, Wojciech Kryściński, Arnav Kumar Jain, Komal Kumar Teru, Sander Land, Michael Lasby, Olivia Lasche, Justin Lee, Patrick Lewis, Jeffrey Li, Jonathan Li, Hangyu Lin, Acyr Locatelli, Kevin Luong, Raymond Ma, Lukas Mach, Marina Machado, Joanne Magbitang, Brenda Malacara Lopez, Aryan Mann, Kelly Marchisio, Olivia Markham, Alexandre Matton, Alex McKinney, Dominic McLoughlin, Jozef Mokry, Adrien Morisot, Autumn Moulder, Harry Moynehan, Maximilian Mozes, Vivek Muppalla, Lidiya Murakhovska, Hemangani Nagarajan, Alekhya Nandula, Hisham Nasir, Shauna Nehra, Josh Netto-Rosen, Daniel Ohashi, James Owers-Bardsley, Jason Ozuzu, Dennis Padilla, Gloria Park, Sam Passaglia, Jeremy Pekmez, Laura Penstone, Aleksandra Piktus, Case Ploeg, Andrew Poulton, Youran Qi, Shubha Raghvendra, Miguel Ramos, Ekagra Ranjan, Pierre Richemond, Cécile Robert-Michon, Aurélien Rodriguez, Sudip Roy, Laura Ruis, Louise Rust, Anubhav Sachan, Alejandro Salamanca, Kailash Karthik Saravanakumar, Isha Satyakam, Alice Schoenauer Sebag, Priyanka Sen, Sholeh Sepehri, Preethi Seshadri, Ye Shen, Tom Sherborne, Sylvie Chang Shi, Sanal Shivaprasad, Vladyslav Shmyhlo, Anirudh Shrinivason, Inna Shteinbuk, Amir Shukayev, Mathieu Simard, Ella Snyder, Ava Spataru, Victoria Spooner, Trisha Starostina, Florian Strub, Yixuan Su, Jimin Sun, Dwarak Talupuru, Eugene Tarassov, Elena Tommasone, Jennifer Tracey, Billy Trend, Evren Tumer, Ahmet Üstün, Bharat Venkitesh, David Venuto, Pat Verga, Maxime Voisin, Alex Wang, Donglu Wang, Shijian Wang, Edmond Wen, Naomi White, Jesse Willman, Marysia Winkels, Chen Xia, Jessica Xie, Minjie Xu, Bowen Yang, Tan Yi-Chern, Ivan Zhang, Zhenyu Zhao, Zhoujie Zhao
263
In this report we describe the development of Command A, a powerful large
language model purpose-built to excel at real-world enterprise use cases.
Command A is an agent-optimised and multilingual-capable model, with support
for 23 languages of global business, and a novel hybrid architecture balancing
efficiency with top of the range performance. It offers best-in-class Retrieval
Augmented Generation (RAG) capabilities with grounding and tool use to automate
sophisticated business processes. These abilities are achieved through a
decentralised training approach, including self-refinement algorithms and model
merging techniques. We also include results for Command R7B which shares
capability and architectural similarities to Command A. Weights for both models
have been released for research purposes. This technical report details our
original training pipeline and presents an extensive evaluation of our models
across a suite of enterprise-relevant tasks and public benchmarks,
demonstrating excellent performance and efficiency.
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang
222
Computer use agents automate digital tasks by directly interacting with
graphical user interfaces (GUIs) on computers and mobile devices, offering
significant potential to enhance human productivity by completing an open-ended
space of user queries. However, current agents face significant challenges:
imprecise grounding of GUI elements, difficulties with long-horizon task
planning, and performance bottlenecks from relying on single generalist models
for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
compositional framework that delegates cognitive responsibilities across
various generalist and specialist models. We propose a novel
Mixture-of-Grounding technique to achieve precise GUI localization and
introduce Proactive Hierarchical Planning, dynamically refining action plans at
multiple temporal scales in response to evolving observations. Evaluations
demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
on three prominent computer use benchmarks. Specifically, Agent S2 achieves
18.9% and 32.7% relative improvements over leading baseline agents such as
Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
Moreover, Agent S2 generalizes effectively to other operating systems and
applications, surpassing previous best methods by 52.8% on WindowsAgentArena
and by 16.52% on AndroidWorld relatively. Code available at
https://github.com/simular-ai/Agent-S.
Kai Yan, Yufei Xu, Zhengyin Du, Xuesong Yao, Zheyu Wang, Xiaowen Guo, Jiecao Chen
2215
The rapid escalation from elementary school-level to frontier problems of the
difficulty for LLM benchmarks in recent years have weaved a miracle for
researchers that we are only inches away from surpassing human intelligence.
However, is the LLMs' remarkable reasoning ability indeed comes from true
intelligence by human standards, or are they simply reciting solutions
witnessed during training at an Internet level? To study this problem, we
propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's
recitation behavior when asked simple reasoning problems but with conditions
subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly,
we found existing cutting-edge LLMs unanimously exhibits extremely severe
recitation behavior; by changing one phrase in the condition, top models such
as OpenAI-o1 and DeepSeek-R1 can suffer 60% performance loss on elementary
school-level arithmetic and reasoning problems. Such findings are a wake-up
call to the LLM community that compels us to re-evaluate the true intelligence
level of cutting-edge LLMs.
Evaluating large language models (LLMs) effectively remains a critical
bottleneck, as traditional static benchmarks suffer from saturation and
contamination, while human evaluations are costly and slow. This hinders timely
or domain-specific assessment, crucial for real-world applications. We
introduce YourBench, a novel, open-source framework that addresses these
limitations by enabling dynamic, automated generation of reliable, up-to-date,
and domain-tailored benchmarks cheaply and without manual annotation, directly
from user-provided documents. We demonstrate its efficacy by replicating 7
diverse MMLU subsets using minimal source text, achieving this for under 15 USD
in total inference costs while perfectly preserving the relative model
performance rankings (Spearman Rho = 1) observed on the original benchmark. To
ensure that YourBench generates data grounded in provided input instead of
relying on posterior parametric knowledge in models, we also introduce
Tempora-0325, a novel dataset of over 7K diverse documents, published
exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models
from 7 major families across varying scales (3-671B parameters) to validate the
quality of generated evaluations through rigorous algorithmic checks (e.g.,
citation grounding) and human assessments. We release the YourBench library,
the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all
evaluation and inference traces to facilitate reproducible research and empower
the community to generate bespoke benchmarks on demand, fostering more relevant
and trustworthy LLM evaluation.
Yucheng Shi, Wenhao Yu, Wenlin Yao, Wenhu Chen, Ninghao Liu
213
GUI agents, powered by large foundation models, can interact with digital
interfaces, enabling various applications in web automation, mobile navigation,
and software testing. However, their increasing autonomy has raised critical
concerns about their security, privacy, and safety. This survey examines the
trustworthiness of GUI agents in five critical dimensions: security
vulnerabilities, reliability in dynamic environments, transparency and
explainability, ethical considerations, and evaluation methodologies. We also
identify major challenges such as vulnerability to adversarial attacks,
cascading failure modes in sequential decision-making, and a lack of realistic
evaluation benchmarks. These issues not only hinder real-world deployment but
also call for comprehensive mitigation strategies beyond task success. As GUI
agents become more widespread, establishing robust safety standards and
responsible development practices is essential. This survey provides a
foundation for advancing trustworthy GUI agents through systematic
understanding and future research.
Pablo Ruiz-Ponce, German Barquero, Cristina Palmero, Sergio Escalera, José García-Rodríguez
192
Generating human motion guided by conditions such as textual descriptions is
challenging due to the need for datasets with pairs of high-quality motion and
their corresponding conditions. The difficulty increases when aiming for finer
control in the generation. To that end, prior works have proposed to combine
several motion diffusion models pre-trained on datasets with different types of
conditions, thus allowing control with multiple conditions. However, the
proposed merging strategies overlook that the optimal way to combine the
generation processes might depend on the particularities of each pre-trained
generative model and also the specific textual descriptions. In this context,
we introduce MixerMDM, the first learnable model composition technique for
combining pre-trained text-conditioned human motion diffusion models. Unlike
previous approaches, MixerMDM provides a dynamic mixing strategy that is
trained in an adversarial fashion to learn to combine the denoising process of
each model depending on the set of conditions driving the generation. By using
MixerMDM to combine single- and multi-person motion diffusion models, we
achieve fine-grained control on the dynamics of every person individually, and
also on the overall interaction. Furthermore, we propose a new evaluation
technique that, for the first time in this task, measures the interaction and
individual quality by computing the alignment between the mixed generated
motions and their conditions as well as the capabilities of MixerMDM to adapt
the mixing throughout the denoising process depending on the motions to mix.
The rapid advancement of multi-modal language models (MLLMs) like GPT-4o has
propelled the development of Omni language models, designed to process and
proactively respond to continuous streams of multi-modal data. Despite their
potential, evaluating their real-world interactive capabilities in streaming
video contexts remains a formidable challenge. In this work, we introduce
OmniMMI, a comprehensive multi-modal interaction benchmark tailored for
OmniLLMs in streaming video contexts. OmniMMI encompasses over 1,121 videos and
2,290 questions, addressing two critical yet underexplored challenges in
existing video benchmarks: streaming video understanding and proactive
reasoning, across six distinct subtasks. Moreover, we propose a novel
framework, Multi-modal Multiplexing Modeling (M4), designed to enable an
inference-efficient streaming model that can see, listen while generating.
Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, Kam-Fai Wong
172
Recent advancements in Large Language Models (LLMs) have significantly
enhanced their ability to perform complex reasoning tasks, transitioning from
fast and intuitive thinking (System 1) to slow and deep reasoning (System 2).
While System 2 reasoning improves task accuracy, it often incurs substantial
computational costs due to its slow thinking nature and inefficient or
unnecessary reasoning behaviors. In contrast, System 1 reasoning is
computationally efficient but leads to suboptimal performance. Consequently, it
is critical to balance the trade-off between performance (benefits) and
computational costs (budgets), giving rise to the concept of reasoning economy.
In this survey, we provide a comprehensive analysis of reasoning economy in
both the post-training and test-time inference stages of LLMs, encompassing i)
the cause of reasoning inefficiency, ii) behavior analysis of different
reasoning patterns, and iii) potential solutions to achieve reasoning economy.
By offering actionable insights and highlighting open challenges, we aim to
shed light on strategies for improving the reasoning economy of LLMs, thereby
serving as a valuable resource for advancing research in this evolving area. We
also provide a public repository to continually track developments in this
fast-evolving field.
Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, Anna Rohrbach
151
Scaling test-time compute has emerged as a key strategy for enhancing the
reasoning capabilities of large language models (LLMs), particularly in tasks
like mathematical problem-solving. A traditional approach, Self-Consistency
(SC), generates multiple solutions to a problem and selects the most common
answer via majority voting. Another common method involves scoring each
solution with a reward model (verifier) and choosing the best one. Recent
advancements in Generative Reward Models (GenRM) reframe verification as a
next-token prediction task, enabling inference-time scaling along a new axis.
Specifically, GenRM generates multiple verification chains-of-thought to score
each solution. Under a limited inference budget, this introduces a fundamental
trade-off: should you spend the budget on scaling solutions via SC or generate
fewer solutions and allocate compute to verification via GenRM? To address
this, we evaluate GenRM against SC under a fixed inference budget.
Interestingly, we find that SC is more compute-efficient than GenRM for most
practical inference budgets across diverse models and datasets. For instance,
GenRM first matches SC after consuming up to 8x the inference compute and
requires significantly more compute to outperform it. Furthermore, we derive
inference scaling laws for the GenRM paradigm, revealing that compute-optimal
inference favors scaling solution generation more aggressively than scaling the
number of verifications. Our work provides practical guidance on optimizing
test-time scaling by balancing solution generation and verification. The code
is available at https://github.com/nishadsinghi/sc-genrm-scaling.
Jewon Lee, Ki-Ung Song, Seungmin Yang, Donguk Lim, Jaeyeon Kim, Wooksu Shin, Bo-Kyeong Kim, Yong Jae Lee, Tae-Ho Kim
152
Visual token reduction lowers inference costs caused by extensive image
features in large vision-language models (LVLMs). Unlike relevant studies that
prune tokens in self-attention-only LVLMs, our work uniquely addresses
cross-attention-based models, which achieve superior performance. We identify
that the key-value (KV) cache size for image tokens in cross-attention layers
significantly exceeds that of text tokens in self-attention layers, posing a
major compute bottleneck. To mitigate this issue, we exploit the sparse nature
in cross-attention maps to selectively prune redundant visual features. Our
Trimmed Llama effectively reduces KV cache demands without requiring additional
training. By benefiting from 50%-reduced visual features, our model can reduce
inference latency and memory usage while achieving benchmark parity.
Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, Maosong Sun, Yang Liu
113
Recently, model merging methods have demonstrated powerful strengths in
combining abilities on various tasks from multiple Large Language Models
(LLMs). While previous model merging methods mainly focus on merging
homogeneous models with identical architecture, they meet challenges when
dealing with Multimodal Large Language Models (MLLMs) with inherent
heterogeneous property, including differences in model architecture and the
asymmetry in the parameter space. In this work, we propose AdaMMS, a novel
model merging method tailored for heterogeneous MLLMs. Our method tackles the
challenges in three steps: mapping, merging and searching. Specifically, we
first design mapping function between models to apply model merging on MLLMs
with different architecture. Then we apply linear interpolation on model
weights to actively adapt the asymmetry in the heterogeneous MLLMs. Finally in
the hyper-parameter searching step, we propose an unsupervised hyper-parameter
selection method for model merging. As the first model merging method capable
of merging heterogeneous MLLMs without labeled data, extensive experiments on
various model combinations demonstrated that AdaMMS outperforms previous model
merging methods on various vision-language benchmarks.
Test-time scaling has emerged as a powerful technique for enhancing the
reasoning capabilities of large language models. However, its effectiveness in
medical reasoning remains uncertain, as the medical domain fundamentally
differs from mathematical tasks in terms of knowledge representation and
decision-making processes. In this paper, we provide the first comprehensive
investigation of test-time scaling for medical reasoning and present m1, a
simple yet effective approach that increases a model's medical reasoning
capability at inference. Our evaluation across diverse medical tasks
demonstrates that test-time scaling consistently enhances medical reasoning,
enabling lightweight fine-tuned models under 10B parameters to establish new
state-of-the-art performance, while our 32B model rivals previous 70B-scale
medical LLMs. However, we identify an optimal reasoning token budget of
approximately 4K, beyond which performance may degrade due to overthinking.
Budget forcing, which extends test-time computation through iterative prompts,
helps models double-check answers but does not necessarily improve the overall
medical QA performance and, in some cases, even introduces errors into
previously correct responses. Our case-by-case analysis identifies insufficient
medical knowledge as a key bottleneck that prevents further performance gains
through test-time scaling. We find that increasing data scale, improving data
quality, and expanding model capacity consistently enhance medical knowledge
grounding, enabling continued performance improvements, particularly on
challenging medical benchmarks where smaller models reach saturation. These
findings underscore fundamental differences between medical and mathematical
reasoning in LLMs, highlighting that enriched medical knowledge, other than
increased reasoning depth alone, is essential for realizing the benefits of
test-time scaling.
Inference-time scaling can enhance the reasoning capabilities of large
language models (LLMs) on complex problems that benefit from step-by-step
problem solving. Although lengthening generated scratchpads has proven
effective for mathematical tasks, the broader impact of this approach on other
tasks remains less clear. In this work, we investigate the benefits and
limitations of scaling methods across nine state-of-the-art models and eight
challenging tasks, including math and STEM reasoning, calendar planning,
NP-hard problems, navigation, and spatial reasoning. We compare conventional
models (e.g., GPT-4o) with models fine-tuned for inference-time scaling (e.g.,
o1) through evaluation protocols that involve repeated model calls, either
independently or sequentially with feedback. These evaluations approximate
lower and upper performance bounds and potential for future performance
improvements for each model, whether through enhanced training or multi-model
inference systems. Our extensive empirical analysis reveals that the advantages
of inference-time scaling vary across tasks and diminish as problem complexity
increases. In addition, simply using more tokens does not necessarily translate
to higher accuracy in these challenging regimes. Results from multiple
independent runs with conventional models using perfect verifiers show that,
for some tasks, these models can achieve performance close to the average
performance of today's most advanced reasoning models. However, for other
tasks, a significant performance gap remains, even in very high scaling
regimes. Encouragingly, all models demonstrate significant gains when inference
is further scaled with perfect verifiers or strong feedback, suggesting ample
potential for future improvements.
Text-to-SQL is a challenging task involving multiple reasoning-intensive
subtasks, including natural language understanding, database schema
comprehension, and precise SQL query formulation. Existing approaches often
rely on handcrafted reasoning paths with inductive biases that can limit their
overall effectiveness. Motivated by the recent success of reasoning-enhanced
models such as DeepSeek R1 and OpenAI o1, which effectively leverage
reward-driven self-exploration to enhance reasoning capabilities and
generalization, we propose a novel set of partial rewards tailored specifically
for the Text-to-SQL task. Our reward set includes schema-linking, AI feedback,
n-gram similarity, and syntax check, explicitly designed to address the reward
sparsity issue prevalent in reinforcement learning (RL). Leveraging group
relative policy optimization (GRPO), our approach explicitly encourages large
language models (LLMs) to develop intrinsic reasoning skills necessary for
accurate SQL query generation. With models of different sizes, we demonstrate
that RL-only training with our proposed rewards consistently achieves higher
accuracy and superior generalization compared to supervised fine-tuning (SFT).
Remarkably, our RL-trained 14B-parameter model significantly outperforms larger
proprietary models, e.g. o3-mini by 4% and Gemini-1.5-Pro-002 by 3% on the BIRD
benchmark. These highlight the efficacy of our proposed RL-training framework
with partial rewards for enhancing both accuracy and reasoning capabilities in
Text-to-SQL tasks.
Lucas Ventura, Antoine Yang, Cordelia Schmid, Gül Varol
72
We address the task of video chaptering, i.e., partitioning a long video
timeline into semantic units and generating corresponding chapter titles. While
relatively underexplored, automatic chaptering has the potential to enable
efficient navigation and content retrieval in long-form videos. In this paper,
we achieve strong chaptering performance on hour-long videos by efficiently
addressing the problem in the text domain with our 'Chapter-Llama' framework.
Specifically, we leverage a pretrained large language model (LLM) with large
context window, and feed as input (i) speech transcripts and (ii) captions
describing video frames, along with their respective timestamps. Given the
inefficiency of exhaustively captioning all frames, we propose a lightweight
speech-guided frame selection strategy based on speech transcript content, and
experimentally demonstrate remarkable advantages. We train the LLM to output
timestamps for the chapter boundaries, as well as free-form chapter titles.
This simple yet powerful approach scales to processing one-hour long videos in
a single forward pass. Our results demonstrate substantial improvements (e.g.,
45.3 vs 26.7 F1 score) over the state of the art on the recent VidChapters-7M
benchmark. To promote further research, we release our code and models at our
project page.
Large language models (LLMs) possess impressive linguistic capabilities but
often fail to faithfully retain factual knowledge, leading to hallucinations
and unreliable outputs. Understanding LLMs' knowledge deficiencies by
exhaustively evaluating against full-scale knowledge bases is computationally
prohibitive, especially for closed-weight models. We propose stochastic error
ascent (SEA), a scalable and efficient framework for discovering knowledge
deficiencies (errors) in closed-weight LLMs under a strict query budget. Rather
than naively probing all knowledge candidates, SEA formulates error discovery
as a stochastic optimization process: it iteratively retrieves new high-error
candidates by leveraging the semantic similarity to previously observed
failures. To further enhance search efficiency and coverage, SEA employs
hierarchical retrieval across document and paragraph levels, and constructs a
relation directed acyclic graph to model error propagation and identify
systematic failure modes. Empirically, SEA uncovers 40.7x more knowledge errors
than Automated Capability Discovery and 26.7% more than AutoBencher, while
reducing the cost-per-error by 599x and 9x, respectively. Human evaluation
confirms the high quality of generated questions, while ablation and
convergence analyses validate the contribution of each component in SEA.
Further analysis on the discovered errors reveals correlated failure patterns
across LLM families and recurring deficits, highlighting the need for better
data coverage and targeted fine-tuning in future LLM development.
Human hands play a central role in interacting, motivating increasing
research in dexterous robotic manipulation. Data-driven embodied AI algorithms
demand precise, large-scale, human-like manipulation sequences, which are
challenging to obtain with conventional reinforcement learning or real-world
teleoperation. To address this, we introduce ManipTrans, a novel two-stage
method for efficiently transferring human bimanual skills to dexterous robotic
hands in simulation. ManipTrans first pre-trains a generalist trajectory
imitator to mimic hand motion, then fine-tunes a specific residual module under
interaction constraints, enabling efficient learning and accurate execution of
complex bimanual tasks. Experiments show that ManipTrans surpasses
state-of-the-art methods in success rate, fidelity, and efficiency. Leveraging
ManipTrans, we transfer multiple hand-object datasets to robotic hands,
creating DexManipNet, a large-scale dataset featuring previously unexplored
tasks like pen capping and bottle unscrewing. DexManipNet comprises 3.3K
episodes of robotic manipulation and is easily extensible, facilitating further
policy training for dexterous hands and enabling real-world deployments.
Reconstructing sharp 3D representations from blurry multi-view images are
long-standing problem in computer vision. Recent works attempt to enhance
high-quality novel view synthesis from the motion blur by leveraging
event-based cameras, benefiting from high dynamic range and microsecond
temporal resolution. However, they often reach sub-optimal visual quality in
either restoring inaccurate color or losing fine-grained details. In this
paper, we present DiET-GS, a diffusion prior and event stream-assisted motion
deblurring 3DGS. Our framework effectively leverages both blur-free event
streams and diffusion prior in a two-stage training strategy. Specifically, we
introduce the novel framework to constraint 3DGS with event double integral,
achieving both accurate color and well-defined details. Additionally, we
propose a simple technique to leverage diffusion prior to further enhance the
edge details. Qualitative and quantitative results on both synthetic and
real-world data demonstrate that our DiET-GS is capable of producing
significantly better quality of novel views compared to the existing baselines.
Our project page is https://diet-gs.github.io
We propose a unified framework that integrates object detection (OD) and
visual grounding (VG) for remote sensing (RS) imagery. To support conventional
OD and establish an intuitive prior for VG task, we fine-tune an open-set
object detector using referring expression data, framing it as a partially
supervised OD task. In the first stage, we construct a graph representation of
each image, comprising object queries, class embeddings, and proposal
locations. Then, our task-aware architecture processes this graph to perform
the VG task. The model consists of: (i) a multi-branch network that integrates
spatial, visual, and categorical features to generate task-aware proposals, and
(ii) an object reasoning network that assigns probabilities across proposals,
followed by a soft selection mechanism for final referring object localization.
Our model demonstrates superior performance on the OPT-RSVG and DIOR-RSVG
datasets, achieving significant improvements over state-of-the-art methods
while retaining classical OD capabilities. The code will be available in our
repository: https://github.com/rd20karim/MB-ORES.