TRON: 面向视觉推理强化学习的目标规则可验证在线环境

摘要

用于视觉推理的强化学习（RL）需要可扩展、可验证且可控的训练信号。现有的视觉RL后训练基于静态精选数据集进行训练，这些数据集包含固定的图像-问题-答案样本，其规模受限于数据收集预算。本文中，我们提出TRON（Targeted, Rule-verifiable Online eNvironments，即目标导向、规则可验证的在线环境），一种在线环境基座：训练推演由可控的生成器-验证器程序按需生成，该程序采样新的潜在视觉状态，渲染图像，提出问题，并精确验证答案。因此，单次运行即可按当前课程所需难度生成无界的新实例流。当前的TRON套件包含520个环境，分为五个能力类别（空间、数学、图表、模式/逻辑和计数）；同一基座既支持在所有类别上训练的单一完整模型，也支持按类别的能力专精模型，无需额外数据收集。我们还引入了一项基座分析，涵盖生成可靠性、实例与层级多样性、跨环境近重复样本以及基础模型按难度划分的通过率。采用METHOD的RL后训练在Qwen3-VL-4B、Qwen2.5-VL-7B和MiMo-VL-7B-SFT上持续提升了十个外部多模态推理基准的性能。

English

Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.