TRON：針對視覺推理強化學習的目標規則可驗證線上環境

摘要

視覺推理的強化學習需要可擴展、可驗證且可控的訓練信號。現有的視覺強化學習後訓練依賴於靜態精選數據集，其樣本為固定的圖像-問題-答案組合，受到收集預算的限制。本研究提出TRON（目標導向、規則可驗證之線上環境），這是一個線上環境基底：訓練的軌跡由可控的生成器-驗證器程式按需求產生，該程式會抽樣新的潛在視覺狀態、渲染圖像、提出問題，並精確驗證答案。因此，單次運行即可根據當前課程所需的難度等級，抽取無限的新鮮實例。目前的TRON套件包含520個環境，分為五大能力類別（空間、數學、圖表、模式/邏輯與計數）；該基底同時支援在所有類別上訓練的單一完整模型，以及按類別劃分的能力專精模型，且無需額外收集數據。我們亦針對基底進行分析，涵蓋生成可靠性、實例與層級多樣性、跨環境近似重複，以及按難度劃分的基礎模型通過率。將強化學習後訓練結合METHOD方法後，在Qwen3-VL-4B、Qwen2.5-VL-7B及MiMo-VL-7B-SFT模型上，於十項外部多模態推理基準測試中持續獲得性能提升。

English

Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.