高性能强化学习环境的自动生成

摘要

将复杂强化学习环境转化为高性能实现传统上需要数月的专业工程开发。我们提出了一种可复用的方法——包含通用提示模板、分层验证和迭代式智能体辅助修复——仅需不到10美元的计算成本即可生成语义等效的高性能环境。我们展示了跨五个环境的三种差异化工作流：直接翻译（无现有性能实现）：EmuRust（通过Rust并行化实现Game Boy模拟器的PPO速度提升1.5倍）和首个GPU并行宝可梦对战模拟器PokeJAX（随机动作5亿步/秒，PPO算法1520万步/秒；较TypeScript参考实现提升22,320倍）。基于现有性能实现的验证翻译：在匹配GPU批处理量时与MJX持平（1.04倍），在HalfCheetah JAX环境中达到Brax的5倍吞吐量；Puffer Pong环境实现PPO加速42倍。新环境创建：首个可部署的JAX版宝可梦卡牌引擎TCGJax（随机动作71.7万步/秒，PPO算法15.3万步/秒；较Python参考实现提升6.6倍），该引擎从网络抓取的规范自动生成。当模型参数量达2亿时，环境开销降至训练时间的4%以下。分层验证（属性测试、交互测试和推演测试）确认所有五个环境均保持语义等效；跨后端策略迁移证实五个环境均实现零模拟差异。TCGJax基于未公开的私有参考实现生成，可作为智能体预训练数据污染的对照基准。本文提供了完整细节——包括代表性提示、验证方法和全部结果——使得编码智能体能直接从论文复现所有翻译实现。

English

Translating complex reinforcement learning (RL) environments into high-performance implementations has traditionally required months of specialized engineering. We present a reusable recipe - a generic prompt template, hierarchical verification, and iterative agent-assisted repair - that produces semantically equivalent high-performance environments for <$10 in compute cost. We demonstrate three distinct workflows across five environments. Direct translation (no prior performance implementation exists): EmuRust (1.5x PPO speedup via Rust parallelism for a Game Boy emulator) and PokeJAX, the first GPU-parallel Pokemon battle simulator (500M SPS random action, 15.2M SPS PPO; 22,320x over the TypeScript reference). Translation verified against existing performance implementations: throughput parity with MJX (1.04x) and 5x over Brax at matched GPU batch sizes (HalfCheetah JAX); 42x PPO (Puffer Pong). New environment creation: TCGJax, the first deployable JAX Pokemon TCG engine (717K SPS random action, 153K SPS PPO; 6.6x over the Python reference), synthesized from a web-extracted specification. At 200M parameters, the environment overhead drops below 4% of training time. Hierarchical verification (property, interaction, and rollout tests) confirms semantic equivalence for all five environments; cross-backend policy transfer confirms zero sim-to-sim gap for all five environments. TCGJax, synthesized from a private reference absent from public repositories, serves as a contamination control for agent pretraining data concerns. The paper contains sufficient detail - including representative prompts, verification methodology, and complete results - that a coding agent could reproduce the translations directly from the manuscript.

高性能强化学习环境的自动生成

Automatic Generation of High-Performance RL Environments

摘要

Support