EnvHarness：喚醒靜態世界以促進智能體學習

摘要

LLM 智慧體透過與環境互動來學習，然而這些環境皆為手工構建且靜態不變：它們對智慧體的弱點視而不見，並在智慧體進步的同時迅速被拋在身後。儘管近期環境生成方法試圖解決此問題，但它們需要特定領域的處理流程，依賴昂貴或不可靠的驗證器，且仍會產生靜態環境。為減輕從零開始重建環境的工程負擔，我們提出環境韁繩（EnvHarness），這是一個由可插拔元件組成的可程式化層，它能包裹靜態環境以重塑其行為，而無需修改底層邏輯。EnvHarness 透過標準介面運作，適用於多種不同領域，同時確保每個重塑後的環境皆保留其原始驗證器。為自動化此流程，我們引入 EnvRigger，它將目標策略視為黑箱，觀察其執行軌跡以針對診斷出的弱點合成 EnvHarness 元件，並透過全新展開來驗證這些元件。在涵蓋四個領域的五個基準測試中，EnvHarness 的表現優於原始環境及特定領域的環境生成流程，在保留實例上實現高達 9.0 分的提升，同時減少了 9.8% 的執行步驟。此外，EnvHarness 為強化學習提供了更優越的最佳化訊號，使策略與其環境得以持續進行有針對性的共同演化。

English

LLM agents learn by interacting with environments, yet these environments are hand-built and static: blind to an agent's weaknesses, and quickly left behind as it improves. While recent environment generation methods attempt to address this, they require domain-specific pipelines, rely on expensive or unreliable verifiers, and still produce static environments. To alleviate the engineering burden of rebuilding environments from scratch, we propose Environment Harness (EnvHarness), a programmable layer of plug-in components that wraps a static environment to reshape its behavior without modifying the underlying logic. Operating through standard interfaces, EnvHarness applies across diverse domains while ensuring every reshaped environment retains its original verifier. To automate this process, we introduce EnvRigger, which treats the target policy as a black box, observing its execution trajectories to synthesize EnvHarness components targeting diagnosed flaws, and validating them via fresh rollouts. Across five benchmarks in four domains, EnvHarness outperforms both original environments and domain-specific environment generation pipelines, achieving up to a 9.0-point improvement on held-out instances with 9.8% fewer execution steps. Furthermore, EnvHarness provides a superior optimization signal for reinforcement learning, enabling continuous, targeted co-evolution of the policy and its environment.