PlannerForge：用於自動駕駛中運動規劃器情境式測試的 LLM 代理

摘要

確保自動駕駛安全是一項關鍵挑戰。情境式測試是用於驗證自動駕駛系統（ADSs）的系統化流程，但其仍是一條碎片化的模組化流程，其中情境生成、檢索、修改、ADS 執行與結果分析由彼此互動極少的獨立工具執行。大型語言模型（LLM）代理已在 ADS 的感知、規劃與控制等子系統中展現潛力。然而，尚無先前研究以統一的 LLM 代理框架涵蓋 ADS 的完整情境式測試流程。我們提出 PlannerForge，一個 LLM 代理框架，其擴展所有情境式測試階段（從情境生成到 ADS 評估），並新增兩個進一步由 LLM 增強的階段：ADS 增強與 ADS 基準測試。我們以 10 個現成 LLM 在 5 種提示條件下，評估 PlannerForge 於所有任務（生成、選擇、修改、模組路由、規劃器測試與增強）的表現。各任務最佳分數介於 0.88 至 1.00，且開源 20-35B 後端在大多數任務上可媲美商業 API。諸如 Qwen3.6:35B 等開源模型在五項任務中的三項可媲美商業 API。將各模組端到端串接後，可保留 83% / 78% 的種子查詢（商業 / 開源）。其在自然語言生成方面優於 Scenario Factory 2.0（Finkeldei et al., 2025）（200 個中可執行者為 193 個 vs. 144 個），並達成 92-96% 的所要求城市、道路與車輛屬性。其在排名第 1 的選擇上優於 BM25（Robertson and Zaragoza, 2009）（92.0% vs. 67.5%），並在物理上有效的編輯方面優於 From-Words-to-Collisions（Gao et al., 2025）（≥94% vs. 31%）。在 N=400 時，成本調校將規劃器成功率從 50.4% 提升至 70.2%，並將碰撞從 19.0% 降至 8.4%，且無需特定領域微調。

English

Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used to validate Autonomous Driving Systems (ADSs), but it remains a fragmented modular pipeline in which scenario generation, retrieval, modification, ADS execution, and results analysis are performed by separate tools with little interaction. Large Language Model (LLM) agents have shown promise across ADS sub-systems such as perception, planning, and control. However, no prior work covers the whole scenario-based testing pipeline for ADSs with a unified LLM-agent framework. We present PlannerForge, an LLM-agent framework that extends all scenario-based testing stages (from Scenario Generation to ADS Assessment) and adds two further LLM-enhanced stages: ADS Enhancement and ADS Benchmarking. We evaluate PlannerForge with 10 off-the-shelf LLMs across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing, and Enhancement) under 5 prompt conditions. Best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends match commercial APIs on most tasks. Open-source models such as Qwen3.6:35B match commercial APIs on three of the five tasks. Chaining the modules end-to-end retains 83% / 78% of seed queries (commercial / open). It outperforms Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-language generation (193 vs. 144 executable of 200) and realises 92-96% of requested city, road and vehicle attributes. It outperforms BM25 (Robertson and Zaragoza, 2009) at rank 1 selection (92.0% vs. 67.5%) and From-Words-to-Collisions (Gao et al., 2025) on physically valid edits (>=94% vs. 31%). At N=400, cost-tuning lifts planner success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4%, without domain-specific fine-tuning.