StarHarness：企業環境中以分層搜索演化測試框架

摘要

我們提出 StarHarness，一個在保持模型權重不變的條件下演化環境特定代理執行框架的框架。演化後的執行框架可包含提示與任務表述、工具介面、技能、由 MCP 支援的提供者、子代理結構，以及代理迴圈配置。StarHarness 根據基線失敗行為對任務進行分層，建構出精簡的演化池；它將提議者可見的搜尋任務與對提議者隱藏的選擇任務分開，並保留一組留出任務以評估泛化能力。在 ITBench SRE、EnterpriseOps-Gym ITSM 與 AutomationBench Finance 中，每個環境在接受 4 至 12 次變更後，執行框架演化將完整基準測試的效能相對於預設執行框架提升了 20 至 35 個百分點。這些增益在未參與演化的任務上持續存在，且無需重新演化即可遷移至 GPT 與 Qwen 模型系列。追蹤分析將這些改進與介面修復、環境慣例，以及能壓縮搜尋空間的運維知識聯繫起來；在若干情境中，這也伴隨著更少的假陽性診斷與更短的軌跡。因此，StarHarness 提供了在工具密集型企業任務中減少模型與環境之間持續不匹配的實用途徑。

English

We present StarHarness, a framework for evolving environment-specific agent harnesses while keeping model weights fixed. The evolved harness can include prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. StarHarness constructs a compact evolution pool by stratifying tasks according to baseline failure behavior, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improves full-benchmark performance by 20-35 percentage points over the default harness after 4-12 accepted changes per environment. These gains persist on tasks excluded from evolution and transfer without re-evolution across GPT and Qwen model families. Trace analysis links the improvements to interface repairs, environment conventions, and operational knowledge that compresses search, with fewer false-positive diagnoses and shorter trajectories in several settings. StarHarness therefore offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks.