StarHarness：面向企业环境的基于分层搜索的演化测试框架

摘要

我们提出 StarHarness，一个在保持模型权重固定的同时演化特定于环境的智能体外壳的框架。演化后的外壳可包含提示与任务设定、工具接口、技能、基于MCP的提供者、子智能体结构以及智能体循环配置。StarHarness 通过根据基线失败行为对任务进行分层来构建紧凑的演化池，将提案者可见的搜索任务与对提案者隐藏的选择任务分离，并保留留出任务以评估泛化能力。在 ITBench SRE、EnterpriseOps-Gym ITSM 和 AutomationBench Finance 上，每个环境接受4至12次变更后，外壳演化将全基准性能相对于默认外壳提升了20至35个百分点。这些收益在未参与演化的任务上依然存在，并且无需重新演化即可跨 GPT 和 Qwen 模型系列迁移。轨迹分析将改进与接口修复、环境约定以及压缩搜索的操作知识联系起来，在多个场景中出现更少的假阳性诊断和更短的轨迹。因此，StarHarness 提供了一种实用的方法，以减少工具丰富的企业任务中持续的模型-环境不匹配。

English

We present StarHarness, a framework for evolving environment-specific agent harnesses while keeping model weights fixed. The evolved harness can include prompt and task framing, tool interfaces, skills, MCP-backed providers, subagent structure, and agent-loop configuration. StarHarness constructs a compact evolution pool by stratifying tasks according to baseline failure behavior, separates proposer-visible search tasks from proposer-hidden selection tasks, and reserves held-out tasks for evaluating generalization. Across ITBench SRE, EnterpriseOps-Gym ITSM, and AutomationBench Finance, harness evolution improves full-benchmark performance by 20-35 percentage points over the default harness after 4-12 accepted changes per environment. These gains persist on tasks excluded from evolution and transfer without re-evolution across GPT and Qwen model families. Trace analysis links the improvements to interface repairs, environment conventions, and operational knowledge that compresses search, with fewer false-positive diagnoses and shorter trajectories in several settings. StarHarness therefore offers a practical way to reduce persistent model-environment mismatch in tool-rich enterprise tasks.