基於樹結構的對話強化策略優化用於紅隊攻擊

摘要

尽管近期在人工智能安全领域取得了快速进展，当前的大型语言模型在多轮交互场景下仍易受对抗性攻击的威胁，攻击者通过策略性地调整对话轮次中的提示，提出了更为严峻且现实的挑战。现有发现安全漏洞的方法，或依赖于人工红队与专家合作，或采用基于预设模板和人工筛选攻击数据的自动化手段，且大多聚焦于单轮攻击。然而，这些方法未能深入探索多轮攻击的广阔可能性空间，忽视了由复杂对话动态及策略性对话规划所催生的新型攻击路径。鉴于最新研究显示，相较于单轮攻击，大型语言模型对多轮攻击表现出显著更高的脆弱性，这一空白尤为关键。我们提出了DialTree-RPO，一种结合树搜索的在线策略强化学习框架，它将对话视为序列决策问题，无需人工筛选数据即可自主发现多样化的多轮攻击策略，实现系统性探索。通过大量实验，我们的方法不仅在10个目标模型上相比之前最先进的方法实现了超过25.9%的攻击成功率提升，还通过学习最大化多轮攻击成功率的对话策略，有效揭示了新的攻击策略。

English

Despite recent rapid progress in AI safety, current large language models remain vulnerable to adversarial attacks in multi-turn interaction settings, where attackers strategically adapt their prompts across conversation turns and pose a more critical yet realistic challenge. Existing approaches that discover safety vulnerabilities either rely on manual red-teaming with human experts or employ automated methods using pre-defined templates and human-curated attack data, with most focusing on single-turn attacks. However, these methods did not explore the vast space of possible multi-turn attacks, failing to consider novel attack trajectories that emerge from complex dialogue dynamics and strategic conversation planning. This gap is particularly critical given recent findings that LLMs exhibit significantly higher vulnerability to multi-turn attacks compared to single-turn attacks. We propose DialTree-RPO, an on-policy reinforcement learning framework integrated with tree search that autonomously discovers diverse multi-turn attack strategies by treating the dialogue as a sequential decision-making problem, enabling systematic exploration without manually curated data. Through extensive experiments, our approach not only achieves more than 25.9% higher ASR across 10 target models compared to previous state-of-the-art approaches, but also effectively uncovers new attack strategies by learning optimal dialogue policies that maximize attack success across multiple turns.

基於樹結構的對話強化策略優化用於紅隊攻擊

Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks

摘要

Support