PrivacyAlign：大型語言模型代理的上下文隱私對齊

摘要

代表使用者行動的AI代理不斷做出決策，而為了讓使用者信任他們的代理，這些決策必須與他們實際的意願一致。對代理而言，隱私是一個重要的對齊問題：代理所發送的每則訊息、貼文或工具呼叫，都是一種關於分享內容、對象及條件是否適當的情境判斷。由於此類判斷取決於社會期望與規範，人類判斷不僅標示隱私侵犯，更有助於定義它們。現有研究在訓練與評估上依賴不可靠的代理指標，而我們則將人類判斷置於代理隱私對齊的核心。我們提出PrivacyAlign，一個包含1,350個樣本、來自599位不同註釋者在當前大型語言模型實際洩漏隱私的多元情境中所提供的3,516條詳細註釋的資料集，並以此將對齊訓練與自動化評估奠基於人類隱私規範之上。基於這些註釋，我們首先展示，讓大型語言模型裁判根據同一提示的參考回應之人類註釋與解釋進行條件化，能使其判斷更為可靠。接著我們引入註釋條件化獎勵建模，在強化學習期間利用這些註釋對新回應進行評分，並展示經此獎勵訓練的小型開源權重代理能更好地與人類隱私規範對齊，在PrivacyAlign及現有代理隱私基準測試上均有顯著提升。

English

AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.