AVA-編碼器：邁向智能體原生的視頻表徵學習

摘要

創作型智能代理仍缺乏從高品質人類電影中學習的有效途徑，限制了其生成電影級影片的能力。一個關鍵挑戰在於缺乏一種結構化的視頻表示，既能忠實呈現電影內容，又能直接用於代理推理與操作。為應對此挑戰，我們提出了代理式視頻自編碼器（AVA-Encoder），這是一個通過代理式自編碼來學習代理原生視頻表示的框架。 AVA-Encoder 將視頻轉化為知識圖譜（KG）表示，再將其重建為視頻。其層次結構與狀態節點存儲結構化文本，而連結的資源層則保存生成的圖像、音頻和視頻。類型化邊以代理易於理解、查詢和編輯的形式，保留這些文本描述與資源之間的關係。視頻重建的差異驅動文本梯度優化框架，將評估反饋轉化為自然語言更新方向，用於外循環中的數據無關編碼策略偽訓練，以及測試時內循環中可選的數據依賴知識圖譜表示精煉。大量實驗表明，AVA-Encoder 相比最強的外部基線提升了20.7個百分點。在僅策略的受控設置下，其偽訓練的鏡頭級代理視頻編碼策略也優於精心人工調校的策略，同時減少了74.3%的系統提示令牌。我們發布了完整的 AVA-Encoder 框架、一個可靠的代理式視頻重建基準測試，以及首個高品質電影知識圖譜表示數據集。

English

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agentic auto-encoding. AVA-Encoder transforms a video into a knowledge graph (KG) representation and then reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between these text descriptions and assets in a form that agents can easily understand, query, and edit. The video reconstruction differences drive a textual-gradient optimization framework, which expresses evaluation feedback as natural-language update directions for Data-Independent Encoding Policy Pseudo-Training in the outer loop and optional Data-Dependent KG Representation Refinement in the test-time inner loop. Extensive experiments show that AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.