基於大型語言模型的音訊描述系統

摘要

音訊描述技術的發展已成為提升影音內容可及性與包容性的關鍵進展。傳統音訊描述製作需耗費大量專業人力，而現有自動化方法仍須經過大量訓練才能整合多模態輸入，並將輸出從字幕風格調整為音訊描述風格。本文提出一種自動化音訊描述生成流程，該流程充分利用GPT-4V(ision)強大的多模態理解與指令跟隨能力。值得注意的是，我們的方法採用現成組件構建，無需額外訓練即可生成既符合自然語言音訊描述製作標準，又能通過基於追蹤的角色識別模組保持跨幀角色語境一致性的音訊描述。在MAD數據集上的全面分析表明，我們的方法在自動音訊描述生產中達到與基於學習的方法相當的性能，CIDEr評分達20.5的實證結果充分佐證了這一點。

English

The development of Audio Description (AD) has been a pivotal step forward in making video content more accessible and inclusive. Traditionally, AD production has demanded a considerable amount of skilled labor, while existing automated approaches still necessitate extensive training to integrate multimodal inputs and tailor the output from a captioning style to an AD style. In this paper, we introduce an automated AD generation pipeline that harnesses the potent multimodal and instruction-following capacities of GPT-4V(ision). Notably, our methodology employs readily available components, eliminating the need for additional training. It produces ADs that not only comply with established natural language AD production standards but also maintain contextually consistent character information across frames, courtesy of a tracking-based character recognition module. A thorough analysis on the MAD dataset reveals that our approach achieves a performance on par with learning-based methods in automated AD production, as substantiated by a CIDEr score of 20.5.

基於大型語言模型的音訊描述系統

LLM-AD: Large Language Model based Audio Description System

摘要

Support