文學文本的AI翻譯「還算不錯」，但讀者仍偏愛人類翻譯。

摘要

文學作品的人工智慧翻譯日益普遍。雖然內容可能翻譯得尚可，但我們對於讀者在沉浸感與文學效果方面的體驗了解不足，而這些面向是自動機器翻譯指標或針對流暢度與充分性的人類評估難以捕捉到的。我們邀請15位熱愛閱讀的讀者，針對近期出版的15本法語、波蘭語及日語小說（譯入英文），比較人類翻譯（HT）與基於代理式大型語言模型（LLM）流程產生的機器翻譯（MT）。讀者在兩種條件下評估約8,000字的摘錄：沉浸式閱讀完整摘錄（30組比較），以及仔細閱讀386組配對的HT-MT片段（772組比較），每本書由兩位讀者以交替呈現順序進行。整體而言，讀者認為MT「還可以」，但偏好HT（在摘錄層級上略為偏好，19/30；在片段層級上更為明顯，522/772），因其更易讀、清晰且具有沉浸感。讀者的標註顯示，MT在同一本書內的品質變異性高於HT。關鍵的是，讀者無法可靠地區分兩者（30次中僅17次猜對），且傾向偏好他們認為是人類翻譯的版本。自動化指標（包括將LLM作為評審的方法）未能反映讀者偏好，且偏向MT。我們發布LAIT（文學人工智慧翻譯），這是一個以讀者為中心的評估資料集，包含1,000條讀者評論、2,000項判斷與偏好評分、以及7,200個跨度層級的標註，同時提供我們的評估協議與支援介面。

English

AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experience it in terms of immersiveness and literary effect, aspects poorly captured by automatic machine translation metrics or human evaluation targeting fluency and adequacy. We ask 15 avid readers to compare recently published human translations (HT) to machine translations (MT) generated with an agentic large language model (LLM)-based pipeline, for 15 recent novels in French, Polish, and Japanese and translated into English. Readers evaluated approximately 8K-word excerpts in two conditions: immersive reading of the whole excerpt (30 comparisons) and close reading of 386 aligned HT-MT chunk pairs (772 comparisons), with two readers per book and in alternating order of presentation. Overall, readers find MT "fine", but prefer HT (slightly at excerpt-level 19/30, more clearly at chunk-level 522/772) for its ease, clarity, and immersive nature. Readers' highlights show that MT's quality varies more within one book than HT's does. Crucially, readers cannot reliably tell the two apart (17/30 guess correctly) and tend to prefer the version they believe to be human. Automatic metrics, including LLM-as-a-judge approaches, fail to recover reader preferences and favor MT. We release LAIT (Literary AI Translation), a reader-centered evaluation dataset with 1K reader comments, 2K judgments and preference ratings, and 7.2K span-level annotations, along with our evaluation protocol and supporting interface.