此引用是否切中要點？

摘要

2023年，一名紐約法官在Mata v. Avianca一案中，對兩名律師提交含有ChatGPT所生成虛構引用的書狀而施以制裁。此類失誤大多可透過資料庫查詢予以察覺；較棘手的問題在於偵測那些指向真實案件、但卻無法支持其所援引命題的引用——此種失效模式在現有針對LLM法律應用之評估中大多遭到忽略。本文透過對兩個法律語料庫中的真實法律引用進行受控擾動，來研究命題層級的引用支持驗證，做法是替換被引用的案件，或僅在同一案件內變更精確頁碼。我們在所得範例上評估十四種模型配置。模型能捕捉93-100%的錯誤案件竄改。然而，對於法院判決意見中的錯誤精確頁碼竄改，模型僅能捕捉37-61%；對法律狀紙則能捕捉52-83%。當模型未能捕捉錯誤精確頁碼竄改時，它們是基於主題重疊而非頁碼層級的支持來接受該引用。規模與擴展推理縮小了差距，但未能消除差距：GPT-5.4在高推理強度下，仍遺漏法院判決意見中40%的精確頁碼不符，以及法律狀紙中18%的不符。提示模型在引用的頁碼處驗證支持內容可提升召回率，但也提高了誤報率。辨識正確的法律主題與驗證所援引命題的支持內容，是兩種截然不同的能力，而當前模型將兩者混為一談。

English

In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.