經常使用 ChatGPT 進行寫作任務的人，對於 AI 生成的文字具有準確且強健的檢測能力。

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

January 26, 2025

作者: Jenna Russell, Marzena Karpinska, Mohit Iyyer

cs.AI

摘要

本文研究人類在檢測商用LLM（GPT-4o、Claude、o1）生成文本方面的表現。我們聘請標註者閱讀300篇非虛構英文文章，將它們標記為人類撰寫或AI生成，並提供段落長度的決策解釋。我們的實驗表明，經常使用LLM進行寫作任務的標註者在檢測AI生成文本方面表現出色，即使沒有接受任何專門培訓或反饋。事實上，五位這樣的「專家」標註者中的多數投票僅將300篇文章中的1篇錯誤歸類，明顯優於我們評估的大多數商用和開源檢測器，即使存在重述和人性化等迴避策略。對專家的自由形式解釋進行的定性分析顯示，他們雖然在很大程度上依賴特定詞彙線索（「AI詞彙」），但也能捕捉到文本內部更複雜的現象（例如，正式性、獨創性、清晰度），這對於自動檢測器來說是具有挑戰性的。我們公開了我們的標註數據集和代碼，以促進未來對人類和自動檢測AI生成文本的研究。

English

In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.