批次大小越大是否總是更好？-- 關於使用 Llama-3 進行模型編輯的實證研究

Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3

May 1, 2024

作者: Junsang Yoon, Akshat Gupta, Gopala Anumanchipalli

cs.AI

摘要

本研究提出了一項針對最新大型語言模型 Llama-3 的目標模型編輯分析。我們探討了針對精確層介入而設計的流行模型編輯技術 - ROME、MEMIT 和 EMMET 的效力。透過評估，跨越三種不同策略：順序編輯、批次編輯和我們稱之為順序批次編輯的混合方法，我們確定了最有效的層進行目標編輯。我們的研究結果顯示，增加編輯批次大小可能會比依序使用較小的編輯批次對相同數量的編輯更明顯地降低模型性能。基於此，我們認為順序模型編輯是擴展模型編輯方法的重要組成部分，未來的研究應該專注於結合批次和順序編輯的方法。這一觀察表明了當前模型編輯方法可能存在的一個潛在限制，即傾向於使用更大的編輯批次大小，我們希望這將為未來對優化批次大小和模型編輯性能進行研究鋪平道路。

English

This study presents a targeted model editing analysis focused on the latest large language model, Llama-3. We explore the efficacy of popular model editing techniques - ROME, MEMIT, and EMMET, which are designed for precise layer interventions. We identify the most effective layers for targeted edits through an evaluation that encompasses up to 4096 edits across three distinct strategies: sequential editing, batch editing, and a hybrid approach we call as sequential-batch editing. Our findings indicate that increasing edit batch-sizes may degrade model performance more significantly than using smaller edit batches sequentially for equal number of edits. With this, we argue that sequential model editing is an important component for scaling model editing methods and future research should focus on methods that combine both batched and sequential editing. This observation suggests a potential limitation in current model editing methods which push towards bigger edit batch sizes, and we hope it paves way for future investigations into optimizing batch sizes and model editing performance.