SaulLM-54B和SaulLM-141B：扩展法律领域的领域自适应

摘要

本文介绍了SaulLM-54B和SaulLM-141B，这是专为法律领域定制的两个大型语言模型（LLMs）。这两个模型分别具有540亿和1410亿参数的架构，基于Mixtral架构。SaulLM-54B和SaulLM-141B的开发受大规模领域自适应的指导，分为三个策略：（1）利用持续预训练，包括超过5400亿法律标记的基础语料库，（2）实施专门的法律指令遵循协议，以及（3）将模型输出与法律解释中的人类偏好进行对齐。在第二和第三步中整合合成生成的数据增强了模型在解释和处理法律文本方面的能力，有效地实现了最先进的性能，并在LegalBench-Instruct上胜过了先前的开源模型。本研究探讨了在这一规模上涉及的领域特定自适应的权衡，提供了可能对使用强解码器模型进行领域自适应的未来研究提供见解。在SaulLM-7B的基础上，本研究完善了这一方法，以生成一个更适合法律任务的LLM。我们在MIT许可下发布了SaulLM-54B和SaulLM-141B的基础、指令和对齐版本，以促进重复使用和协作研究。

English

In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language models (LLMs) tailored for the legal sector. These models, which feature architectures of 54 billion and 141 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is guided by large-scale domain adaptation, divided into three strategies: (1) the exploitation of continued pretraining involving a base corpus that includes over 540 billion of legal tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming previous open-source models on LegalBench-Instruct. This work explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks. We are releasing base, instruct, and aligned versions on top of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and collaborative research.

SaulLM-54B和SaulLM-141B：扩展法律领域的领域自适应

SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

摘要

Summary

Support

Support