ChipNeMo：用于芯片设计的领域自适应LLMs
ChipNeMo: Domain-Adapted LLMs for Chip Design
October 31, 2023
作者: Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, Kishor Kunal, Xiaowei Li, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Kaizhe Xu, Haoxing Ren
cs.AI
摘要
ChipNeMo旨在探索大型语言模型（LLMs）在工业芯片设计中的应用。我们不直接使用现成的商业或开源LLMs，而是采用以下领域自适应技术：定制分词器、领域自适应持续预训练、带有领域特定指令的监督微调（SFT）和领域自适应检索模型。我们在芯片设计的三个选定的LLM应用上评估了这些方法：工程助手聊天机器人、EDA脚本生成以及错误摘要和分析。我们的结果显示，这些领域自适应技术能够显著提高LLM在这三个评估应用中的性能，使得在各种设计任务上，模型尺寸最多能减小5倍，而性能相似或更好。我们的发现还表明，我们当前结果与理想结果之间仍有改进空间。我们相信，进一步研究领域自适应LLM方法将有助于未来缩小这一差距。
English
ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.