ChipNeMo：針對晶片設計進行領域適應的LLM模型

ChipNeMo: Domain-Adapted LLMs for Chip Design

October 31, 2023

作者: Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, Kishor Kunal, Xiaowei Li, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Kaizhe Xu, Haoxing Ren

cs.AI

摘要

ChipNeMo旨在探索大型語言模型（LLMs）在工業芯片設計中的應用。我們不直接部署現成的商用或開源LLMs，而是採用以下領域適應技術：自定義分詞器、領域適應持續預訓練、帶有特定領域指令的監督微調（SFT）和領域適應檢索模型。我們在三個選定的芯片設計LLM應用上評估這些方法：工程助理聊天機器人、EDA腳本生成以及錯誤摘要和分析。我們的結果顯示，這些領域適應技術使LLM在三個評估應用中的性能顯著提升，使得在各種設計任務上可以實現高達5倍的模型尺寸減小，並實現類似或更好的性能。我們的研究結果還表明，我們目前的結果與理想結果之間仍有改進空間。我們相信，進一步研究領域適應的LLM方法將有助於在未來縮小這一差距。

English

ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: custom tokenizers, domain-adaptive continued pretraining, supervised fine-tuning (SFT) with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our results show that these domain adaptation techniques enable significant LLM performance improvements over general-purpose base models across the three evaluated applications, enabling up to 5x model size reduction with similar or better performance on a range of design tasks. Our findings also indicate that there's still room for improvement between our current results and ideal outcomes. We believe that further investigation of domain-adapted LLM approaches will help close this gap in the future.