DeepSeek-Coder-V2：突破代码智能闭源模型的障碍

DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

June 17, 2024

作者: DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, Wenfeng Liang

cs.AI

摘要

我们介绍了DeepSeek-Coder-V2，这是一个开源的专家混合（MoE）代码语言模型，在代码特定任务中实现了与GPT4-Turbo可比的性能。具体来说，DeepSeek-Coder-V2是在DeepSeek-V2的中间检查点进一步预训练的，额外增加了6万亿个标记。通过这种持续的预训练，DeepSeek-Coder-V2显著增强了DeepSeek-V2的编码和数学推理能力，同时在一般语言任务中保持了可比性能。与DeepSeek-Coder-33B相比，DeepSeek-Coder-V2在各个与代码相关的任务以及推理和一般能力方面都取得了显著进展。此外，DeepSeek-Coder-V2将其对编程语言的支持从86种扩展到338种，同时将上下文长度从16K扩展到128K。在标准基准评估中，DeepSeek-Coder-V2在编码和数学基准测试中表现优于GPT4-Turbo、Claude 3 Opus和Gemini 1.5 Pro等闭源模型。

English

We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-V2, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.