微調語言模型以文本形式生成穩定的無機材料

摘要

我們提出對大型語言模型進行微調，以生成穩定材料。儘管這種做法有些不尋常，但在將大型語言模型微調為基於文本編碼的原子數據時，實施簡單且可靠，約有90%的樣本結構遵守原子位置和電荷的物理約束。通過從學習的機器學習潛力和黃金標準密度泛函理論計算的能量超出凸起計算，我們展示了我們最強的模型（微調的LLaMA-2 70B）可以生成材料，預測為亞穩定的概率約為CDVAE（一種競爭擴散模型）的兩倍（49% vs 28%）。由於文本提示的固有靈活性，我們的模型可以同時用於無條件生成穩定材料、填充部分結構以及文本條件生成。最後，我們展示了語言模型捕捉晶體結構的關鍵對稱性能力隨著模型規模的增加而提高，這表明預訓練的大型語言模型的偏見出奇地適合原子數據。

English

We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

微調語言模型以文本形式生成穩定的無機材料

Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

摘要

Support