從文本生成物理穩定且可建造的樂高設計

摘要

我們推出了LegoGPT，這是首個從文本提示生成物理穩定的樂高積木模型的方法。為實現這一目標，我們構建了一個大規模、物理穩定的樂高設計數據集，並配以相關描述，訓練了一個自迴歸大型語言模型，通過下一個令牌預測來預測應添加的下一個積木。為了提高生成設計的穩定性，我們在自迴歸推理過程中採用了高效的有效性檢查和物理感知回滾機制，利用物理定律和組裝約束來修剪不可行的令牌預測。實驗結果表明，LegoGPT能夠生成穩定、多樣且美觀的樂高設計，這些設計與輸入的文本提示緊密契合。此外，我們還開發了一種基於文本的樂高紋理生成方法，用於創建彩色和帶有紋理的設計。我們展示了這些設計不僅可由人工手動組裝，也能通過機械臂自動完成。同時，我們在項目網站https://avalovelace1.github.io/LegoGPT/上發布了包含超過47,000個樂高結構、涵蓋28,000多個獨特3D物體並附有詳細描述的新數據集StableText2Lego，以及我們的代碼和模型。

English

We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: https://avalovelace1.github.io/LegoGPT/.

從文本生成物理穩定且可建造的樂高設計

Generating Physically Stable and Buildable LEGO Designs from Text

摘要

Support