从文本生成物理稳定且可搭建的乐高设计

Generating Physically Stable and Buildable LEGO Designs from Text

May 8, 2025

作者: Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, Jun-Yan Zhu

cs.AI

摘要

我们推出了LegoGPT，这是首个能够根据文本提示生成物理稳定的乐高积木模型的方法。为实现这一目标，我们构建了一个大规模、物理稳定的乐高设计数据集，并配以相应的描述文本，进而训练了一个自回归大型语言模型，通过下一令牌预测来推断应添加的下一个积木。为了提升生成设计的稳定性，我们在自回归推理过程中采用了高效的合法性检查与物理感知回滚机制，利用物理定律和组装约束条件剔除不可行的令牌预测。实验表明，LegoGPT能够生成稳定、多样且美观的乐高设计，这些设计与输入文本提示高度契合。此外，我们还开发了一种基于文本的乐高纹理生成方法，用于创建带有颜色和纹理的设计。我们展示了这些设计不仅可由人工手动组装，还能通过机械臂自动完成。同时，我们公开了新的数据集StableText2Lego，其中包含超过47,000个乐高结构，对应28,000多个独特的3D物体，并附有详细描述，以及我们的代码和模型，项目网站为：https://avalovelace1.github.io/LegoGPT/。

English

We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: https://avalovelace1.github.io/LegoGPT/.