自回归马赛克：探测纯文本语言模型中的二维空间推理能力

摘要

仅在文本和代码上训练的大型语言模型（LLMs）有时能生成绘制可识别图像的程序。然而，尚不清楚这是否反映了模型对二维空间布局的内部表征，还是仅仅意味着其具备将空间描述转化为代码的能力。我们引入了自回归马赛克（AM-Bench）基准，该基准将这些因素区分开来：首先，翻译任务以文字形式向模型提供一幅图像的完整几何描述作为提示，并要求其生成能产生该图像的代码；其次，布局任务要求模型根据一个不完整的提示来组合图像。在八个开放权重的仅文本和代码模型中，所有模型都能可靠地将指定几何描述转换为代码，但它们在开放式布局任务上的表现差异显著，这表明这些差异并不能仅用代码生成能力来解释。输出媒介消融进一步表明，模型所使用的表达接口或媒介至关重要：将程序化代码替换为原始SVG可提高所有模型的布局得分。最后，对模型激活的探测表明，模型在生成之前就已存在一个粗略的布局计划，但该计划仅反映提示词所隐含的布局。在生成过程中，模型跟踪不断演变的几何状态，而非执行一个最初固定的计划。总体而言，这些结果表明，纯文本LLMs中的二维空间性能同时取决于模型本身和输出媒介，且不能仅用代码生成能力来解释。

English

Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.