自迴歸馬賽克：探測純文字語言模型中的二維空間推理

摘要

僅以文字與程式碼訓練的大型語言模型（LLM），有時能產生可繪出可辨識圖像的程式。然而，目前仍不清楚這究竟反映模型內部具備二維空間佈局的表徵，抑或僅是將空間描述轉譯為程式碼的能力。我們提出「自迴歸馬賽克」（Autoregressive Mosaics, AM-Bench）基準測試，用以區分上述因素：首先，翻譯任務要求模型根據以文字完整指定圖片幾何結構的提示，生成對應的繪圖程式碼；其次，佈局任務要求模型根據未完整指定的提示自行構圖。在八個僅以文字與程式碼訓練的開放權重模型中，所有模型都能可靠地將指定幾何結構翻譯成程式碼，但它們在開放式佈局任務上的表現差異甚大，顯示這些差異無法僅以程式碼生成能力解釋。輸出媒介的消融實驗進一步顯示，模型所使用的表達介面或媒介確實會造成影響：以原始SVG取代程序式程式碼，一致地提升了所有模型的佈局分數。最後，對模型激活狀態的探測顯示，在生成展開之前，模型中已存在粗略的佈局計畫，但此計畫僅反映提示所隱含的佈局；而在生成期間，模型追蹤的是不斷演變的幾何狀態，而非執行一開始便固定的計畫。整體而言，這些結果表明，純文字LLM的二維空間表現同時取決於模型本身與輸出媒介，並非僅由程式碼生成能力所能解釋。

English

Large language models (LLMs) trained only on text and code can sometimes generate programs that draw recognizable images. However, it is unclear whether this reflects an internal representation of 2D spatial layout or simply the ability to translate spatial descriptions into code. We introduce Autoregressive Mosaics (AM-Bench), a benchmark that separates these factors: First, a translation task gives a model a fully specified geometry of a picture in words as a prompt and asks for the code that produces it. Second, a layout task requires the model to compose an image from an underspecified prompt. Across eight open-weight text-and-code-only models, all models reliably translate specified geometry into code, but their open-ended layout performance differs substantially, indicating that these differences are not explained by code-generation ability alone. An output-medium ablation further shows that the interface or medium of expression that the model uses matters: replacing procedural code with raw SVG improves layout scores across all models. Finally, probing model activations shows that a coarse layout plan is present before generation, but reflects only the layout implied by the prompt. During generation, models track the evolving geometric state instead of executing an initially fixed plan. Overall, these results show that 2D spatial performance in text-only LLMs depends on both the model and the output medium, and is not explained by code-generation ability alone.