還有什麼需要修復的？探討對話生成產出物中修訂傳播的經濟效益測試時計算

摘要

大型語言模型（LLMs）常在對話中透過生成與修訂的反覆循環，協助使用者產生產物（artifact）。此處的挑戰在於，當使用者在修訂時僅指定一個局部變更，LLM 必須識別相關的依賴關係，並將該變更傳播到產物中所有受影響的部分。本文研究 LLM 在對話式生成產物上的此能力；在此情境中，產物上下文及其依賴關係可能嵌入於對話歷史中。為了貼近實際應用，我們也探索在此新情境中具成本效益的測試時計算。具體而言，我們為此情境引入一個新基準，並使用 gpt-oss-20b/120b、gpt-5.4-mini 與 qwen3.5-9b/27b/122b 在此基準上評估九種修訂方法，包括順序反思與並行取樣變體。結果顯示，基線方法的準確率介於 68.3% 至 93%；最具成本效益的方法是使用基於 LLM 的選取或 medoid（中心點）選取，從三個並行樣本中選擇一個，能將準確率提升 2.2% 至 9.7%。我們的程式碼與資料集位於 https://github.com/ntt-dkiku/llm-revision-propagation。

English

Large Language Models (LLMs) often help users generate artifacts through iterative cycles of generation and revision in conversation. A challenge here is that, when users specify only a local change during revision, LLMs must instead identify the relevant dependencies and propagate the revision to all affected parts of the artifact. This paper studies this ability of LLMs on conversationally generated artifacts, where the artifact context and its dependencies may be embedded in the conversation history. Toward practical use, we also explore cost-effective test-time compute for this new setting. Specifically, we introduce a new benchmark for this setting, and evaluate nine revision methods, including sequential reflection and parallel sampling variants, using gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b on the benchmark. The results show that baselines achieve accuracies of 68.3--93%, and the most cost-effective method is selecting from three parallel samples using either LLM-based or medoid selection, which improves accuracy by 2.2--9.7%. Our code and dataset are available at https://github.com/ntt-dkiku/llm-revision-propagation.