UnpredictaBench：评估大语言模型分布随机性的基准

摘要

我们介绍了UnpredictaBench，这是一项评估大型语言模型（LLMs）捕捉真实潜在分布能力的测试。随着LLMs越来越多地被用作其他实体的替代品（例如，在经济模拟中替代人类），许多模型倾向于坍缩到单一合理答案，导致无法捕捉真实系统的不可预测性。近期在提升输出多样性方面的工作对此场景并不足够：模拟需要的是与目标分布校准的样本，而不仅仅是多样化的输出。UnpredictaBench提炼了该问题的一个简化但基础版本：从个体目标分布中抽取结果，包括经典统计分布、随机程序诱导的分布，以及描述随机过程的自然语言场景。我们引入了448个这样的问题，并配合KS@N这一通用评估指标，该指标通过Kolmogorov-Smirnov统计检验量化模型输出近似黑盒目标分布的程度。这表示在样本量为N时，我们未能拒绝模型样本与真实样本差异的比率，更大的N意味着更高难度。在测试开放与专有模型时，我们发现分布能力存在巨大差异。例如，当模型生成样本量为100时（KS@100，我们的标准指标），得分范围从接近0到超过20%。没有模型能在KS@100上达到40%以上，表明分布采样作为一种能力仍有显著提升空间。尽管增加推理步骤可以略微提升分数，但我们发现没有立即可行的解决方案。UnpredictaBench表明，即使简单的分布模拟仍然具有挑战性，这使其成为将LLMs用作复杂系统替代品的必要第一步。

English

We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems.