CAVEWOMAN：大型語言模型在語言輸入與輸出壓縮下的行為表現

摘要

「少說話、省文法、省代幣。」這種原始人風格被廣泛推崇為降低推理成本的方式，但實際能否節省成本，取決於被壓縮的是哪個通道（使用者的提示詞或模型生成的回應）。我們提出 Cavewoman，一種雙通道評估協議，針對每一次生成，從任務準確率、實際每項目的成本，以及與模型無約束參考文本的一致性進行評分。我們在五個資料集、五個壓縮等級上評估了八個模型，且兩個通道均針對相同項目進行測量。輸出壓縮在多數 API 模型上降低了實際成本（每模型 1.4 至 2.4 倍，最佳情況可達 3 倍），並且在公有價格定價下，對所有四個開放權重模型均有效。輸入壓縮則產生相反效果，是一種嚴格的雙輸局面：它反而提高淨成本而非降低（五個基準平均約 1.15 倍，最差資料集可達 1.8 倍，更強壓縮下達 2.7 倍），原因在於模型會以更長的回應來補償，即使準確率已大幅下降。在相同設定下，表面文本與無約束參考產生分歧：在非推理模型中，約有一半的生成結果雖然正確，但其表面文本不再包含模型自身無約束基線生成的內容。此分歧在經過長度控制重新評分、多重比較校正，以及以互補語義指標進行的複現檢驗後依然存在。程式碼與資料可見於 https://github.com/danielle34/cavewoman。

English

"Talk short. Drop grammar. Save token." This caveman style is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed. We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement against the model's unconstrained reference. We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items. Output compression cuts realized cost on most API models (1.4-2.4x per model, up to 3x in the best case) and on all four open-weight models under public-tier pricing. Input compression has the opposite effect, a strict lose-lose: it raises net cost rather than lowering it (~1.15x on the five-benchmark mean, up to 1.8x on the worst dataset and 2.7x under stronger compression), because models compensate with longer responses even as accuracy collapses. Under the same setting, surface text diverges from the unconstrained reference: on the non-reasoning models, roughly half of all generations are correct yet their surface text no longer entails the model's own unconstrained baseline generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under complementary semantic measures. Code and data are available at https://github.com/danielle34/cavewoman.