微調逆轉的引力詮釋

摘要

針對無害資料的微調，可能部分逆轉訓練早期所習得的行為。安全性可能在良性後對齊更新後受到侵蝕，已遺忘的能力可能重新浮現，潛在特徵可能透過看似無關的監督進行轉移，而相關的後對齊脆弱性也出現在其他生成式設定中。我們認為這些現象透過共同的訓練歷史視角來審視，具有分析價值。我們的假設是幾何性的：早期的大規模訓練階段形成主導行為流形，而後續的對齊或專業化階段則是相對於這些流形的淺層位移。因此，後續的微調會繼承一個持續存在的回歸分量，指向該主導流形的見證者。我們稱之為微調回歸的重力解釋。在我們的主要設定中，表徵漂移迅速獲得一個沿著歷史定義回歸方向（v_rev）的分量。在主實驗序列中，與v_rev的對齊從第一次更新後的 cos = 0.429 ± 0.052 上升至步驟20時的 0.647 ± 0.021。在24個運行步驟配對中，每個觀測到的對齊值均超過等向性激活空間虛無假設的第99百分位數。我們證明，選擇性地阻斷沿v_rev的運動，可將T=100時的最終對齊從0.648 ± 0.009改變為-0.211 ± 0.021，並將危害性從19.0% ± 4.0%降低至8.5% ± 1.5%，且幾乎不影響任務表現。這些結果支持v_rev在我們設定中扮演後對齊早期回歸的因果相關中介者。重要的是，我們並不主張v_rev是唯一的安全方向，也不主張主導流形能被直接觀測；相反，我們識別出一個穩健的、由歷史定義的方向，它能解釋並部分控制早期的回歸動態。

English

Fine-tuning on harmless data can partially undo behaviors acquired earlier in training. Safety can erode under benign post-alignment updates, unlearned capabilities can re-emerge, latent traits can transfer through apparently unrelated supervision, and related post-alignment fragility appears in other generative settings. We argue these phenomena are usefully viewed through a common training-history lens. Our hypothesis is geometric: large early training phases create dominant behavioral manifolds, while later alignment or specialization phases are shallower displacements from them. Subsequent fine-tuning can therefore inherit a persistent reversion component pointing back toward a witness of the dominant manifold. We call this the gravitational interpretation of fine-tuning reversion. Across our main settings, representational drift rapidly acquires a component along a history-defined reversion direction (v_rev). In our main track, alignment with v_rev rises from cos = 0.429 +/- 0.052 after the first update to 0.647 +/- 0.021 by step 20. Across 24 run-step pairs, every observed alignment exceeds the p99 of an isotropic activation-space null. We demonstrate that selectively blocking motion along v_rev changes the final alignment at T=100 from 0.648 +/- 0.009 to -0.211 +/- 0.021 and reduces harmfulness from 19.0% +/- 4.0% to 8.5% +/- 1.5% with little task cost. These results support v_rev as a causally relevant mediator of early post-alignment reversion in our setup. Importantly, we do not claim that v_rev is the unique safety direction, nor that the dominant manifold is directly observed; rather, we identify a robust, history-defined direction that explains and partially controls early reversion dynamics.