精细调节逆转的引力解释

摘要

在无害数据上的微调可能部分逆转训练早期习得的行为。良性后对齐更新可能导致安全性退化，已遗忘的能力会重新浮现，潜在特性会通过看似无关的监督信号发生迁移，类似的后对齐脆弱性也出现在其他生成场景中。我们认为这些现象可通过共同的训练历史视角加以理解。我们的假设源于几何学：早期大规模训练阶段会形成主导行为流形，而后期对齐或专业化阶段仅是对这些流形的浅层位移。因此，后续微调会继承一个指向主导流形回波方向的持久回归成分——我们称之为微调回归的引力解释。在我们的主要实验设置中，表征漂移会迅速获得沿历史定义回归方向（v_rev）的分量。在主干实验中，对齐度从首次更新后的余弦值0.429±0.052升至第20步时的0.647±0.021。在24个运行-步数对中，所有观测到的对齐度均超过各向同性激活空间零假设的第99百分位数。我们证明，选择性阻断沿v_rev的运动可将T=100时的最终对齐度从0.648±0.009降至-0.211±0.021，并将有害性从19.0%±4.0%降至8.5%±1.5%，且任务代价极小。这些结果支持v_rev作为本实验中早期后对齐回归的因果相关中介。需要强调的是，我们并不主张v_rev是唯一的安全方向，也不认为能直接观测到主导流形；而是识别出一个鲁棒的、由历史定义的方向，该方向既能解释又能部分控制早期回归动态。

English

Fine-tuning on harmless data can partially undo behaviors acquired earlier in training. Safety can erode under benign post-alignment updates, unlearned capabilities can re-emerge, latent traits can transfer through apparently unrelated supervision, and related post-alignment fragility appears in other generative settings. We argue these phenomena are usefully viewed through a common training-history lens. Our hypothesis is geometric: large early training phases create dominant behavioral manifolds, while later alignment or specialization phases are shallower displacements from them. Subsequent fine-tuning can therefore inherit a persistent reversion component pointing back toward a witness of the dominant manifold. We call this the gravitational interpretation of fine-tuning reversion. Across our main settings, representational drift rapidly acquires a component along a history-defined reversion direction (v_rev). In our main track, alignment with v_rev rises from cos = 0.429 +/- 0.052 after the first update to 0.647 +/- 0.021 by step 20. Across 24 run-step pairs, every observed alignment exceeds the p99 of an isotropic activation-space null. We demonstrate that selectively blocking motion along v_rev changes the final alignment at T=100 from 0.648 +/- 0.009 to -0.211 +/- 0.021 and reduces harmfulness from 19.0% +/- 4.0% to 8.5% +/- 1.5% with little task cost. These results support v_rev as a causally relevant mediator of early post-alignment reversion in our setup. Importantly, we do not claim that v_rev is the unique safety direction, nor that the dominant manifold is directly observed; rather, we identify a robust, history-defined direction that explains and partially controls early reversion dynamics.