RDP LoRA：面向大语言模型参数高效适配的几何驱动识别方法

摘要

尽管存在低秩自适应（LoRA）等参数高效方法，大型语言模型（LLM）的微调机制仍存在结构不确定性。由于对内部表征的层级特异性作用认知不足，当前层适配决策多依赖启发式策略。我们将隐藏状态的演化建模为高维几何轨迹，并提出采用拉默-道格拉斯-普克（RDP）算法——一种无需参数且无需训练的多边形简化方法，在保留全局结构跃迁的同时剔除局部冗余变化，以此识别表征路径上的关键转折点。关键创新在于，这些几何枢轴不仅用于分析，更直接作为参数高效微调过程中确定适配层级的决策信号。通过将这种几何感知的层级选择策略集成至Qwen3-8B-Base模型的LoRA微调框架，仅使用RDP算法选定的13个层级即在MMLU-Math数据集上取得81.67%的优异表现，显著优于全36层适配（79.32%）、随机13层选择（75.56%）以及基线模型（74.25%）。结果表明，利用表征轨迹的内在几何特性可为模型适配过程中的层级选择提供鲁棒、可解释且无需训练的优化信号。

English

Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.

RDP LoRA：面向大语言模型参数高效适配的几何驱动识别方法

RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

摘要

Support