RDP LoRA：面向大语言模型参数高效自适应几何驱动识别法

摘要

尽管存在低秩自适应(LoRA)等参数高效方法，大型语言模型(LLM)的微调机制仍存在结构性认知空白。由于对内部表征的层级特异性作用理解不足，当前适配层选择多依赖启发式策略。我们将隐藏状态的演化建模为高维几何轨迹，并提出采用拉默-道格拉斯-普克算法——一种无需参数且无需训练的多边形简化方法，在保留全局结构跃迁的同时剔除局部冗余变化，以此识别表征路径上的关键转折点。创新性地，我们不仅将这些几何枢轴点用于分析，更将其作为直接决策信号来确定参数高效微调中需要适配的层级。通过将这种几何感知的层级选择策略集成至Qwen3-8B-Base模型的LoRA微调，在仅适配13个RDP选定层的情况下，于MMLU-Math基准测试取得81.67%的优异表现，显著优于全36层适配(79.32%)、随机13层选择(75.56%)及基线模型(74.25%)。这些结果表明，利用表征轨迹的固有几何特性可为模型适配中的层级优化提供鲁棒、可解释且无需训练的信号指导。

English

Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.

RDP LoRA：面向大语言模型参数高效自适应几何驱动识别法

RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

摘要

Support