基于房顶模型的手持设备大语言模型硬件协同设计扩展法则

摘要

视觉-语言-动作模型（VLA）已成为物理人工智能的关键范式，并日益广泛应用于自动驾驶车辆、机器人和智能空间。在这些资源受限的终端设备场景中，选择合适的大型语言模型（LLM）骨干网络至关重要：模型必须在精度与严格的推理延迟及硬件效率约束之间取得平衡。这使得软硬件协同设计成为终端LLM部署中具有颠覆性意义的要求——每个硬件平台都需要量身定制的架构解决方案。我们提出了一种同时涵盖模型精度与推理性能的硬件协同设计法则，具体通过将训练损失建模为架构超参数的显式函数，并采用屋顶线模型表征推理延迟。我们在NVIDIA Jetson Orin平台上实证评估了1,942种候选架构，并对筛选出的170个模型分别进行100亿标记训练，以拟合架构与训练损失之间的缩放定律。通过将该缩放定律与延迟建模相结合，我们建立了精度-延迟的直接对应关系，并确定了硬件协同设计LLM的帕累托前沿。我们进一步将架构搜索形式化为精度与性能的联合优化问题，推导出工业级硬件和应用预算下的可行设计区域。该方法将架构选择周期从数月缩短至数天：在目标硬件上与Qwen2.5-0.5B保持相同延迟时，我们协同设计的架构在WikiText-2数据集上实现了困惑度降低19.42%。据我们所知，这是首个面向终端LLM部署的体系化硬件协同设计缩放定律框架。我们将公开相关代码与模型检查点。

English

Vision-Language-Action Models (VLAs) have emerged as a key paradigm of Physical AI and are increasingly deployed in autonomous vehicles, robots, and smart spaces. In these resource-constrained on-device settings, selecting an appropriate large language model (LLM) backbone is a critical challenge: models must balance accuracy with strict inference latency and hardware efficiency constraints. This makes hardware-software co-design a game-changing requirement for on-device LLM deployment, where each hardware platform demands a tailored architectural solution. We propose a hardware co-design law that jointly captures model accuracy and inference performance. Specifically, we model training loss as an explicit function of architectural hyperparameters and characterise inference latency via roofline modelling. We empirically evaluate 1,942 candidate architectures on NVIDIA Jetson Orin, training 170 selected models for 10B tokens each to fit a scaling law relating architecture to training loss. By coupling this scaling law with latency modelling, we establish a direct accuracy-latency correspondence and identify the Pareto frontier for hardware co-designed LLMs. We further formulate architecture search as a joint optimisation over precision and performance, deriving feasible design regions under industrial hardware and application budgets. Our approach reduces architecture selection from months to days. At the same latency as Qwen2.5-0.5B on the target hardware, our co-designed architecture achieves 19.42% lower perplexity on WikiText-2. To our knowledge, this is the first principled and operational framework for hardware co-design scaling laws in on-device LLM deployment. We will make the code and related checkpoints publicly available.

基于房顶模型的手持设备大语言模型硬件协同设计扩展法则

Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs

摘要

Support