V2V-LLM：利用多模式大型語言模型的車輛對車輛協作自主駕駛

V2V-LLM: Vehicle-to-Vehicle Cooperative Autonomous Driving with Multi-Modal Large Language Models

February 14, 2025

作者: Hsu-kuang Chiu, Ryo Hachiuma, Chien-Yi Wang, Stephen F. Smith, Yu-Chiang Frank Wang, Min-Hung Chen

cs.AI

摘要

目前的自動駕駛車輛主要依賴其個別感應器來理解周圍場景並規劃未來軌跡，然而當感應器發生故障或被遮擋時，這種方法可能變得不可靠。為解決這個問題，提出了透過車輛間通信的合作感知方法，但這些方法往往專注於檢測和追蹤。這些方法如何有助於整體合作規劃表現仍未深入探討。受最近使用大型語言模型（LLMs）構建自動駕駛系統的進展啟發，我們提出了一個將LLM整合到合作自動駕駛中的新問題設定，並提出了車輛間問答（V2V-QA）數據集和基準。我們還提出了我們的基準方法車輛間大型語言模型（V2V-LLM），該方法使用LLM將來自多個連接的自動駕駛車輛（CAVs）的感知信息融合，並回答與駕駛相關的問題：定位、顯著物體識別和規劃。實驗結果顯示，我們提出的V2V-LLM可以成為執行合作自動駕駛中各種任務的有前途的統一模型架構，並優於使用不同融合方法的其他基準方法。我們的工作還開創了一個新的研究方向，可以提高未來自動駕駛系統的安全性。我們的項目網站：https://eddyhkchiu.github.io/v2vllm.github.io/。

English

Current autonomous driving vehicles rely mainly on their individual sensors to understand surrounding scenes and plan for future trajectories, which can be unreliable when the sensors are malfunctioning or occluded. To address this problem, cooperative perception methods via vehicle-to-vehicle (V2V) communication have been proposed, but they have tended to focus on detection and tracking. How those approaches contribute to overall cooperative planning performance is still under-explored. Inspired by recent progress using Large Language Models (LLMs) to build autonomous driving systems, we propose a novel problem setting that integrates an LLM into cooperative autonomous driving, with the proposed Vehicle-to-Vehicle Question-Answering (V2V-QA) dataset and benchmark. We also propose our baseline method Vehicle-to-Vehicle Large Language Model (V2V-LLM), which uses an LLM to fuse perception information from multiple connected autonomous vehicles (CAVs) and answer driving-related questions: grounding, notable object identification, and planning. Experimental results show that our proposed V2V-LLM can be a promising unified model architecture for performing various tasks in cooperative autonomous driving, and outperforms other baseline methods that use different fusion approaches. Our work also creates a new research direction that can improve the safety of future autonomous driving systems. Our project website: https://eddyhkchiu.github.io/v2vllm.github.io/ .