Robusto-2：在利马與紐約市對人類與視覺語言模型進行自動駕駛基準測試

摘要

隨著自動駕駛汽車持續在全球擴展，並採用如視覺語言模型（VLM）這類多模態系統作為其行為模型的認知核心，這些系統在新的環境中——尤其是在不同地理區域的分佈外（OOD）邊界案例場景——將如何有效泛化？本文探討了這個開放性問題，透過一項全因子分析，比較利馬的人類駕駛員、紐約市的人類駕駛員以及VLM，讓它們觀看來自利馬與紐約市的行車記錄器影片，並在視覺問答（VQA）框架下提出多種問題。我們特別選擇這兩個城市，因為它們都是極具挑戰性的駕駛環境，目前尚無自動駕駛汽車公司在此營運。所提問題涵蓋四大類別：事實性、評分、反事實與推理。我們發現人類與VLM在回應上存在分歧——但此差異受問題類型調節——且人類駕駛員無論來自何處（利馬或紐約），其回答模式相似。出乎意料的是，我們並未觀察到地理因素對回答（無論是人類或VLM）造成顯著差異，這可能源於兩地場景本身的高度分佈外特性。我們的資料集可於此取得：https://huggingface.co/datasets/Artificio/robusto-2

English

As Self-Driving Cars continue to expand internationally and use multi-modal systems such as VLMs as a cognitive backbone for their Action models; how well will these systems generalize in new settings, in particular out-of-distribution (OOD) edge-case scenarios in new geographies? In this paper, we study this open question by providing a full factorial analysis with human drivers of Lima, human drivers from New York City, and VLMs and showing them dashcam footage collected from Lima and New York City -- prompting them with a variety of questions under a Visual Question Answering (VQA) paradigm. In particular, we pick these two cities as they are highly challenging driving locations where no Self-Driving Car company currently operates in, and ask questions that span 4 categories: Factual, Ratings, Counterfactual and Reasoning. We find that Humans and VLMs diverge in their responses -- though this is modulated by the type of questions asked, and that Humans answer similarly independent of where they are from (Lima/NYC). To our surprise, we did not find a strong difference in terms of answers (Humans or VLMs) that was modulated by geography, likely due to their high out-of-distribution nature. Our dataset is available at: https://huggingface.co/datasets/Artificio/robusto-2