評估多模態大型語言模型作為無人機控制的通用視覺-語言-行動代理：指揮、接近、追蹤與搜尋

摘要

多模態大型語言模型（MLLMs）對影像與影片具有強大的感知能力。我們要追問的是：這份能力在多大程度上能延伸至行動層面——亦即直接將一個 MLLM 放入無人機的控制迴路中，而其完整動作空間僅在提示（prompt）中宣告。近期已有系統朝此設定發展，卻日益限縮模型的決策空間。我們反向將其重新放寬。我們提出 DroneCATS-Agent，一個將 MLLM 視為可抽換元件的架構，以及 DroneCATS，一個以模型為自變數的基準。我們的智能體不只是朝向某個像素飛行，而是將偏航與搜索、在不確定時斟酌判斷、以及自我宣告抵達的責任都交由模型承擔——全程無需微調，也無需函式呼叫的結構化定義。我們針對四項核心能力評估前沿模型與開放模型——接近可見目標、追蹤移動目標、搜索初始視野之外的區域，以及指揮多無人機機隊——結果顯示，即便最簡單的具身場景也遠未獲得解決。關鍵的是，為了找出在邊緣端最先失靈的環節，我們的模型陣容一路縮小至 2B 參數。研究結果揭露一個鮮明悖論：失敗的並非飛行本身。小型開放模型往往比前沿模型更可靠地進入成功半徑，卻因過早宣告抵達、或從不宣告抵達，而在該回合中落敗。多無人機指揮更加劇了這種差距——小型模型的失敗方式，是在不同視角之間盲目複製單一座標。若將這些模型視為視覺-語言-行動智能體，其空間感知仍然站得住腳，但行動協議卻崩潰了。區分可部署的邊緣模型與前沿模型的，不是導航能力，而是能否紀律性地維持已宣告的協議、並輸出正確的終止動作。待解的開放問題，在於以機載運算成本縮小此差距——亦即打造一個既能持續規劃、又確切知道何時完成的快速模型——而 DroneCATS 正是為了衡量這段距離而建構。

English

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.