FlashRT: 引導代理部署即時多模態應用的代理框架

摘要

即時多模態應用（包括語音代理與互動式視訊生成）將異質模型組成管線，其高效部署需要針對特定應用決定放置（placement）、串流（streaming）及模型內平行化（intra-model parallelism）策略。現有的服務系統與自動平行化編譯器僅支援有限的轉換與固定的工作負載假設，因此針對新應用實現高效能仍需手動打造最佳化實作。我們提出 FlashRT，這是一個代理框架（agent harness），能夠引導編碼代理（coding agents）將開發者簡易撰寫的參考實作（reference implementations）提升為最佳化的多 GPU 部署，並靈活權衡延遲（latency）與吞吐量（throughput）等目標指標。透過全新的鏈式程式範例（chain-of-program paradigm），FlashRT 指導通用編碼代理進行多輪轉換流程：代理先將參考實作轉換為中間表示（IR）以擷取資料相依性與持久狀態範圍，透過循序解譯器（sequential interpreter）驗證此 IR，並執行靜態分析（static analyses）以識別候選轉換。接著，代理在測量門控最佳化迴圈（measurement-gated optimization loop）中反覆實作、驗證並測試每個候選方案，從而產出能適應不同硬體預算的有效部署。在涵蓋視訊世界模型（video world models）與多模態大型語言模型（multimodal LLMs）等多項應用中，FlashRT 將參考實作轉化為高效部署，在 NVIDIA B200 GPU 上實現最高約 70 倍的延遲降低與 2.8 倍的吞吐量提升。在 AMD MI355X GPU 上，FlashRT 達到相同的峰值延遲降低，並將峰值吞吐量提升擴增至 3.6 倍，證明在專家最佳化較不成熟的平台上，代理驅動的最佳化更具可擴展性。事實上，針對 Qwen3-Omni 文字轉語音推論，與 AMD MI355X 上的專家級 vLLM-Omni 實作相比，FlashRT 將回應延遲降低了 65%。

English

Real-time multimodal applications, including voice agents and interactive video generation, compose heterogeneous models into pipelines whose efficient deployment requires application-specific decisions about placement, streaming, and intra-model parallelism. Existing serving systems and auto-parallelism compilers commit to limited transformations and fixed workload assumptions, so achieving high performance on a new application requires hand-crafting an efficient implementation. We present FlashRT, an agent harness that guides coding agents to lift simple developer-written reference implementations into optimized multi-GPU deployments that flexibly weigh target metrics like latency and throughput. Using a new chain-of-program paradigm, FlashRT directs a generic coding agent through a multi-pass transformation process where an agent transforms the reference into an intermediate representation (IR) to capture data dependencies and persistent-state scopes, validates this IR via a sequential interpreter, and performs static analyses to identify candidate transformations. Then, the agent iteratively implements, verifies, and benchmarks each candidate under a measurement-gated optimization loop to produce effective deployments that span different hardware budgets. Across various applications, including video world models and multimodal LLMs, FlashRT converts reference implementations into highly efficient deployments, delivering up to ~70x latency reduction and 2.8x throughput improvement on NVIDIA B200 GPUs. On AMD MI355X GPUs, FlashRT matches the peak latency reduction while increasing peak throughput improvement to 3.6x, demonstrating that agent-driven optimization can be more scalable on platforms with less mature expert optimization. In fact, for Qwen3-Omni text-to-audio inference, FlashRT reduces response latency by 65% compared to the expert vLLM-Omni implementation on AMD MI355X.