TurboServe：高效且經濟地提供串流影片生成服務

摘要

流式视频生成正逐渐成為一種新興的服務工作負載，用戶透過長時間運行的會話與之互動，系統會逐步、逐區塊地生成影片。不同於離線影片生成或典型的LLM服務，流式視頻生成必須在活躍與空閒期間保存會話狀態，反覆調度正在進行的會話，並在嚴格的延遲目標內交付每個區塊。這在多用戶、多GPU環境中帶來了兩項主要的服務挑戰：會話持續時間異質性（長期運行的會話會使放置決策隨時間推移變得次優），以及時間性用戶需求異質性（活躍會話數量在爆發期與空閒期之間急劇波動）。我們提出TurboServe，這是第一個專為流式視頻生成工作負載設計的服務系統。TurboServe將服務問題表述為一個在線調度問題，協同協調會話放置與GPU供應。其閉環調度演算法結合了遷移感知的放置控制器（透過跨GPU重新平衡會話以降低最大每區塊延遲）與負載驅動的自動擴縮控制器（根據工作負載變化調整GPU預算以提高成本效率）。為在運行時支援這些決策，TurboServe實現了合併區塊處理（在相同GPU上批次處理並行的活躍會話）、GPU-CPU卸載（用於會話暫停與恢復），以及基於NCCL的GPU-GPU遷移（用於在線重新平衡）。我們基於生數科技的真實生產軌跡，在多種模型規模與最多64塊NVIDIA B300 GPU的集群上評估了TurboServe。與基線服務配置相比，TurboServe平均降低了37.5%的最壞情況每區塊延遲，並節省了37.2%的總GPU營運成本。我們的程式碼已公開於 https://github.com/shengshu-ai/TurboServe。

English

Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.