TurboServe：高效且经济地提供流式视频生成服务

摘要

流式视频生成正成为一种新兴的服务负载，用户通过与长时间运行的会话交互，逐块渐进式生成视频。与离线视频生成或典型的LLM服务不同，流式视频生成必须在活动与空闲时段间保持会话状态，重复调度进行中的会话，并在严格的延迟目标下交付每个视频块。这就在多用户、多GPU环境中产生了两个关键的服务挑战：会话时长异质性——长时间运行的会话随时间推移使放置决策次优化；以及用户需求的时间异质性——活动会话数量在突发高峰与空闲时段间剧烈波动。我们提出TurboServe，首个专为流式视频生成负载设计的服务系统。TurboServe将服务形式化为一个在线调度问题，联合协调会话放置与GPU配置。其闭环调度算法结合了迁移感知放置控制器（通过跨GPU重新平衡会话以降低最大每块延迟）与负载驱动的自动扩缩控制器（根据负载变化自适应调整GPU预算以提升成本效率）。为支持上述运行时决策，TurboServe实现了用于同一GPU上并发活动会话批处理的合并块处理、用于会话挂起与恢复的GPU-CPU卸载，以及基于NCCL的GPU-GPU迁移以实现在线重平衡。我们使用生数科技的真实生产轨迹，在多种模型规模及最多64块NVIDIA B300 GPU的集群上评估了TurboServe。与基准服务配置相比，TurboServe将最坏情况下每块延迟平均降低37.5%，总GPU运营成本平均降低37.2%。我们的代码已在 https://github.com/shengshu-ai/TurboServe 公开。

English

Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.