TurboServe: ストリーミング動画生成を効率的かつ経済的に提供する

要旨

流式视频生成正成为一种新兴的服务负载，用户通过长期存活的会话与之交互，以逐块方式逐步生成视频。与离线视频生成或典型的LLM服务不同，流式视频生成必须在活跃与空闲期间保持会话状态，反复调度进行中的会话，并在严格的延迟目标下交付每个视频块。这就在多用户、多GPU环境中产生了两个关键的服务挑战：会话时长异构性——长期运行的会话会随时间推移使放置决策变得次优；以及时间维度上的用户需求异构性——活跃会话数量在流量突发与空闲期之间剧烈波动。我们提出TurboServe，这是首个专门为流式视频生成工作负载设计的服务系统。TurboServe将服务形式化为一个在线调度问题，联合协调会话放置与GPU配置。其闭环调度算法结合了一个感知迁移的放置控制器（在GPU之间重新平衡会话以降低每块最大延迟）和一个负载驱动的自动缩放控制器（根据工作负载变化调整GPU预算以提升成本效率）。为了在运行时支持这些决策，TurboServe实现了合并块处理以批处理同一GPU上的并发活跃会话、用于会话挂起与恢复的GPU-CPU卸载，以及用于在线重新平衡的基于NCCL的GPU间迁移。我们使用生数科技的实际生产跟踪数据，在多种模型规模和最多64块NVIDIA B300 GPU的集群上对TurboServe进行了评估。与基准服务配置相比，TurboServe将最坏情况下的每块延迟平均降低37.5%，并将GPU总运行成本平均降低37.2%。我们的代码已在 https://github.com/shengshu-ai/TurboServe 公开。

English

Streaming video generation is emerging as a new serving workload in which users interact with long-lived sessions that generate video progressively, chunk by chunk. Unlike offline video generation or typical LLM serving, streaming video generation must preserve session state across active and idle periods, repeatedly schedule ongoing sessions, and deliver each chunk under a tight latency target. This creates two key serving challenges in multi-user, multi-GPU environments: session duration heterogeneity, where long-running sessions make placement decisions suboptimal over time, and temporal user-demand heterogeneity, where the number of active sessions fluctuates sharply across bursts and idle periods. We present TurboServe, the first serving system designed specifically for streaming video generation workloads. TurboServe formulates serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduling algorithm combines a migration-aware placement controller, which rebalances sessions across GPUs to reduce the maximum per-chunk latency, with a load-driven autoscaling controller, which adapts the GPU budget to workload variation for improved cost efficiency. To support these decisions at runtime, TurboServe implements coalesced chunk processing for batching concurrent active sessions on the same GPU, GPU-CPU offloading for session suspension and resumption, and NCCL-based GPU-GPU migration for online rebalancing. We evaluate TurboServe on real-world production traces from Shengshu Technology across multiple model sizes and GPU clusters with up to 64 NVIDIA B300 GPUs. Compared with baseline serving configurations, TurboServe reduces worst-case per-chunk latency by 37.5% and total GPU operating cost by 37.2% on average. Our code is publicly available at https://github.com/shengshu-ai/TurboServe.