CoDeF：用於時間一致視頻處理的內容變形場

摘要

我們提出了一種名為內容變形場（CoDeF）的新型視頻表示形式，由一個規範內容場和一個記錄從規範圖像（即從規範內容場呈現的）到每個單獨幀的變換的時間變形場組成。對於目標視頻，這兩個場是通過精心設計的渲染管道共同優化以重建它。我們特意在優化過程中引入了一些正則化，促使規範內容場從視頻中繼承語義（例如對象形狀）。通過這種設計，CoDeF自然支持將圖像算法應用於視頻處理，即可以將圖像算法應用於規範圖像，並通過時間變形場輕鬆將結果傳播到整個視頻。我們通過實驗表明，CoDeF能夠將圖像到圖像的轉換提升為視頻到視頻的轉換，並將關鍵點檢測提升為關鍵點跟踪而無需任何訓練。更重要的是，由於我們的提升策略僅在一個圖像上部署算法，與現有的視頻到視頻轉換方法相比，我們在處理的視頻中實現了更優越的跨幀一致性，甚至能夠跟踪水和煙霧等非剛性對象。項目頁面可在https://qiuyu96.github.io/CoDeF/找到。

English

We present the content deformation field CoDeF as a new type of video representation, which consists of a canonical content field aggregating the static contents in the entire video and a temporal deformation field recording the transformations from the canonical image (i.e., rendered from the canonical content field) to each individual frame along the time axis.Given a target video, these two fields are jointly optimized to reconstruct it through a carefully tailored rendering pipeline.We advisedly introduce some regularizations into the optimization process, urging the canonical content field to inherit semantics (e.g., the object shape) from the video.With such a design, CoDeF naturally supports lifting image algorithms for video processing, in the sense that one can apply an image algorithm to the canonical image and effortlessly propagate the outcomes to the entire video with the aid of the temporal deformation field.We experimentally show that CoDeF is able to lift image-to-image translation to video-to-video translation and lift keypoint detection to keypoint tracking without any training.More importantly, thanks to our lifting strategy that deploys the algorithms on only one image, we achieve superior cross-frame consistency in processed videos compared to existing video-to-video translation approaches, and even manage to track non-rigid objects like water and smog.Project page can be found at https://qiuyu96.github.io/CoDeF/.

CoDeF：用於時間一致視頻處理的內容變形場

CoDeF: Content Deformation Fields for Temporally Consistent Video Processing

摘要

Support