ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Information Science

A low-latency inpainting method for unstably transmitted videos

  • 1.

    Department of Automation, University of Science and Technology of China, Hefei 230027, China

  • 2.

    College of Telecommunications Information Engineering, Nanjing University of Posts and Telecommunications, Nanjing 210003, China

  • 3.

    Agora,Inc., Shanghai 200082, China

Cite this:
https://doi.org/10.52396/JUST-2020-0032
  • Received Date: 24 December 2020
  • Rev Recd Date: 05 February 2021
  • Publish Date: 31 October 2021
    Abstract

  • Abstract

    Video traffic has gradually occupied the majority of mobile traffic, and video damage in unstable transmission remains a common and urgent problem. The difficulty of inpainting these damaged videos is that the holes randomly appear in random video frames, which are hard to be well settled with both low latency and high accuracy. We are the pioneer to look into the video inpainting task in unstable transmission and propose a low-latency video inpainting method which consists of two stages: In the coarsely inpainting stage, we achieve the extraction of damaged two-dimensional optical flow from reference frames, and establish a linear prediction model to coarsely inpaint the damaged frames according to the temporal consistency of motions. In the fine inpainting stage, a Partial Convolutional Frame Completion network(PCFC-Net) is proposed to synthesize all reference information and calculate a fine inpainting result. Compared with that of the state-of-the-art baselines, the waiting time for reference frames is greatly reduced while PSNR and SSIM are improved by 4.0%～12.7% on DAVIS dataset.

    Abstract

    Video traffic has gradually occupied the majority of mobile traffic, and video damage in unstable transmission remains a common and urgent problem. The difficulty of inpainting these damaged videos is that the holes randomly appear in random video frames, which are hard to be well settled with both low latency and high accuracy. We are the pioneer to look into the video inpainting task in unstable transmission and propose a low-latency video inpainting method which consists of two stages: In the coarsely inpainting stage, we achieve the extraction of damaged two-dimensional optical flow from reference frames, and establish a linear prediction model to coarsely inpaint the damaged frames according to the temporal consistency of motions. In the fine inpainting stage, a Partial Convolutional Frame Completion network(PCFC-Net) is proposed to synthesize all reference information and calculate a fine inpainting result. Compared with that of the state-of-the-art baselines, the waiting time for reference frames is greatly reduced while PSNR and SSIM are improved by 4.0%～12.7% on DAVIS dataset.
    • FullText(HTML)
  • loading
    • References(12)
  • [1]
    CISCO. Cisco visual networking index:Global mobile data traffic forecast update, 2017-2022. [2020-12-24], https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-738429.pdf. 2019.
    [2]
    Alatas O, Yan P, Shah M. Spatio-temporal regularity flow (SPREF): Its estimation and applications. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(5): 584-589.
    [3]
    Shih T K, Tang N C, Hwang J N. Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(3): 347-360.
    [4]
    Chung B, Yim C. Bi-sequential video error concealment method using adaptive homography-based registration. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(6): 1535-1549.
    [5]
    Wang C, Huang H, Han X, et al. Video inpainting by jointly learning temporal structure and spatial details. Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: IEEE, 2019, 33: 5232-5239.
    [6]
    Xu R, Li X, Zhou B, et al. Deep flow-guided video inpainting. 2019, arXiv:1905.02884.
    [7]
    Kim D, Woo S, Lee J Y, et al. Deep video inpainting. Proceedings of the Conference on Computer Vision and Pattern Recognition. Long Beach,USA: IEEE, 2019: 5792-5801.
    [8]
    Ilg E, Mayer N, Saikia T, et al. FlowNet 2.0: Evolution of optical flow estimation with deep networks. Proceedings of the Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 2462-2470.
    [9]
    Johnson J, Alahi A,Li F F. Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision. Amsterdam, Netherlands: IEEE, 2016: 694-711.
    [10]
    Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. Proceedings of the Conference On Computer Vision And Pattern Recognition. Boston, USA: IEEE, 2015: 1-9.
    [11]
    Xu N, Yang L, Fan Y, et al. Youtube-VOS: Sequence-to-sequence video object segmentation. Proceedings of the European Conference on Computer Vision. Munich, Germnay: IEEE, 2018: 585-601.
    [12]
    Pont-Tuset J, Perazzi F, Caelles S, et al. The 2017 DAVIS challenge on video object segmentation. 2017, arXiv:1704.00675.
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    Get Citation
    PDF
    XML
    [1]
    CISCO. Cisco visual networking index:Global mobile data traffic forecast update, 2017-2022. [2020-12-24], https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-738429.pdf. 2019.
    [2]
    Alatas O, Yan P, Shah M. Spatio-temporal regularity flow (SPREF): Its estimation and applications. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(5): 584-589.
    [3]
    Shih T K, Tang N C, Hwang J N. Exemplar-based video inpainting without ghost shadow artifacts by maintaining temporal continuity. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(3): 347-360.
    [4]
    Chung B, Yim C. Bi-sequential video error concealment method using adaptive homography-based registration. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(6): 1535-1549.
    [5]
    Wang C, Huang H, Han X, et al. Video inpainting by jointly learning temporal structure and spatial details. Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: IEEE, 2019, 33: 5232-5239.
    [6]
    Xu R, Li X, Zhou B, et al. Deep flow-guided video inpainting. 2019, arXiv:1905.02884.
    [7]
    Kim D, Woo S, Lee J Y, et al. Deep video inpainting. Proceedings of the Conference on Computer Vision and Pattern Recognition. Long Beach,USA: IEEE, 2019: 5792-5801.
    [8]
    Ilg E, Mayer N, Saikia T, et al. FlowNet 2.0: Evolution of optical flow estimation with deep networks. Proceedings of the Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017: 2462-2470.
    [9]
    Johnson J, Alahi A,Li F F. Perceptual losses for real-time style transfer and super-resolution. European Conference on Computer Vision. Amsterdam, Netherlands: IEEE, 2016: 694-711.
    [10]
    Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. Proceedings of the Conference On Computer Vision And Pattern Recognition. Boston, USA: IEEE, 2015: 1-9.
    [11]
    Xu N, Yang L, Fan Y, et al. Youtube-VOS: Sequence-to-sequence video object segmentation. Proceedings of the European Conference on Computer Vision. Munich, Germnay: IEEE, 2018: 585-601.
    [12]
    Pont-Tuset J, Perazzi F, Caelles S, et al. The 2017 DAVIS challenge on video object segmentation. 2017, arXiv:1704.00675.
    Recommended articles
    TrendMD
    Volume 51 Issue 10 page: 717-724
    Cover

    Article Metrics

    Article views (264) PDF downloads(2856)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return