ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Information Science and Technology 20 April 2022

Steganalysis of neural networks based on parameter statistical bias

  • School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230029, China

Cite this:
https://doi.org/10.52396/JUSTC-2021-0197
More Information
  • Author Bio:

    Yi Yin is working towards the PhD degree at School of Cyberspace Science and Technology, University of Science and Technology of China. Her area of interests includes Neural Network Steganography

    Kejiang Chen is a postdoctoral researcher at the School of Cyberspace Science and Technology, University of Science and Technology of China. His research interests include information hiding and artificial intelligence security. He has published more than 20 papers in IEEE TDSC, TIFS, TVCG, TCSVT, CVPR, ICCV and other journals and conferences

  • Corresponding author: E-mail: chenkj@ustc.edu.cn
  • Received Date: 07 September 2021
  • Accepted Date: 01 December 2021
    • Available Online: 20 April 2022
    Abstract

  • Abstract

    Many pretrained deep learning models have been released to help engineers and researchers develop deep learning-based systems or conduct research with minimall effort. Previous work has shown that at secret message can be embedded in neural network parameters without compromising the accuracy of the model. Malicious developers can, therefore, hide malware or other baneful information in pretrained models, causing harm to society. Hence, reliable detection of these vicious pretrained models is urgently needed. We analyze existing approaches for hiding messages and find that they will ineluctably cause biases in the parameter statistics. Therefore, we propose steganalysis methods for steganography on neural network parameters that extract statistics from benign and malicious models and build classifiers based on the extracted statistics. To the best of our knowledge, this is the first study on neural network steganalysis. The experimental results reveal that our proposed algorithm can effectively detect a model with an embedded message. Notably, our detection methods are still valid in cases where the payload of the stego model is low.

    Graphical abstract

      Malicious developers can hide malware or other baneful information into the pretrained model imperceptibly, which does harm to computer society. However, steganography on neural network will modify the statistical distribution of the model. We propose steganalysis methods for steganography on neural network parameters by extracting statistics from benign and malicious models and building classifiers based on the statistics.

    Abstract

    Many pretrained deep learning models have been released to help engineers and researchers develop deep learning-based systems or conduct research with minimall effort. Previous work has shown that at secret message can be embedded in neural network parameters without compromising the accuracy of the model. Malicious developers can, therefore, hide malware or other baneful information in pretrained models, causing harm to society. Hence, reliable detection of these vicious pretrained models is urgently needed. We analyze existing approaches for hiding messages and find that they will ineluctably cause biases in the parameter statistics. Therefore, we propose steganalysis methods for steganography on neural network parameters that extract statistics from benign and malicious models and build classifiers based on the extracted statistics. To the best of our knowledge, this is the first study on neural network steganalysis. The experimental results reveal that our proposed algorithm can effectively detect a model with an embedded message. Notably, our detection methods are still valid in cases where the payload of the stego model is low.

    • Public Summary

    • Different steganography of neural networks will lead to different bias in the statistics of the parameters. By designing valid features to capture bias and training the classifier, we can effectively detect the injected network.
    • The length of the secret message embedded by the malicious developer is unknown. The experimental results show that even when the payload is low, our detecting method still works.
    • Since the method used to embed information in the injected network is unknown, an effective framework allows for a more comprehensive detection of the injected network.

    FullText(HTML)
  • loading
    • References(33)
  • [1]
    Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518 (7540): 529–533. doi: 10.1038/nature14236
    [2]
    Lin X, Rivenson Y, Yardimci N T, et al. All-optical machine learning using diffractive deep neural networks. Science, 2018, 361 (6406): 1004–1008. doi: 10.1126/science.aat8084
    [3]
    Hirschberg J, Manning C D. Advances in natural language processing. Science, 2015, 349 (6245): 261–266. doi: 10.1126/science.aaa8685
    [4]
    Mathis A, Mamidanna P, Cury K M, et al. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 2018, 21 (9): 1281–1289. doi: 10.1038/s41593-018-0209-y
    [5]
    LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86 (11): 2278–2324. doi: 10.1109/5.726791
    [6]
    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014, arXiv: 1409.1556. https://arxiv.53yu.com/abs/1409.1556
    [7]
    Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1–9. doi: 10.1109/CVPR.2015.7298594
    [8]
    He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. doi: 10.1109/CVPR.2016.90
    [9]
    Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning. PMLR, 2019: 6105-6114. http://proceedings.mlr.press/v97/tan19a.html
    [10]
    Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779–788. doi: 10.1109/CVPR.2016.91
    [11]
    Taigman Y, Yang M, Ranzato M A, et al. DeepFace: Closing the gap to human-level performance in face verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 1701–1708. doi: 10.1145/3065386
    [12]
    Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 25: 1097–1105. doi: 10.1145/3065386
    [13]
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521 (7553): 436–444. doi: 10.1038/nature14539
    [14]
    Song C, Ristenpart T, Shmatikov V. Machine learning models that remember too much. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security., 2017: 587–601. doi: 10.1145/3133956.3134077
    [15]
    Liu T, Liu Z, Liu Q, et al. StegoNet: Turn deep neural network into a stegomalware. Annual Computer Security Applications Conference, 2020: 928–938. doi: 10.1145/3427228.3427268
    [16]
    Han S, Mao H, Dally W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2015, arXiv: 1510.00149. https://arxiv.53yu.com/abs/1510.00149
    [17]
    Dumitrescu S, Wu X, Memon N. On steganalysis of random LSB embedding in continuous-tone images. Proceedings of the International Conference on Image Processing. IEEE, 2002, 3: 641–644. doi: 10.1109/ICIP.2002.1039052
    [18]
    Dumitrescu S, Wu X, Wang Z. Detection of LSB steganography via sample pair analysis. International Workshop on Information Hiding. Berlin, Heidelberg: Springer, 2002: 355-372. https://sci.bban.top/pdf/10.1109/tsp.2003.812753.pdf#view=FitH
    [19]
    Westfeld A, Pfitzmann A. Attacks on steganographic systems. International workshop on information hiding. Berlin, Heidelberg: Springer, 1999: 61-76. https://linkspringer.53yu.com/chapter/10.1007/10719724_5
    [20]
    Fridrich J, Goljan M, Du R. Reliable detection of LSB steganography in color and grayscale images. Proceedings of the 2001 Workshop on Multimedia and Security: New Challenges, 2001: 27–30.
    [21]
    Fridrich J, Goljan M. Practical steganalysis of digital images: State of the art. Security and Watermarking of Multimedia Contents IV. International Society for Optics and Photonics, 2002, 4675: 1–13. doi: 10.1117/12.465263
    [22]
    Kahan W. IEEE standard 754 for binary floating-point arithmetic. Lecture Notes on the Status of IEEE, 1996, 754(94720-1776): 11. http://li.mit.edu/Archive/Activities/Archive/CourseWork/Ju_Li/MITCourses/18.335/Doc/IEEE754/ieee754.pdf
    [23]
    Suarez-Tangil G, Tapiador J E, Peris-Lopez P. Stegomalware: Playing hide and seek with malicious components in smartphone apps. International Conference on Information Security and Cryptology. Springer, Cham, 2014: 496-515. https://linkspringer.53yu.com/chapter/10.1007/978-3-319-16745-9_27
    [24]
    Freedman D A. Statistical Models: Theory and Practice. Cambridge University Press, 2009. https://xs.dailyheadlines.cc/books?hl=zh-CN&lr=&id=fW_9BV5Wpf8C&oi=fnd&pg=PR1&dq=Statistical+models:+theory+and+practice.+Cambridge+University+Press,+2009.&ots=2iLcXDDULK&sig=LIKNKcP1bq7U0-rDYveTovtwoPE
    [25]
    Cox D R. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 1958, 20 (2): 215–232. doi: 10.1111/j.2517-6161.1958.tb00292.x
    [26]
    Walker S H, Duncan D B. Estimation of the probability of an event as a function of several independent variables. Biometrika, 1967, 54 (1−2): 167–179. doi: 10.1093/biomet/54.1-2.167
    [27]
    Krizhevsky A. Learning Multiple Layers of Features From Tiny Images. ACM Press, 2009. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.186.4550&rep=rep1&type=pdf
    [28]
    Alani M M. Testing randomness in ciphertext of block-ciphers using DieHard tests. Int. J. Comput. Sci. Netw. Secur, 2010, 10(4): 53-57. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.186.4550&rep=rep1&type=pdf
    [29]
    Rukhin A, Soto J, Nechvatal J, et al. A statistical test suite for random and pseudorandom number generators for cryptographic applications. Booz-allen and hamilton inc mclean va, 2001. https://agris.fao.org/agris-search/search.do?recordID=US201300122719
    [30]
    Hernandez J C, Sierra J M, Seznec A. The SAC test: a new randomness test, with some applications to PRNG analysis. International Conference on Computational Science and Its Applications. Berlin, Heidelberg, Springer, 2004: 960-967. https://linkspringer.53yu.com/chapter/10.1007/978-3-540-24707-4_108
    [31]
    Ryabko B Y, Stognienko V S, Shokin Y I. A new test for randomness and its application to some cryptographic problems. Journal of Statistical Planning and Inference, 2004, 123 (2): 365–376. doi: 10.1016/S0378-3758(03)00149-6
    [32]
    Tiny ImageNet. https://tiny-imagenet.herokuapp.com, 2019-11-01.
    [33]
    Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017, arXiv: 1704.04861. https://arxiv.53yu.com/abs/1704.04861
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    Get Citation
    PDF
    XML

    Figure  1.  An example of binary interchange formats for a float 32 number, where the green part is the sign, the yellow part is the biased exponent, and the red part is the trailing significant field.

    Figure  2.  Illustration of the framework we proposed for the neural network steganalysis.

    Figure  3.  Averaged accuracy over the bit planes from the 14th to 18th bit plane of ResNet34 trained on CIFAR10 with a payload of 1.0 bit per parameter for each subclassifier.

    Figure  4.  Results for the detection of ResNet34, VGG16, and EfficientNetB0 trained on CIFAR10 using different statistics as features respectively. (a) results for correlated value encoding steganalysis; and (b) results for sign encoding steganalysis.

    [1]
    Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature, 2015, 518 (7540): 529–533. doi: 10.1038/nature14236
    [2]
    Lin X, Rivenson Y, Yardimci N T, et al. All-optical machine learning using diffractive deep neural networks. Science, 2018, 361 (6406): 1004–1008. doi: 10.1126/science.aat8084
    [3]
    Hirschberg J, Manning C D. Advances in natural language processing. Science, 2015, 349 (6245): 261–266. doi: 10.1126/science.aaa8685
    [4]
    Mathis A, Mamidanna P, Cury K M, et al. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience, 2018, 21 (9): 1281–1289. doi: 10.1038/s41593-018-0209-y
    [5]
    LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86 (11): 2278–2324. doi: 10.1109/5.726791
    [6]
    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. 2014, arXiv: 1409.1556. https://arxiv.53yu.com/abs/1409.1556
    [7]
    Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1–9. doi: 10.1109/CVPR.2015.7298594
    [8]
    He K, Zhang X, Ren S, et al. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770–778. doi: 10.1109/CVPR.2016.90
    [9]
    Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning. PMLR, 2019: 6105-6114. http://proceedings.mlr.press/v97/tan19a.html
    [10]
    Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779–788. doi: 10.1109/CVPR.2016.91
    [11]
    Taigman Y, Yang M, Ranzato M A, et al. DeepFace: Closing the gap to human-level performance in face verification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 1701–1708. doi: 10.1145/3065386
    [12]
    Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 25: 1097–1105. doi: 10.1145/3065386
    [13]
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521 (7553): 436–444. doi: 10.1038/nature14539
    [14]
    Song C, Ristenpart T, Shmatikov V. Machine learning models that remember too much. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security., 2017: 587–601. doi: 10.1145/3133956.3134077
    [15]
    Liu T, Liu Z, Liu Q, et al. StegoNet: Turn deep neural network into a stegomalware. Annual Computer Security Applications Conference, 2020: 928–938. doi: 10.1145/3427228.3427268
    [16]
    Han S, Mao H, Dally W J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. 2015, arXiv: 1510.00149. https://arxiv.53yu.com/abs/1510.00149
    [17]
    Dumitrescu S, Wu X, Memon N. On steganalysis of random LSB embedding in continuous-tone images. Proceedings of the International Conference on Image Processing. IEEE, 2002, 3: 641–644. doi: 10.1109/ICIP.2002.1039052
    [18]
    Dumitrescu S, Wu X, Wang Z. Detection of LSB steganography via sample pair analysis. International Workshop on Information Hiding. Berlin, Heidelberg: Springer, 2002: 355-372. https://sci.bban.top/pdf/10.1109/tsp.2003.812753.pdf#view=FitH
    [19]
    Westfeld A, Pfitzmann A. Attacks on steganographic systems. International workshop on information hiding. Berlin, Heidelberg: Springer, 1999: 61-76. https://linkspringer.53yu.com/chapter/10.1007/10719724_5
    [20]
    Fridrich J, Goljan M, Du R. Reliable detection of LSB steganography in color and grayscale images. Proceedings of the 2001 Workshop on Multimedia and Security: New Challenges, 2001: 27–30.
    [21]
    Fridrich J, Goljan M. Practical steganalysis of digital images: State of the art. Security and Watermarking of Multimedia Contents IV. International Society for Optics and Photonics, 2002, 4675: 1–13. doi: 10.1117/12.465263
    [22]
    Kahan W. IEEE standard 754 for binary floating-point arithmetic. Lecture Notes on the Status of IEEE, 1996, 754(94720-1776): 11. http://li.mit.edu/Archive/Activities/Archive/CourseWork/Ju_Li/MITCourses/18.335/Doc/IEEE754/ieee754.pdf
    [23]
    Suarez-Tangil G, Tapiador J E, Peris-Lopez P. Stegomalware: Playing hide and seek with malicious components in smartphone apps. International Conference on Information Security and Cryptology. Springer, Cham, 2014: 496-515. https://linkspringer.53yu.com/chapter/10.1007/978-3-319-16745-9_27
    [24]
    Freedman D A. Statistical Models: Theory and Practice. Cambridge University Press, 2009. https://xs.dailyheadlines.cc/books?hl=zh-CN&lr=&id=fW_9BV5Wpf8C&oi=fnd&pg=PR1&dq=Statistical+models:+theory+and+practice.+Cambridge+University+Press,+2009.&ots=2iLcXDDULK&sig=LIKNKcP1bq7U0-rDYveTovtwoPE
    [25]
    Cox D R. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 1958, 20 (2): 215–232. doi: 10.1111/j.2517-6161.1958.tb00292.x
    [26]
    Walker S H, Duncan D B. Estimation of the probability of an event as a function of several independent variables. Biometrika, 1967, 54 (1−2): 167–179. doi: 10.1093/biomet/54.1-2.167
    [27]
    Krizhevsky A. Learning Multiple Layers of Features From Tiny Images. ACM Press, 2009. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.186.4550&rep=rep1&type=pdf
    [28]
    Alani M M. Testing randomness in ciphertext of block-ciphers using DieHard tests. Int. J. Comput. Sci. Netw. Secur, 2010, 10(4): 53-57. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.186.4550&rep=rep1&type=pdf
    [29]
    Rukhin A, Soto J, Nechvatal J, et al. A statistical test suite for random and pseudorandom number generators for cryptographic applications. Booz-allen and hamilton inc mclean va, 2001. https://agris.fao.org/agris-search/search.do?recordID=US201300122719
    [30]
    Hernandez J C, Sierra J M, Seznec A. The SAC test: a new randomness test, with some applications to PRNG analysis. International Conference on Computational Science and Its Applications. Berlin, Heidelberg, Springer, 2004: 960-967. https://linkspringer.53yu.com/chapter/10.1007/978-3-540-24707-4_108
    [31]
    Ryabko B Y, Stognienko V S, Shokin Y I. A new test for randomness and its application to some cryptographic problems. Journal of Statistical Planning and Inference, 2004, 123 (2): 365–376. doi: 10.1016/S0378-3758(03)00149-6
    [32]
    Tiny ImageNet. https://tiny-imagenet.herokuapp.com, 2019-11-01.
    [33]
    Howard A G, Zhu M, Chen B, et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. 2017, arXiv: 1704.04861. https://arxiv.53yu.com/abs/1704.04861
    Recommended articles
    TrendMD
    Volume 52 Issue 1 page: 1
    Cover

    Article Metrics

    Article views (1243) PDF downloads(4160)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return