ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Research Articles 19 May 2023

LightAD: Accelerating AutoDebias with Adaptive Sampling

  • School of Information Science and Technology, University of Science of Technology of China, Hefei 230000, China

Cite this:
https://doi.org/10.52396/JUSTC-2022-0100
More Information
  • Corresponding author: Jiawei Chen, cjwustc@ustc.edu.cn
  • Accepted Date: 03 April 2023
    • Available Online: 19 May 2023
    Abstract

  • Abstract

    In recommendation systems, the bias issue is ubiquitous as the data is collected from user behaviors rather than reasonable experiments. AutoDebias, which resorts to meta learning to find appropriate debiasing configurations, i.e., pseudo-labels and confidence weights for all user-item pairs, has been demonstrated as a generic and effective solution in tackling various biases. Nevertheless, setting pseudo-labels and weights for every user-item pair can be a time-consuming process. Therefore, AutoDebias suffers from a huge computational cost, making it less applicable to real cases. Although stochastic gradient descent with a uniform sampler can be applied to accelerate training, it would significantly deteriorate model convergence and stability. To overcome this problem, we propose LightAutoDebias (short as LightAD), which equips AutoDebias with a specialized importance sampling strategy. The sampler can adaptively and dynamically draw informative training instances, which brings provably better convergence and stability than the standard uniform sampler. Extensive experiments on three benchmark datasets validate that our LightAD accelerates AutoDebias by several magnitudes while maintaining almost equal accuracy.

    Graphical abstract

    Abstract

    In recommendation systems, the bias issue is ubiquitous as the data is collected from user behaviors rather than reasonable experiments. AutoDebias, which resorts to meta learning to find appropriate debiasing configurations, i.e., pseudo-labels and confidence weights for all user-item pairs, has been demonstrated as a generic and effective solution in tackling various biases. Nevertheless, setting pseudo-labels and weights for every user-item pair can be a time-consuming process. Therefore, AutoDebias suffers from a huge computational cost, making it less applicable to real cases. Although stochastic gradient descent with a uniform sampler can be applied to accelerate training, it would significantly deteriorate model convergence and stability. To overcome this problem, we propose LightAutoDebias (short as LightAD), which equips AutoDebias with a specialized importance sampling strategy. The sampler can adaptively and dynamically draw informative training instances, which brings provably better convergence and stability than the standard uniform sampler. Extensive experiments on three benchmark datasets validate that our LightAD accelerates AutoDebias by several magnitudes while maintaining almost equal accuracy.
    • FullText(HTML)
  • loading
    • References(45)
  • [1]
    He X, Liao L, Zhang H, et al. Neural collaborative filtering. In: WWW '17: Proceedings of the 26th International Conference on World Wide Web. Perth, Australia: ACM, 2017: 173–182.
    [2]
    Yuan F, He X, Karatzoglou A, et al. Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 1469–1478.[LinkOut]
    [3]
    Sun F, Liu J, Wu J, et al. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York: ACM, 2019: 1441–1450.[LinkOut]
    [4]
    Abdollahpouri H, Burke R, Mobasher B. Controlling popularity bias in learning-to-rank recommendation. Proceedings of the Eleventh ACM Conference on Recommender Systems. New York: ACM, 2017: 42–46.[LinkOut]
    [5]
    Liu D, Cheng P, Dong Z, et al. A general knowledge distillation framework for counterfactual recommendation via uniform data. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 831–840.[LinkOut]
    [6]
    Schnabel T, Swaminathan A, Singh A, et al. Recommendations as treatments: Debiasing learning and evaluation. Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. New York: ACM, 2016: 1670–1679.[LinkOut]
    [7]
    Wang X, Bendersky M, Metzler D, et al. Learning to rank with selection bias in personal search. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. New York: ACM, 2016: 115–124.[LinkOut]
    [8]
    Hernández-Lobato J M, Houlsby N, Ghahramani Z. Probabilistic matrix factorization with non-random missing data. Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32. New York: ACM, 2014: Ⅱ–1512.[LinkOut]
    [9]
    Steck H. Evaluation of recommendations: Rating-prediction and ranking. Proceedings of the 7th ACM conference on Recommender systems. New York: ACM, 2013: 213–220.[LinkOut]
    [10]
    Chen J, Dong H, Qiu Y, et al. AutoDebias: learning to debias for recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA: ACM, 2021: 21–30.[LinkOut]
    [11]
    Chen J, Dong H, Wang X, et al. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 2023, 41: 1–39.
    [12]
    Marlin B, Zemel R S ", Roweis S, et al. Collaborative filtering and the missing at random assumption". 2012: arXiv: 1206.5267. https://arxiv.org/abs/1206.5267"[LinkOut]
    [13]
    Steck H. Training and testing of recommender systems on data missing not at random. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2010: 713–722.[LinkOut]
    [14]
    Krishnan S, Patel J, Franklin M J, et al. A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. Proceedings of the 8th ACM Conference on Recommender systems. New York: ACM, 2014: 137–144.[LinkOut]
    [15]
    Liu Y, Cao X, Yu Y. Are You influenced by others when rating? : Improve rating prediction by conformity modeling. Proceedings of the 10th ACM Conference on Recommender Systems. New York: ACM, 2016: 269–272.[LinkOut]
    [16]
    Tang J, Gao H, Liu H. mTrust: Discerning multi-faceted trust in a connected world. Proceedings of the fifth ACM international conference on Web search and data mining. New York: ACM, 2012: 93–102.[LinkOut]
    [17]
    Ma H, King I, Lyu M R. Learning to recommend with social trust ensemble. Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. New York: ACM, 2009: 203–210.[LinkOut]
    [18]
    Hu Y, Koren Y, Volinsky C. Collaborative filtering for implicit feedback datasets. In: 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2009: 263–272.[LinkOut]
    [19]
    Pan R, Scholz M. Mind the gaps: Weighting the unknown in large-scale one-class collaborative filtering. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2009: 667–676.[LinkOut]
    [20]
    Chen J, Wang C, Zhou S, et al. Fast adaptively weighted matrix factorization for recommendation with implicit feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 3470–3477. doi: 10.1609/aaai.v34i04.5751
    [21]
    Dupret G E, Piwowarski B. A user browsing model to predict search engine click data from past observations. Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. New York: ACM, 2008: 331–338.[LinkOut]
    [22]
    Zhang W V, Jones R. Comparing click logs and editorial labels for training query rewriting. In: WWW 2007 Workshop on Query Log Analysis: Social And Technological Challenges. Banff, Canada: ACM, 2007.
    [23]
    Craswell N, Zoeter O, Taylor M, et al. An experimental comparison of click position-bias models. Proceedings of the 2008 International Conference on Web Search and Data Mining. New York: ACM, 2008: 87–94.[LinkOut]
    [24]
    Guo F, Liu C, Kannan A, et al. Click chain model in web search. Proceedings of the 18th international conference on World wide web. New York: ACM, 2009: 11–20.[LinkOut]
    [25]
    Zhu Z A, Chen W, Minka T, et al. A novel click model and its applications to online advertising. Proceedings of the third ACM international conference on Web search and data mining. New York: ACM, 2010: 321–330.[LinkOut]
    [26]
    Kamishima T, Akaho S, Asoh H, et al. Correcting popularity bias by enhancing recommendation neutrality. In: RecSys '14: Proceedings of the 8th ACM Conference on Recommender Systems. Foster City, USA: ACM, 2014.
    [27]
    Zheng Y, Gao C, Li X, et al. Disentangling user interest and conformity for recommendation with causal embedding. Proceedings of the Web Conference 2021. New York: ACM, 2021: 2980–2991.[LinkOut]
    [28]
    Krishnan A, Sharma A, Sankar A, et al. An adversarial approach to improve long-tail performance in neural collaborative filtering. Proceedings of the 27th ACM International Conference on Information and Knowledge Management. New York: ACM, 2018: 1491–1494.[LinkOut]
    [29]
    He R, McAuley J. VBPR: Visual Bayesian personalized ranking from implicit feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30: ■–■.[LinkOut]
    [30]
    Wu Y, DuBois C, Zheng A X, et al. Collaborative denoising auto-encoders for top-N recommender systems. Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. New York: ACM, 2016: 153–162.[LinkOut]
    [31]
    He X, Deng K, Wang X, et al. LightGCN: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 639–648.[LinkOut]
    [32]
    He X, Du X, Wang X, et al. Outer product-based neural collaborative filtering. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2018: 2227–2233.[LinkOut]
    [33]
    Yu H F, Bilenko M, Lin C J. Selection of negative samples for one-class matrix factorization. Proceedings of the 2017 SIAM international conference on data mining. Philadelphia, PA: Society for Industrial and Applied Mathematics, 2017: 363–371.[LinkOut]
    [34]
    Park D H, Chang Y. Adversarial sampling and training for semi-supervised information retrieval. WWW '19: The World Wide Web Conference. New York: ACM, 2019: 1443–1453.[LinkOut]
    [35]
    Rendle S, Freudenthaler C. Improving pairwise learning for item recommendation from implicit feedback. Proceedings of the 7th ACM international conference on Web search and data mining. New York: ACM, 2014: 273–282.[LinkOut]
    [36]
    Ding J, Quan Y, He X, et al. Reinforced negative sampling for recommendation with exposure data. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2019: 2230–2236.[LinkOut]
    [37]
    Ding J, Feng F, He X, et al. An improved sampler for Bayesian personalized ranking by leveraging view data. Proceedings of the The Web Conference 2018. New York: ACM, 2018: 13–14.[LinkOut]
    [38]
    Chen J, Wang C, Zhou S, et al. SamWalker: Social recommendation with informative sampling strategy. WWW '19: The World Wide Web Conference. New York: ACM, 2019: 228–239.[LinkOut]
    [39]
    Haussler D. Probably approximately correct learning. University of California, Santa Cruz, Computer Research Laboratory, 1990.
    [40]
    Sun W, Khenissi S, Nasraoui O, et al. Debiasing the human-recommender system feedback loop in collaborative filtering. Proceedings of The 2019 World Wide Web Conference. New York: ACM, 2019: 645–651.[LinkOut]
    [41]
    Gleich D F, Lim L H. Rank aggregation via nuclear norm minimization. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2011: 60–68.[LinkOut]
    [42]
    Koren Y, Bell R, Volinsky C. Matrix factorization techniques for recommender systems. Computer, 2009, 42: 30–37.
    [43]
    Wang X, Zhang R, Sun Y and Qi J. Doubly robust joint learning for recommendation on data missing not at random. In: ICML '19: Proceedings of the 36th International Conference on Machine Learning. Long Beach, USA: PMLR, 2019: 6638–6647.
    [44]
    Ai Q, Bi K, Luo C, et al. Unbiased learning to rank with unbiased propensity estimation. SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. New York: ACM, 2018: 385–394.[LinkOut]
    [45]
    Ovaisi Z, Ahsan R, Zhang Y, et al. Correcting for selection bias in learning-to-rank systems. Proceedings of The Web Conference 2020. New York: ACM, 2020: 1863–1873.[LinkOut]
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    Get Citation
    PDF
    XML

    Figure  2.  Training process of LightAD-Fixed and LightAD-Uniform

    Figure  1.  The sampling framework of LightAD

    Figure  3.  The influence of hyperparameter β on self-paced sampling strategy

    [1]
    He X, Liao L, Zhang H, et al. Neural collaborative filtering. In: WWW '17: Proceedings of the 26th International Conference on World Wide Web. Perth, Australia: ACM, 2017: 173–182.
    [2]
    Yuan F, He X, Karatzoglou A, et al. Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 1469–1478.[LinkOut]
    [3]
    Sun F, Liu J, Wu J, et al. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York: ACM, 2019: 1441–1450.[LinkOut]
    [4]
    Abdollahpouri H, Burke R, Mobasher B. Controlling popularity bias in learning-to-rank recommendation. Proceedings of the Eleventh ACM Conference on Recommender Systems. New York: ACM, 2017: 42–46.[LinkOut]
    [5]
    Liu D, Cheng P, Dong Z, et al. A general knowledge distillation framework for counterfactual recommendation via uniform data. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 831–840.[LinkOut]
    [6]
    Schnabel T, Swaminathan A, Singh A, et al. Recommendations as treatments: Debiasing learning and evaluation. Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48. New York: ACM, 2016: 1670–1679.[LinkOut]
    [7]
    Wang X, Bendersky M, Metzler D, et al. Learning to rank with selection bias in personal search. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. New York: ACM, 2016: 115–124.[LinkOut]
    [8]
    Hernández-Lobato J M, Houlsby N, Ghahramani Z. Probabilistic matrix factorization with non-random missing data. Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32. New York: ACM, 2014: Ⅱ–1512.[LinkOut]
    [9]
    Steck H. Evaluation of recommendations: Rating-prediction and ranking. Proceedings of the 7th ACM conference on Recommender systems. New York: ACM, 2013: 213–220.[LinkOut]
    [10]
    Chen J, Dong H, Qiu Y, et al. AutoDebias: learning to debias for recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA: ACM, 2021: 21–30.[LinkOut]
    [11]
    Chen J, Dong H, Wang X, et al. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 2023, 41: 1–39.
    [12]
    Marlin B, Zemel R S ", Roweis S, et al. Collaborative filtering and the missing at random assumption". 2012: arXiv: 1206.5267. https://arxiv.org/abs/1206.5267"[LinkOut]
    [13]
    Steck H. Training and testing of recommender systems on data missing not at random. Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2010: 713–722.[LinkOut]
    [14]
    Krishnan S, Patel J, Franklin M J, et al. A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. Proceedings of the 8th ACM Conference on Recommender systems. New York: ACM, 2014: 137–144.[LinkOut]
    [15]
    Liu Y, Cao X, Yu Y. Are You influenced by others when rating? : Improve rating prediction by conformity modeling. Proceedings of the 10th ACM Conference on Recommender Systems. New York: ACM, 2016: 269–272.[LinkOut]
    [16]
    Tang J, Gao H, Liu H. mTrust: Discerning multi-faceted trust in a connected world. Proceedings of the fifth ACM international conference on Web search and data mining. New York: ACM, 2012: 93–102.[LinkOut]
    [17]
    Ma H, King I, Lyu M R. Learning to recommend with social trust ensemble. Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. New York: ACM, 2009: 203–210.[LinkOut]
    [18]
    Hu Y, Koren Y, Volinsky C. Collaborative filtering for implicit feedback datasets. In: 2008 Eighth IEEE International Conference on Data Mining. IEEE, 2009: 263–272.[LinkOut]
    [19]
    Pan R, Scholz M. Mind the gaps: Weighting the unknown in large-scale one-class collaborative filtering. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2009: 667–676.[LinkOut]
    [20]
    Chen J, Wang C, Zhou S, et al. Fast adaptively weighted matrix factorization for recommendation with implicit feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34: 3470–3477. doi: 10.1609/aaai.v34i04.5751
    [21]
    Dupret G E, Piwowarski B. A user browsing model to predict search engine click data from past observations. Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. New York: ACM, 2008: 331–338.[LinkOut]
    [22]
    Zhang W V, Jones R. Comparing click logs and editorial labels for training query rewriting. In: WWW 2007 Workshop on Query Log Analysis: Social And Technological Challenges. Banff, Canada: ACM, 2007.
    [23]
    Craswell N, Zoeter O, Taylor M, et al. An experimental comparison of click position-bias models. Proceedings of the 2008 International Conference on Web Search and Data Mining. New York: ACM, 2008: 87–94.[LinkOut]
    [24]
    Guo F, Liu C, Kannan A, et al. Click chain model in web search. Proceedings of the 18th international conference on World wide web. New York: ACM, 2009: 11–20.[LinkOut]
    [25]
    Zhu Z A, Chen W, Minka T, et al. A novel click model and its applications to online advertising. Proceedings of the third ACM international conference on Web search and data mining. New York: ACM, 2010: 321–330.[LinkOut]
    [26]
    Kamishima T, Akaho S, Asoh H, et al. Correcting popularity bias by enhancing recommendation neutrality. In: RecSys '14: Proceedings of the 8th ACM Conference on Recommender Systems. Foster City, USA: ACM, 2014.
    [27]
    Zheng Y, Gao C, Li X, et al. Disentangling user interest and conformity for recommendation with causal embedding. Proceedings of the Web Conference 2021. New York: ACM, 2021: 2980–2991.[LinkOut]
    [28]
    Krishnan A, Sharma A, Sankar A, et al. An adversarial approach to improve long-tail performance in neural collaborative filtering. Proceedings of the 27th ACM International Conference on Information and Knowledge Management. New York: ACM, 2018: 1491–1494.[LinkOut]
    [29]
    He R, McAuley J. VBPR: Visual Bayesian personalized ranking from implicit feedback. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30: ■–■.[LinkOut]
    [30]
    Wu Y, DuBois C, Zheng A X, et al. Collaborative denoising auto-encoders for top-N recommender systems. Proceedings of the Ninth ACM International Conference on Web Search and Data Mining. New York: ACM, 2016: 153–162.[LinkOut]
    [31]
    He X, Deng K, Wang X, et al. LightGCN: Simplifying and powering graph convolution network for recommendation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2020: 639–648.[LinkOut]
    [32]
    He X, Du X, Wang X, et al. Outer product-based neural collaborative filtering. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2018: 2227–2233.[LinkOut]
    [33]
    Yu H F, Bilenko M, Lin C J. Selection of negative samples for one-class matrix factorization. Proceedings of the 2017 SIAM international conference on data mining. Philadelphia, PA: Society for Industrial and Applied Mathematics, 2017: 363–371.[LinkOut]
    [34]
    Park D H, Chang Y. Adversarial sampling and training for semi-supervised information retrieval. WWW '19: The World Wide Web Conference. New York: ACM, 2019: 1443–1453.[LinkOut]
    [35]
    Rendle S, Freudenthaler C. Improving pairwise learning for item recommendation from implicit feedback. Proceedings of the 7th ACM international conference on Web search and data mining. New York: ACM, 2014: 273–282.[LinkOut]
    [36]
    Ding J, Quan Y, He X, et al. Reinforced negative sampling for recommendation with exposure data. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization, 2019: 2230–2236.[LinkOut]
    [37]
    Ding J, Feng F, He X, et al. An improved sampler for Bayesian personalized ranking by leveraging view data. Proceedings of the The Web Conference 2018. New York: ACM, 2018: 13–14.[LinkOut]
    [38]
    Chen J, Wang C, Zhou S, et al. SamWalker: Social recommendation with informative sampling strategy. WWW '19: The World Wide Web Conference. New York: ACM, 2019: 228–239.[LinkOut]
    [39]
    Haussler D. Probably approximately correct learning. University of California, Santa Cruz, Computer Research Laboratory, 1990.
    [40]
    Sun W, Khenissi S, Nasraoui O, et al. Debiasing the human-recommender system feedback loop in collaborative filtering. Proceedings of The 2019 World Wide Web Conference. New York: ACM, 2019: 645–651.[LinkOut]
    [41]
    Gleich D F, Lim L H. Rank aggregation via nuclear norm minimization. Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2011: 60–68.[LinkOut]
    [42]
    Koren Y, Bell R, Volinsky C. Matrix factorization techniques for recommender systems. Computer, 2009, 42: 30–37.
    [43]
    Wang X, Zhang R, Sun Y and Qi J. Doubly robust joint learning for recommendation on data missing not at random. In: ICML '19: Proceedings of the 36th International Conference on Machine Learning. Long Beach, USA: PMLR, 2019: 6638–6647.
    [44]
    Ai Q, Bi K, Luo C, et al. Unbiased learning to rank with unbiased propensity estimation. SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. New York: ACM, 2018: 385–394.[LinkOut]
    [45]
    Ovaisi Z, Ahsan R, Zhang Y, et al. Correcting for selection bias in learning-to-rank systems. Proceedings of The Web Conference 2020. New York: ACM, 2020: 1863–1873.[LinkOut]
    Recommended articles
    TrendMD

    Article Metrics

    Article views (295) PDF downloads(1712)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return