ISSN 0253-2778

CN 34-1054/N

open
Free to Publish. Free to Read.
Open AccessOpen Access JUSTC Original Paper

An incremental cost-sensitive support vector machine

  • 1.

    Jiangsu Engineering Center of Network Monitoring, Nanjing 210044, China

  • 2.

    College of Computer Science & Software, Nanjing University of Information Science & Technology, Nanjing 210044, China

Cite this:
https://doi.org/10.3969/j.issn.0253-2778.2016.09.003
  • Received Date: 01 March 2016
  • Accepted Date: 17 September 2016
  • Rev Recd Date: 17 September 2016
  • Publish Date: 30 September 2016
    Abstract

  • Abstract

    Cost-sensitive learning is an important field in machine learning, which widely exists in real-world applications, such as cancer diagnosis, credit application, etc. Cost-sensitive support vector machine proposed by Masnadi et al. handles cost-sensitive problems through making the hinge loss function cost-sensitive, which has better generalization accuracy than other traditional cost-sensitive algorithms. In practice data are obtained one batch after another. Conventional batch algorithms would waste a lot of time when appending samples, because they should re-train the model from scratch. To make the cost-sensitive support vector machine more practical in on-line learning problems, an incremental cost-sensitive support vector machine algorithm was proposed, which can directly update the trained model without re-training it from scratch when appending samples. Experiment study on several datasets show that our algorithm is significantly more efficient than batch algorithms of the cost-sensitive support vector machine.

    Abstract

    Cost-sensitive learning is an important field in machine learning, which widely exists in real-world applications, such as cancer diagnosis, credit application, etc. Cost-sensitive support vector machine proposed by Masnadi et al. handles cost-sensitive problems through making the hinge loss function cost-sensitive, which has better generalization accuracy than other traditional cost-sensitive algorithms. In practice data are obtained one batch after another. Conventional batch algorithms would waste a lot of time when appending samples, because they should re-train the model from scratch. To make the cost-sensitive support vector machine more practical in on-line learning problems, an incremental cost-sensitive support vector machine algorithm was proposed, which can directly update the trained model without re-training it from scratch when appending samples. Experiment study on several datasets show that our algorithm is significantly more efficient than batch algorithms of the cost-sensitive support vector machine.
    • FullText(HTML)
  • loading
    • References(30)
  • [1]
    SHENG V S, LING C X. Thresholding for making classifiers cost-sensitive[C]// Proceedings of the 21st National Conference on Artificial Intelligence. Boston: AAAI Press, 2006: 476.
    [2]
    PARK Y J, CHUN S H, KIM B C. Cost-sensitive case-based reasoning using a genetic algorithm: Application to medical diagnosis[J]. Artificial Intelligence in Medicine, 2011, 51(2): 133-145.
    [3]
    ELKAN C. The foundations of cost-sensitive learning[C]// Proceedings of the 17th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers, 2001: 973-978.
    [4]
    DOMINGOS P. Metacost: A general method for making classifiers cost-sensitive[C]// Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Lisbon: ACM Press, 1999: 155-164.
    [5]
    CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: Synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2011, 16(1): 321-357.
    [6]
    DRUMMOND C, HOLTE R C. C4. 5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling[C]// Proceedings of the LCML Workshop on Learning from Imbalanced Datasets II. 2003: 1-8.
    [7]
    MARGINEANTU D D, DIETTERICH T G. Bootstrap methods for the cost-sensitive evaluation of classifiers[R]. Corvallis, OR: Oregon State University, 2000.
    [8]
    TING K M. A comparative study of cost-sensitive boosting algorithms[C]// Proceedings of the 17th International Conference on Machine Learning. San Francisco: Morgan Kaufmann, 2000: 983-990.
    [9]
    TING K M. An instance-weighting method to induce cost-sensitive trees[J]. IEEE Transactions on Knowledge and Data Engineering, 2002, 14(3): 659-665.
    [10]
    ZHOU Z H, LIU X Y. Training cost-sensitive neural networks with methods addressing the class imbalance problem[J]. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(1): 63-77.
    [11]
    LIN Y, LEE Y, WAHBA G. Support vector machines for classification in nonstandard situations[J]. Machine Learning, 2002, 46(1): 191-202.
    [12]
    GEIBEL P, BREFELD U, WYSOTZKI F. Perceptron and SVM learning with generalized cost models[J]. Intelligent Data Analysis, 2004, 8(5): 439-455.
    [13]
    SCHLKOPF B, SMOLA A J. Learning with Kernels: support Vector Machines, Regularization, Optimization, and Beyond[M].Cambridge:MIT Press, 2001.
    [14]
    MASNADI-SHIRAZI H, VASCONCELOS N. Risk minimization, probability elicitation, and cost-sensitive SVMs[C]// Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel: IEEE Press, 2010: 759-766.
    [15]
    MASNADI-SHIRAZI H, VASCONCELOS N, IRANMEHR A. Cost-Sensitive Support Vector Machines[J]. arXiv preprint, 2012: arXiv:1212.0975.
    [16]
    CORTES C, VAPNIK V. Support-vector networks[J]. Machine learning, 1995, 20(3): 273-297.
    [17]
    BACH F R, HECKERMAN D, HORVITZ E. Considering cost asymmetry in learning classifiers[J]. Journal of Machine Learning Research, 2006, 7(4): 1713-1741.
    [18]
    DAVENPORT M, BARANIUK R G, SCOTT C D. Controlling false alarms with support vector machines[C]// Proceedings of the International Conference on Acoustics, Speech and Signal Processing. IEEE Press, 2006: 589-592.
    [19]
    CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology, 2011, 2(3): 389-396.
    [20]
    WANG J, ZHAO P, HOI S C. Cost-sensitive online classification[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(10): 2425-2438.
    [21]
    ZHENG J, SHEN F, FAN H, et al. An online incremental learning support vector machine for large-scale data[J]. Neural Computing and Applications, 2013, 22(5): 1023-1035.
    [22]
    PLATT J C. Fast training of support vector machines using sequential minimal optimization[C]// Advances in Kernel Methods. Cambridge, USA: MIT Press, 1999: 185-208.
    [23]
    POGGIO G C T. Incremental and decremental support vector machine learning[C]// Proceedings of the 2000 Conference on Advances in Neural Information Processing Systems, MIT Press, 2001: 409.
    [24]
    GU B, WANG J D, CHEN H. On-line off-line ranking support vector machine and analysis[C]// IEEE International Joint Conference on Neural Networks. 2008: 1364-1369.
    [25]
    GU B, WANG J D, YU Y C, et al. Accurate on-line ν-support vector learning[J]. Neural Networks, 2012, 27: 51-59.
    [26]
    GU B, SHENG V S, WANG Z, et al. Incremental learning for ν-support vector regression[J]. Neural Networks, 2015, 67(C): 140-150.
    [27]
    GU B, SHENG V S, TAY K Y, et al. Incremental support vector learning for ordinal regression[J]. Neural Networks and Learning Systems, IEEE Transactions on, 2015, 26(7): 1403-1416.
    [28]
    SCHOTT J R. Matrix analysis for statistics[J]. Exvi Print, 2005, 30(5): xvi,456.
    [29]
    GU B, SHENG V S. Feasibility and finite convergence analysis for accurate on-line-support vector machine[J]. IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(8): 1304-1315.
    [30]
    CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2011, 2(3): 389-396.)
    • Supplements(0)
    • Related Articles
    • Track Citations
  • 加载中

Catalog

    |
    Get Citation
    PDF
    XML
    [1]
    SHENG V S, LING C X. Thresholding for making classifiers cost-sensitive[C]// Proceedings of the 21st National Conference on Artificial Intelligence. Boston: AAAI Press, 2006: 476.
    [2]
    PARK Y J, CHUN S H, KIM B C. Cost-sensitive case-based reasoning using a genetic algorithm: Application to medical diagnosis[J]. Artificial Intelligence in Medicine, 2011, 51(2): 133-145.
    [3]
    ELKAN C. The foundations of cost-sensitive learning[C]// Proceedings of the 17th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers, 2001: 973-978.
    [4]
    DOMINGOS P. Metacost: A general method for making classifiers cost-sensitive[C]// Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Lisbon: ACM Press, 1999: 155-164.
    [5]
    CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: Synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2011, 16(1): 321-357.
    [6]
    DRUMMOND C, HOLTE R C. C4. 5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling[C]// Proceedings of the LCML Workshop on Learning from Imbalanced Datasets II. 2003: 1-8.
    [7]
    MARGINEANTU D D, DIETTERICH T G. Bootstrap methods for the cost-sensitive evaluation of classifiers[R]. Corvallis, OR: Oregon State University, 2000.
    [8]
    TING K M. A comparative study of cost-sensitive boosting algorithms[C]// Proceedings of the 17th International Conference on Machine Learning. San Francisco: Morgan Kaufmann, 2000: 983-990.
    [9]
    TING K M. An instance-weighting method to induce cost-sensitive trees[J]. IEEE Transactions on Knowledge and Data Engineering, 2002, 14(3): 659-665.
    [10]
    ZHOU Z H, LIU X Y. Training cost-sensitive neural networks with methods addressing the class imbalance problem[J]. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(1): 63-77.
    [11]
    LIN Y, LEE Y, WAHBA G. Support vector machines for classification in nonstandard situations[J]. Machine Learning, 2002, 46(1): 191-202.
    [12]
    GEIBEL P, BREFELD U, WYSOTZKI F. Perceptron and SVM learning with generalized cost models[J]. Intelligent Data Analysis, 2004, 8(5): 439-455.
    [13]
    SCHLKOPF B, SMOLA A J. Learning with Kernels: support Vector Machines, Regularization, Optimization, and Beyond[M].Cambridge:MIT Press, 2001.
    [14]
    MASNADI-SHIRAZI H, VASCONCELOS N. Risk minimization, probability elicitation, and cost-sensitive SVMs[C]// Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel: IEEE Press, 2010: 759-766.
    [15]
    MASNADI-SHIRAZI H, VASCONCELOS N, IRANMEHR A. Cost-Sensitive Support Vector Machines[J]. arXiv preprint, 2012: arXiv:1212.0975.
    [16]
    CORTES C, VAPNIK V. Support-vector networks[J]. Machine learning, 1995, 20(3): 273-297.
    [17]
    BACH F R, HECKERMAN D, HORVITZ E. Considering cost asymmetry in learning classifiers[J]. Journal of Machine Learning Research, 2006, 7(4): 1713-1741.
    [18]
    DAVENPORT M, BARANIUK R G, SCOTT C D. Controlling false alarms with support vector machines[C]// Proceedings of the International Conference on Acoustics, Speech and Signal Processing. IEEE Press, 2006: 589-592.
    [19]
    CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology, 2011, 2(3): 389-396.
    [20]
    WANG J, ZHAO P, HOI S C. Cost-sensitive online classification[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(10): 2425-2438.
    [21]
    ZHENG J, SHEN F, FAN H, et al. An online incremental learning support vector machine for large-scale data[J]. Neural Computing and Applications, 2013, 22(5): 1023-1035.
    [22]
    PLATT J C. Fast training of support vector machines using sequential minimal optimization[C]// Advances in Kernel Methods. Cambridge, USA: MIT Press, 1999: 185-208.
    [23]
    POGGIO G C T. Incremental and decremental support vector machine learning[C]// Proceedings of the 2000 Conference on Advances in Neural Information Processing Systems, MIT Press, 2001: 409.
    [24]
    GU B, WANG J D, CHEN H. On-line off-line ranking support vector machine and analysis[C]// IEEE International Joint Conference on Neural Networks. 2008: 1364-1369.
    [25]
    GU B, WANG J D, YU Y C, et al. Accurate on-line ν-support vector learning[J]. Neural Networks, 2012, 27: 51-59.
    [26]
    GU B, SHENG V S, WANG Z, et al. Incremental learning for ν-support vector regression[J]. Neural Networks, 2015, 67(C): 140-150.
    [27]
    GU B, SHENG V S, TAY K Y, et al. Incremental support vector learning for ordinal regression[J]. Neural Networks and Learning Systems, IEEE Transactions on, 2015, 26(7): 1403-1416.
    [28]
    SCHOTT J R. Matrix analysis for statistics[J]. Exvi Print, 2005, 30(5): xvi,456.
    [29]
    GU B, SHENG V S. Feasibility and finite convergence analysis for accurate on-line-support vector machine[J]. IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(8): 1304-1315.
    [30]
    CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2011, 2(3): 389-396.)
    Recommended articles
    TrendMD
    Volume 46 Issue 9 page: 727-735
    Cover

    Article Metrics

    Article views (32) PDF downloads(86)
    Proportional views

    Copyright © Editorial Office of JUSTC, All Rights Reserved. 皖ICP备05002528号

    Supported by: Beijing Renhe Information Technology Co. Ltd    

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return