12 - Ridge-SVM learning models

Summary

Introduction

It is well known that a classifier's effectiveness depends strongly on the distribution of the (training and testing) datasets. Consequently, we will not know in advance the best possible classifiers for data analysis. This prompts the need to develop a versatile classifier endowed with an adequate set of adjustable parameters to cope with various real-world application scenarios. Two common ways to enhance the robustness of the classifiers are by means of (1) using a proper ridge factor to mitigate over-fitting problems (as adopted by KRR) and/or (2) selecting an appropriate number of support vectors to participate in the decision making (as adopted by SVM). Both regularization mechanisms are meant to enhance the robustness of the learned models and, ultimately, improve the generalization performance.

This chapter introduces the notion of a weight–error curve (WEC) for characterization of kernelized supervised learning models, including KDA, KRR, SVM, and Ridge-SVM. Under the LSP condition, the decision vector can be “voted” as a weighted sum of training vectors in the intrinsic space – each vector is assigned a weight in voting. The weights can be obtained by solving the kernelized learning model. In addition, each vector is also associated with an error, which is dictated by its distance from the decision boundary. In short, given a learned model, each training vector is endowed with two parameters: weight and error. These parameters collectively form the so-called WEC. The analysis of the WEC leads us to a new type of classifier named Ridge-SVM.