Health Research and Policy

Workshop in Biostatistics - Abstract

DATE: May 1, 2014
TIME: 1:15 - 3:00 pm
LOCATION: Medical School Office Building, Rm x303
TITLE: Multiclass Support Vector Machines
SPEAKER: Patrick J.F. Groenen
Professor of Statistics, Econometric Institute, Erasmus School of Economics, Erasmus University Rotterdam, The Netherlands

In biology, there are many situations where a binary outcome needs to be predicted from a set of predictors. An obvious example is the prediction of presence or absence of a certain illness. For such a binary classification problem, support vector machines (SVMs) have become increasingly popular over the last ten years. Their out-of-sample prediction performance often compares favorable to alternative binary classification methods such as logistic regression and linear discriminant analysis. One of the reasons for this is that the SVM is robust against outliers, allows nonlinear prediction, and avoids overfitting by regularizing the loss function using a quadratic penalty. However, in the case that more than two classes need to be predicted often a series of binary SVMs are performed (one-versus-all or between all pairs of classes, one-versus-one). A disadvantage of such methods is that they are heuristics that do not simultaneously estimate all parameters in a single model.

In this presentation, we discuss a new multiclass SVM loss function that is based on a geometric representation of each class by a vertex of a simplex. As with the binary SVM, an object that is predicted to be nearest to its class receives a zero error and if the object is closer to another class the error consists of a function of the distance to the zero-error region. The present approach is flexible in the hinge function that is used for calculating the error. It builds on the Huberized hinge errors that have as special cases the linear and quadratic hinges. It is also flexible in how these errors are added: we propose to use the $L_p$ norm of the Huberized hinge error. This general loss function has some existing multiclass SVM loss functions as special cases.

We will start the presentation with an introduction to the binary support vector machine, discuss the use of hinge functions, penalties, and nonlinear predictions. Then, our extension to the multiclass situation is given. We also discuss how well the method works for several know data sets. It turns out that our method is competitive with other multiclass classification methods approaches in terms of speed and out-of-sample predictions.

Suggested readings:
Groenen, P.J.F., Nalbantov, G., Bioch, J.C. (2008). SVM-Maj: a majorization approach to linear support vector machines with different hinge errors. Advances in Data Analysis and Classification, 2, 17-43. DOI: 10.1007/s11634-008-0020-9

Footer Links: