3rd Workshop on Ensemble Methods

This workshop is jointly organised by the working groups

• Computational Statistics (DR-IBG),
• Statistical Computing (GMDS) and the
• Austrian Association for Statistical Computing.

and is supported by

• SFB 386 Statistical Analysis of Discrete Structures

Organisers

• Axel Benner, DKFZ Heidelberg
• Torsten Hothorn, Universität Erlangen-Nürnberg
• Gunnar Rätsch, Friedrich Miescher Laboratory of the Max Planck Society
• Gerhard Tutz, LMU München

Local Organiser

• Gerhard Tutz, LMU München

Aims of the Workshop

Based on the success of the Ensemble Workshops 2004 and 2005, the workshop aims at discussing further theoretical and practical developments for prediction models such as boosting, random forests or support vector machines. The workshop tries to bring together the machine learning view with the statistical view on ensemble methods.

Schedule

Talks will be given in German or English (speaker's choice), a LCD projector as well as a laptop for PDF or Powerpoint presentations is available in the lecture room.

The workshop starts on Friday, 17.2 at 14.00 and ends on Saturday, 18.2 at 12.50.

Abstracts

• Karin T. Eckel, Annette Pfahlberg, Olaf Gefeller and Torsten Hothorn, Erlangen: Flexible Modeling of Malignant Melanoma Survival

  Abstract: In this talk we investigate diagnostic and predictive capabilities of flexible ensemble methods modeling the survival time of patients suffering from malignant melanoma (Kölmel et al., 2005) in comparison to the well established proportional hazards model. Based on a bagging algorithm for censored data (Hothorn et al., 2004), a random forest type algorithm as well as a combination of the proportional hazards model with bagging is suggested. Benchmark experiments are used to compare the performance of the competing algorithms. We apply the integrated Brier score (Graf et al., 1999) as an adequate measure for the goodness of prediction. For the purpose of comparing the regression relationships represented by different models, we define a specific univariate measure for the estimated survival times: Kaplan-Meier estimates for each patient are described by the area under the curve (AUC). Using this AUC we introduce slightly modified CARTscans (Nason et al., 2004) to depict the regression relationships. This graphical tool is able to display predicted values across a four-dimensional subspace and thus we are able to show risk prediction in respect to four covariates.

• Johannes Fürnkranz, Darmstadt: Ensembles of Pairwise Classifiers

  Abstract: Conventional inductive machine learning techniques generalize a training set of pre-classified examples into a concept description that is able to classify unseen instances from the same domain. Ensemble techniques are able to increase the classification accuracy by learning multiple concept descriptions from a single dataset and combining their individual predictions. The most prominent examples for such techniques are bagging and boosting. However, there are alternative ways of forming an ensemble that depend on the problem structure. Examples from our work include hyperlink ensembles for hypertext categorization and ensembles of binary classifiers that are used for solving a multi-class problem. In this talk, I will review our work on ensembles of pairwise classifiers. I will show empirically that pairwise classification performs well for rule learning and decision tree learning algorithms, as well as show theoretically that it scales well. We will also illustrate a straight-forward way for combining it with bagging. Finally, we will move from pairwise classification to the more general scenario of pairwise preference learning, a learning setting that includes several other learning scenarios as special cases.

• Torsten Hothorn and Peter Bühlmann, Erlangen/Zürich: Boosting mit komponentenweisen Kleinste-Quadrate Methoden: Software und Beispiele.

  Abstract: Boosting mit komponentenweise linearen Modellen und smooting splines wurde von Bühlmann und Yu (2003) und Bühlmann (2006) vorgeschlagen und untersucht. Diese Verfahren, insbesondere die Benutzung von einfachen linearen Modellen als `base learner', führen zu einer einfachen und effektiven Form von Regularisierung und sind insbesondere auch rechentechnisch interessant.

  Wir geben einen Einblick in aktuelle Entwicklungsarbeiten zur Implementation von verschiedenen Boosting-Varianten in einem R-Paket (Arbeitstitel `gboost'). Anhand von Beispielen zur Klassifikation und Regression mit unter Umständen zensierten Daten, insbesondere in hochdimensionalen Problemen, werden die Grundzüge der bisher implementierten Methoden illustriert.

• Inke R. König and Andreas Ziegler, Lübeck: Practical experiences in validating a prognostic model

  Abstract: In developing a prognostic model, data dependent methods are usually utilized to optimize the fit in the data at hand. For example, using support vector machines, suitable function parameters for the kernel functions need to be selected. This data dependent optimization increases the risk of overfitting which is even increased with small sample sizes. To estimate the extent of overfitting, a prognostic model often is internally validated. Here, the same data set is artificially split into separate samples in which model development and testing can be performed. By contrast, a stringent model validation requires at least two independent data sets. Depending on the second data set, temporal and external validation are distinguished: temporal validation is based on data points that have been collected at the same centers but at a later time point, external validation uses data points from different centers. In the presentation, different techniques for validating a prognostic model are described and discussed. Prognostic models that were developed using a number of ensemble methods are compared with regard to differences in temporal and external validation. For this, exemplary data sets to predict functional independence in stroke patients are utilized.

• Nicole Krämer, Berlin: Boosting for Functional Data

  Abstract: We deal with the task of supervised learning if the data is of functional type, that is, the variables that we observe are (discretized) curves. In this talk, we explain how to extend Boosting methods to problems that involve functional data. This can be done by expanding each observed function in terms of appropriate basis functions. Hot favorites are B-splines, Fourier expansions, wavelets or functions that are derived from the data (e.g. by applying functional principal component analysis). We can then apply any Boosting algorithm to this finite-dimensional representation of the data. During the talk, we discuss the merits of this functional approach.

• Björn H. Menze, Heidelberg: Klassifikation spektraler Daten - feature selection durch randomForests

  Abstract: Spektren sind hochkorrelierte Daten und enthalten viel redundante Information. Für Anwendungen aus biomedizinischen Umfeld gilt zudem haeufig P>> N. Anhand einiger Beispiele sollen typische Probleme in der Entwicklung anwendungsorientierter Klassifikationssysteme spektraler Daten diskutiert werden. Die Daten entstammen der in vivo Magnetresonanz- und Infrarot-Spektroskopie, sowie der Massen-Spektrometrie und representieren Anwendungen zur Hirn-Tumor Detektion, der Entwicklung veterinaerer Bluttests und der Analyse von Bakterien-Lysaten.

  Breimans randomForest wird als wichtige Methode der feature selection vorgestellt und quantitativ mit vergleichbaren Standardverfahren verglichen. Abschliessend werden (kurz) randomForest Varianten aufgezeigt, die auf spektralen Daten Vorteile gegenueber der originalen Implementierung aufweisen.

• Joaquin Quiñonero Candela, TU Berlin: Ensemble Methods and Probabilistic Predictions

  Abstract: The Evaluating Predictive Uncertainty Challenge was set up to evaluate the ability of Machine Learning algorithms to provide good ``probabilistic predictions'', rather than just the usual ``point predictions'' with no measure of uncertainty, in regression and classification problems. The Challenge did run in November and December 2004, and its website remains open for submissions: http://predict.kyb.tuebingen.mpg.de/. Participants had to compete on a number of regression and classification tasks, and were evaluated by both traditional losses that only take into account point predictions and losses we proposed that evaluate the quality of the probabilistic predictions. The participants employed a large variety of methods, ranging from Neural Networks to Decision Trees, from Support Vector Machines to Gaussian Processes, and many others. Interestingly for this Workshop, Ensemble Methods seemed to consistently give the best results, seemingly irrespective of the underlying base model. Bagging, Bayesian averaging and Boosting are represented. I will present the Challenge, analyze the results, and provide some general discussion.

• Gunnar Rätsch, Manfred Warmuth and Jun Liao, Tübingen: Totally Corrective Boosting Algorithms that Maximize the Margin

  Abstract: We consider boosting algorithms that maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy to the last distribution subject to linear constraints. For example AdaBoost constrains the edge of the last hypothesis w.r.t. the updated distribution to be at most \gamma=0 (equivalently its weighted error is constrained to be half). In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A more principled Boosting method is to be "totally corrective" in that the edges of all the past hypotheses are constrained to be at most \gamma, where \gamma is suitably adapted.

  Using new techniques, we prove the same iteration bounds for the totally corrective algorithms as for their corrective versions. Moreover with adaptive \gamma, the algorithms can be shown to provably maximize the margin. Experimentally, the totally corrective versions clearly outperform the corrective ones and are competitive with the most efficient algorithm so far, LPBoost, for which there are no iteration bounds known.

• Stefan Rüping, St. Augustin: On the combination of accurate and understandable models

  Abstract: Next to prediction accuracy, interpretability is a critical goal in a knowledge discovery process (Giraud-Carrier, 1998). An understandable model allows assess the reliability of the learner's predictions far better than by simple statistical error measure and enables the user to discover modeling errors like biased sampling that are otherwise hard to detect. However, understandability is hard to achieve, as psychological research has long since established the fact that the human capability of dealing with complex dependencies is astonishingly limited (e.g., (Miller, 1956)). While it is relatively easy to find an accurate and an understandable model independently, the combination of both goals is a challenging task, which cannot be solved by current ensemble methods. This talk will explore different ensemble strategies to solving both problems simultaneously.

• Carolin Strobl, München: Bias in Variable Importance Measures of Ensemble Methods based on Classification Trees

  Abstract: Ensemble methods can highly increase the predictive accuracy of classification trees, which usually serve as weak learners in ensemble methods for classification. Most ensemble methods also provide variable importance measures for variable selection purposes. However, these variable importance measures suffer from a serious flaw that is carried forward from the individual classification trees: in all standard classification tree algorithms variable selection is biased e.g. when the potential predictor variables vary in their number of categories. It is outlined and supported by simulation results how variable importance measures are affected by variable selection bias in random forests and boosting. A method is suggested to eliminate the effect on the variable importance measures by means of eliminating variable selection bias in individual classification trees.

• Martin Theus, Augsburg: TWIX – classification method

  Abstract: Tree methods are popular tools in classification. Tree ensembles - as used in methods like Bagging or Random Forests - try to overcome the variability of a single tree and can improve the accuracy of a single tree in most cases . Both of these methods rely on randomness to generate the tree ensembles. The TWIX algorithm uses second best splits to generate a tree ensemble. The resulting TWIX-trees usually outperform classical greedy trees and the aggregation of a TWIX- ensemble can improve the results of bagged trees. This talk will illustrate the TWIX-algorithm and discuss the challenges in tree selection. The underlying R-package is introduced along with several experimental diagnostic functions to illuminate the structure of tree ensembles.

• Jan Ulbricht, München: An Alternative Approach to Regularization and Variable Selection in Regression Modelling

  Abstract: In the context of linear regression models alternative shrinkage methods have been proposed which outperform the OLS estimate in terms of mean squared error and accuracy of prediction. Common regularization methods are penalized least squares with an appropriate chosen penalty term, such as the bridge penalty type (Frank & Friedmann, 1993). The lasso, as one special case of this penalty type, does continuous shrinkage but simultaneously also variable selection. The more recently proposed elastic net (Zou & Hastie, 2005) is based on a linear combination of the ridge penalty and the lasso using two tuning parameters. The elastic net has two distinct advantages: it does variable selection which makes it an adequate tool for high dimensional predictors and it includes highly correlated predictors rather than selecting one of them. In this talk we propose an alternative penalty term which shares these advantages and emphasizes the grouping effect which means that the regression coefficients of a group of highly correlated variables tend to be equal.

Conference Site

The conference will take place at the pink coloured building (Schweinchenbau), Leopoldstr.13, 80802 Munich. The talks will be given in room HS 3232, building 3. Public transport: U3 or U6 Giselastraße. On the the LMU Map you can find the building near spot 9 (click on it to get a VR tour). From the Institute of Statistics opposite spot 3 the conference venue can be reached in about 5 minutes by foot.

The workshop dinner will take place at Friday, 17.02., 19:30, at the "Alter Simpl", Türkenstr. 57, 80799 Munich.

Participation and Accommodation

Participants can attend the workshop free of charge, the number of participants is restricted to 30 persons. To register, please drop an email to Gerhard Tutz.

Preliminary list of participants

• Karin Eckel

• Johannes Fürnkranz

• Torsten Hothorn

• Nicole Kraemer

• Joaquin Quiñonero Candela

• Gunnar Raetsch

• Stefan Rüping

• Carolin Strobl

• Martin Theus

• Jan Ulbricht

• Donna Ankerst

• Thomas Augustin

• Anton Belooussov

• Harald Binder

• Bernhard Bornstein

• Anne-Laure Boulesteix

• Manuela Hummel

• Helmut Küchenhoff

• Stefan Lang

• Florian Leitenstorfer

• Florian Reithinger