Alex Baretta

Friedman's stochastic gradient boosting machine (S-GBM) is a powerful machine learning algorithm that models the training dataset through an ensemble of decision trees. Like deep learning, S-GBM is a universal function approximator. Unlike deep learning, where the network training process involves gradient descent on the neural network's parameters, S-GBM relies on a gradient descent in "functional space." Every step of the algorithm constructs a piecewise function in the form of a decision tree, whose addition to the model maximizes the reduction in training loss.

This approach has several advantages over deep learning. In particular, it is possible to construct a cross-validation path, showing the tradeoff between variance and bias as a function of the number of trees in the ensemble. Early termination of the algorithm based on the cross-validation path obviates the need to decide the algorithm's hyperparameters ahead of time, in contrast with deep learning, where the network topology is fixed. S-GBM works remarkably well for structured data with a large number of categorical or ordinal, but not necessarily metric, variables.

Dawg is an efficient implementation of S-GBM in OCaml. I worked on it from 2016 to 2017.