Decision Trees

A desirable characteristic of regression models, where our aim is to predict a continuous target, is that if we are seeking to predict the target associated with two examples which have very similar features, then we would expect their predicted targets to be similar as well.

Decision Trees, which partition the input space up into subsets and then assign the same predicted value to every input in a given subset, do not guarantee that similar inputs will give similar outputs; crossing from one subset to another can lead to very different predictions. They do, however, have the ability to learn non-linear relationships between features, can handle datasets with a mixture of continuous and categorical features with different ranges, and perform well on datasets with a limited number of examples.

Serving as the building block for the widely used Random Forests and Gradient Boosting algorithms, it's well worth getting to know Decision Trees in detail.

Online resources

Click the links below to access the Jupyter Notebooks for Decision Trees