Logistic Regression

It's a common point of confusion that one of the most widely used models for classification has the word 'regression' in its name. This is because although we might be interested in predicting a binary (0-or-1) label, logistic regression actually predicts the probability that a label takes a specific value, meaning that we're actually predicting a continuous variable which can take values between 0 and 1.

As with linear regression, logistic regression models are interpretable and use a weighted sum of the features to predict the label. However, to ensure that predictions are between 0 and 1, logistic regression applies a non-linear function to the weighted sum; a consequence of this is that a closed-form solution doesn't exist in the way that it does for linear regression, meaning that we need to find a different way of estimating the model parameters. The most common approach is known as Iteratively Reweighted Least Squares, which you can implement in the notebooks below.

Online resources

• A cheatsheet on logistic regression by Chris Piech of Stanford;
• Section 4.4 of Hastie et al.'s Elements of Statistical Learning for a full derivation of the IRLS algorithm;
• A youtube video on the derivation of the IRLS algorithm.

Click the links below to access the Jupyter Notebooks for logistic regression

• Logistic Regression - Empty [Online notebook | .ipynb file]
• Logistic Regression - Redacted [Online notebook | .ipynb file]
• Logistic Regression - Complete [Online notebook | .ipynb file | HTML file]