Polynomial Regression

TL;DR

Models relationships between variables that are not well described by a straight line by fitting a polynomial curve.
Choose the polynomial degree (e.g., linear, quadratic, cubic, or higher-order) by comparing model performance, often via cross-validation.
Can be fitted with the same techniques as linear regression (e.g., gradient descent) but is prone to overfitting; regularization and cross-validation help mitigate that.

Definition

Polynomial regression is a type of regression analysis in which the relationship between the independent variable x and the dependent variable y is modeled as an nth degree polynomial.

Explanation

Polynomial regression models replace the straight-line relationship of linear regression with a polynomial equation of chosen degree. The approach can be implemented using the same fitting techniques as linear regression; for example, coefficients can be estimated with gradient descent. Common types include linear (degree 1), quadratic (degree 2), cubic (degree 3), and higher order polynomial regression (degree higher than 3). Selecting the degree that best fits the data is typically done by comparing the performance of different polynomial models using techniques such as cross-validation. While polynomial regression can provide more accurate predictions than linear regression for non-linear relationships, high-degree polynomials can lead to overfitting, so regularization and validation are important to prevent fitting noise in the data.

Examples

Altitude and temperature

Consider predicting the temperature of a city based on the altitude of the city. Plotting altitude and temperature for different cities may produce a curve rather than a straight line. A linear regression model may not accurately predict temperature from altitude in this case, whereas a polynomial regression model can fit a curve to the data and provide more accurate predictions.

House price and size

When predicting the price of a house based on its size, a linear regression model can be used, but plotting size versus price for many houses may reveal a non-linear relationship. A polynomial regression model can fit a curve to that data and provide more accurate predictions.

Use cases

Modeling relationships between variables that are not linear.

Notes or pitfalls

Polynomial regression models can be prone to overfitting, especially when the polynomial degree is high.
To avoid overfitting, select the polynomial degree using techniques such as cross-validation and apply regularization to prevent fitting the noise in the data.

Linear regression
Cross-validation
Gradient descent
Regularization
Quadratic regression
Cubic regression