Finding the gradient of the best-fit line

Consider \(n\) data points \((x_i, y_i)\) along with a poor estimate line:

Assume that the estimate line has this formula:

\[ y = \beta x + \alpha \]

where \(\alpha\) is the bias and \(\beta\) is the gradient.

The best-fit line is the line which has the smallest sum of squared residual (SSR). A squared residual is \((y_i - \hat{y_i})^2\), where \(\hat{y_i}\) is the estimate from the line for \(x_i\). We already know that for any slope, the bias that gives the smallest SSR is:

\[ \alpha = \overline{y} - \beta \ \overline{x}\]

This means the best-fit line must pass through the mean point, whatever its slope turns out to be. So we can substitute this value of \(\alpha\) and look for the slope \(\beta\) that gives the smallest SSR among lines through the mean point.

The mean point in this example is at \((9.5,5.9)\). Our goal is the find the gradient that results in the smallest SSR.

Let the residual at \(i\) be \(e_i\). This means:

\[\begin{align} \sum_{i=1}^n e_i^2 &= \sum_{i=1}^n (y_i - \hat{y_i})^2 \\ &= \sum_{i=1}^n (y_i - (\beta x_i + \alpha))^2\end{align}\]

Substituting \(\alpha\):

\[\begin{align} \sum_{i=1}^n e_i^2 &= \sum_{i=1}^n (y_i - (\beta x_i + \overline{y} - \beta \ \overline{x}))^2 \\ &= \sum_{i=1}^n (y_i - \beta x_i - \overline{y} + \beta \ \overline{x})^2 \\ &= \sum_{i=1}^n ((y_i - \overline{y}) + \beta(\overline{x} - x_i))^2 \end{align}\]

Now let's differentiate with respect to \(\beta\):

\[\begin{align} \frac{d}{d \beta} \sum_{i=1}^n e_i^2 &= \frac{d}{d \beta} \sum_{i=1}^n ((y_i - \overline{y}) + \beta(\overline{x} - x_i))^2 \\ &= \sum_{i=1}^n 2 ((y_i - \overline{y}) + \beta(\overline{x} - x_i))(\overline{x} - x_i)\\ &= \sum_{i=1}^n 2((y_i - \overline{y})(\overline{x} - x_i) + \beta(\overline{x} - x_i)^2)\end{align}\]

Setting the gradient to 0:

\[\begin{align} 0 &= \sum_{i=1}^n 2((y_i - \overline{y})(\overline{x} - x_i) + \beta(\overline{x} - x_i)^2) \\ &= \sum_{i=1}^n ((y_i - \overline{y})(\overline{x} - x_i) + \beta(\overline{x} - x_i)^2) \\ &= \sum_{i=1}^n (y_i - \overline{y})(\overline{x} - x_i) + \beta \sum_{i=1}^n (\overline{x} - x_i)^2 \end{align}\]

Now lets make \(\beta\) the subject:

\[\begin{align} \beta &= \frac{-\sum_{i=1}^n (y_i - \overline{y})(\overline{x} - x_i) }{\sum_{i=1}^n (\overline{x} - x_i)^2} \\ \\ &= \frac{\sum_{i=1}^n (\overline{y} - y_i)(\overline{x} - x_i) }{\sum_{i=1}^n (\overline{x} - x_i)^2} \end{align}\]

This means that the slope for a best-fit line is given by the above formula. We did get this formula by setting the derivative to 0, but we also need to check whether this critical point is a minimum or a maximum. For that we need to differentiate SSR again:

\[\begin{align} \frac{d}{d \beta} \frac{d}{d \beta} \sum_{i=1}^n e_i^2 &= \frac{d}{d \beta}\sum_{i=1}^n 2((y_i - \overline{y})(\overline{x} - x_i) + \beta(\overline{x} - x_i)^2) \\ &= \frac{d}{d \beta}\sum_{i=1}^n 2(y_i - \overline{y})(\overline{x} - x_i) + \frac{d}{d \beta}\sum_{i=1}^n 2 \beta (\overline{x} - x_i)^2 \\ &= 0 + \sum_{i=1}^n 2 (\overline{x} - x_i)^2 \frac{d}{d \beta} \beta \\ &= 2 \sum_{i=1}^n (\overline{x} - x_i)^2 \end{align}\]

If all \(x_i\) are not equal, then \(\sum_{i=1}^n (\overline{x} - x_i)^2 \gt 0\). This means the double derivative is would be \(\gt 0\), which also means the formula we found for \(\beta\) does in fact give the minimum SSR.

If all \(x_i\) are equal, the data points lie on a vertical line, which has no slope, so no value of \(\beta\) can describe it. In that case the denominator of the formula is 0 and \(\beta\) is undefined.

For the example above, the mean is at \((9.5,5.9)\), and \(\beta \approx 0.275\).