News / Blog: Knowledge Base

How does the least squares method of estimating regression coefficients work?

The least squares method is a statistical technique for estimating regression coefficients in a linear regression. The goal is to find the line that minimizes the sum of squared vertical distances (residuals) between the observed dependent variable values and the values predicted by the regression line.

Here is a step-by-step explanation of the least squares procedure:

Collect data: collect data on the dependent (y) and independent (x) variables. Each data point consists of a pair (x, y).

Model specification: choose a linear regression model that describes the relationship between x and y. The model is of the form y = β0 + β1x + ɛ, where β0 and β1 are the regression coefficients to be estimated and ɛ is the error term.

Calculating the predictions: calculate the predicted values ŷ for each data point by substituting the regression equation with the estimated coefficients β0 and β1.

Calculate the residuals: calculate the difference between the observed y-values and the predicted ŷ-values. The residuals are represented as e = y - ŷ.

Calculate sum of squares of residuals: square each residual value and sum the squared residuals to obtain the sum of squared residuals (RSS): RSS = Σ(e²).

Estimating the coefficients: Estimate the regression coefficients β0 and β1 by minimizing the RSS. The estimates can be found using mathematical formulas or optimization algorithms such as the so-called "normal equation" or the "gradient descent" method.

Model evaluation: evaluate the goodness of the model by calculating statistical measures such as the coefficient of determination (R²) or the standard error of the estimate. These measures indicate how well the regression line fits the data and how good the predictions are.

Least squares is a widely used method for estimating regression coefficients because it yields the coefficients that are closest to the residuals and thus provide the best fit to the underlying data structure.