The result of fitting a set of data points with a quadratic function

Conic fitting a set of points using least-squares approximation

The method of least squares is a parameter estimation method in regression analysis based on minimizing the sum of the squares of the residuals (a residual being the difference between an observed value and the fitted value provided by a model) made in the results of each individual equation.

The most important application is in data fitting. When the problem has substantial uncertainties in the independent variable (the x variable), then simple regression and least-squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares.

Least squares problems fall into two categories: linear or ordinary least squares and nonlinear least squares, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed-form solution. The nonlinear problem is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.

Polynomial least squares describes the variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve.

When the observations come from an exponential family with identity as its natural sufficient statistics and mild-conditions are satisfied (e.g. for normal, exponential, Poisson and binomial distributions), standardized least-squares estimates and maximum-likelihood estimates are identical.^[1] The method of least squares can also be derived as a method of moments estimator.

The following discussion is mostly presented in terms of linear functions but the use of least squares is valid and practical for more general families of functions. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information), the least-squares method may be used to fit a generalized linear model.

The least-squares method was officially discovered and published by Adrien-Marie Legendre (1805),^[2] though it is usually also co-credited to Carl Friedrich Gauss (1809),^[3][4] who contributed significant theoretical advances to the method,^[4] and may have also used it in his earlier work in 1794 and 1795.^[5][4]

History

The method

Carl Friedrich Gauss

The first clear and concise exposition of the method of least squares was published by Legendre in 1805.^[7] The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the Earth. Within ten years after Legendre's publication, the method of least squares had been adopted as a standard tool in astronomy and geodesy in France, Italy, and Prussia, which constitutes an extraordinarily rapid acceptance of a scientific technique.^[6]

In 1809 Carl Friedrich Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795.^[8] This naturally led to a priority dispute with Legendre. However, to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the normal distribution. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that the arithmetic mean is indeed the best estimate of the location parameter by changing both the probability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.

An early demonstration of the strength of Gauss's method came when it was used to predict the future location of the newly discovered asteroid Ceres. On 1 January 1801, the Italian astronomer Giuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the Sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the Sun without solving Kepler's complicated nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.

In 1810, after reading Gauss's work, Laplace, after proving the central limit theorem, used it to give a large sample justification for the method of least squares and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the Gauss–Markov theorem.

The idea of least-squares analysis was also independently formulated by the American Robert Adrain in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.^[9]

Problem statement

The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) ${\displaystyle (x_{i},y_{i})\!}$, i = 1, …, n, where ${\displaystyle x_{i}\!}$ is an independent variable and ${\displaystyle y_{i}\!}$ is a dependent variable whose value is found by observation. The model function has the form ${\displaystyle f(x,{\boldsymbol {\beta }})}$, where m adjustable parameters are held in the vector ${\displaystyle {\boldsymbol {\beta }}}$. The goal is to find the parameter values for the model that "best" fits the data. The fit of a model to a data point is measured by its residual, defined as the difference between the observed value of the dependent variable and the value predicted by the model:

${\displaystyle r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }}).}$

The residuals are plotted against corresponding ${\displaystyle x}$ values. The random fluctuations about ${\displaystyle r_{i}=0}$ indicate a linear model is appropriate.

The least-squares method finds the optimal parameter values by minimizing the sum of squared residuals, ${\displaystyle S}$:^[10]

${\displaystyle S=\sum _{i=1}^{n}r_{i}^{2}.}$

In the simplest case ${\displaystyle f(x_{i},{\boldsymbol {\beta }})=\beta }$ and the result of the least-squares method is the arithmetic mean of the input data.

An example of a model in two dimensions is that of the straight line. Denoting the y-intercept as ${\displaystyle \beta _{0}}$ and the slope as ${\displaystyle \beta _{1}}$, the model function is given by ${\displaystyle f(x,{\boldsymbol {\beta }})=\beta _{0}+\beta _{1}x}$. See linear least squares for a fully worked out example of this model.

A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, x and z, say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.

To the right is a residual plot illustrating random fluctuations about ${\displaystyle r_{i}=0}$, indicating that a linear model${\displaystyle (Y_{i}=\alpha +\beta x_{i}+U_{i})}$ is appropriate. ${\displaystyle U_{i}}$ is an independent, random variable.^[10]  

The residuals are plotted against the corresponding ${\displaystyle x}$ values. The parabolic shape of the fluctuations about ${\displaystyle r_{i}=0}$ indicates a parabolic model is appropriate.

If the residual points had some sort of a shape and were not randomly fluctuating, a linear model would not be appropriate. For example, if the residual plot had a parabolic shape as seen to the right, a parabolic model ${\displaystyle (Y_{i}=\alpha +\beta x_{i}+\gamma x_{i}^{2}+U_{i})}$ would be appropriate for the data. The residuals for a parabolic model can be calculated via ${\displaystyle r_{i}=y_{i}-{\hat {\alpha }}-{\hat {\beta }}x_{i}-{\widehat {\gamma }}x_{i}^{2}}$.^[10]

Limitations

This regression formulation considers only observational errors in the dependent variable (but the alternative total least squares regression can account for errors in both variables). There are two rather different contexts with different implications:

• Regression for prediction. Here a model is fitted to provide a prediction rule for application in a similar situation to which the data used for fitting apply. Here the dependent variables corresponding to such future application would be subject to the same types of observation error as those in the data used for fitting. It is therefore logically consistent to use the least-squares prediction rule for such data.
• Regression for fitting a "true relationship". In standard regression analysis that leads to fitting by least squares there is an implicit assumption that errors in the independent variable are zero or strictly controlled so as to be negligible. When errors in the independent variable are non-negligible, models of measurement error can be used; such methods can lead to parameter estimates, hypothesis testing and confidence intervals that take into account the presence of observation errors in the independent variables.^[11] An alternative approach is to fit a model by total least squares; this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model-fitting.

Solving the least squares problem

The minimum of the sum of squares is found by setting the gradient to zero. Since the model contains m parameters, there are m gradient equations:

$${\displaystyle {\frac {\partial S}{\partial \beta _{j}}}=2\sum _{i}r_{i}{\frac {\partial r_{i}}{\partial \beta _{j}}}=0,\ j=1,\ldots ,m,}$$

and since ${\displaystyle r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }})}$, the gradient equations become

$${\displaystyle -2\sum _{i}r_{i}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}=0,\ j=1,\ldots ,m.}$$

The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.^[12]

Linear least squares

Main article: Linear least squares

A regression model is a linear one when the model comprises a linear combination of the parameters, i.e.,

$${\displaystyle f(x,{\boldsymbol {\beta }})=\sum _{j=1}^{m}\beta _{j}\phi _{j}(x),}$$

where the function ${\displaystyle \phi _{j}}$ is a function of ${\displaystyle x}$.^[12]

Letting ${\displaystyle X_{ij}=\phi _{j}(x_{i})}$ and putting the independent and dependent variables in matrices ${\displaystyle X}$ and ${\displaystyle Y}$, respectively, we can compute the least squares in the following way. Note that ${\displaystyle D}$ is the set of all data.^[12][13]

$${\displaystyle L(D,{\boldsymbol {\beta }})=\left\|Y-X{\boldsymbol {\beta }}\right\|^{2}=(Y-X{\boldsymbol {\beta }})^{\mathsf {T}}(Y-X{\boldsymbol {\beta }})=Y^{\mathsf {T}}Y-Y^{\mathsf {T}}X{\boldsymbol {\beta }}-{\boldsymbol {\beta }}^{\mathsf {T}}X^{\mathsf {T}}Y+{\boldsymbol {\beta }}^{\mathsf {T}}X^{\mathsf {T}}X{\boldsymbol {\beta }}}$$

The gradient of the loss is:

$${\displaystyle {\frac {\partial L(D,{\boldsymbol {\beta }})}{\partial {\boldsymbol {\beta }}}}={\frac {\partial \left(Y^{\mathsf {T}}Y-Y^{\mathsf {T}}X{\boldsymbol {\beta }}-{\boldsymbol {\beta }}^{\mathsf {T}}X^{\mathsf {T}}Y+{\boldsymbol {\beta }}^{\mathsf {T}}X^{\mathsf {T}}X{\boldsymbol {\beta }}\right)}{\partial {\boldsymbol {\beta }}}}=-2X^{\mathsf {T}}Y+2X^{\mathsf {T}}X{\boldsymbol {\beta }}}$$

Setting the gradient of the loss to zero and solving for ${\displaystyle {\boldsymbol {\beta }}}$, we get:^[13][12]

$${\displaystyle -2X^{\mathsf {T}}Y+2X^{\mathsf {T}}X{\boldsymbol {\beta }}=0\Rightarrow X^{\mathsf {T}}Y=X^{\mathsf {T}}X{\boldsymbol {\beta }}}$$

$${\displaystyle {\boldsymbol {\hat {\beta }}}=\left(X^{\mathsf {T}}X\right)^{-1}X^{\mathsf {T}}Y}$$

Non-linear least squares

Main article: Non-linear least squares

There is, in some cases, a closed-form solution to a non-linear least squares problem – but in general there is not. In the case of no closed-form solution, numerical algorithms are used to find the value of the parameters ${\displaystyle \beta }$ that minimizes the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation:

$${\displaystyle {\beta _{j}}^{k+1}={\beta _{j}}^{k}+\Delta \beta _{j},}$$

where a superscript k is an iteration number, and the vector of increments ${\displaystyle \Delta \beta _{j}}$ is called the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order Taylor series expansion about ${\displaystyle {\boldsymbol {\beta }}^{k}}$:

$${\displaystyle {\begin{aligned}f(x_{i},{\boldsymbol {\beta }})&=f^{k}(x_{i},{\boldsymbol {\beta }})+\sum _{j}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}\left(\beta _{j}-{\beta _{j}}^{k}\right)\\&=f^{k}(x_{i},{\boldsymbol {\beta }})+\sum _{j}J_{ij}\,\Delta \beta _{j}.\end{aligned}}}$$

The Jacobian J is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. The residuals are given by

$${\displaystyle r_{i}=y_{i}-f^{k}(x_{i},{\boldsymbol {\beta }})-\sum _{k=1}^{m}J_{ik}\,\Delta \beta _{k}=\Delta y_{i}-\sum _{j=1}^{m}J_{ij}\,\Delta \beta _{j}.}$$

To minimize the sum of squares of ${\displaystyle r_{i}}$, the gradient equation is set to zero and solved for ${\displaystyle \Delta \beta _{j}}$:

$${\displaystyle -2\sum _{i=1}^{n}J_{ij}\left(\Delta y_{i}-\sum _{k=1}^{m}J_{ik}\,\Delta \beta _{k}\right)=0,}$$

which, on rearrangement, become m simultaneous linear equations, the normal equations:

$${\displaystyle \sum _{i=1}^{n}\sum _{k=1}^{m}J_{ij}J_{ik}\,\Delta \beta _{k}=\sum _{i=1}^{n}J_{ij}\,\Delta y_{i}\qquad (j=1,\ldots ,m).}$$

The normal equations are written in matrix notation as

$${\displaystyle \left(\mathbf {J} ^{\mathsf {T}}\mathbf {J} \right)\Delta {\boldsymbol {\beta }}=\mathbf {J} ^{\mathsf {T}}\Delta \mathbf {y} .}$$

These are the defining equations of the Gauss–Newton algorithm.

Example

Consider a simple example drawn from physics. A spring should obey Hooke's law which states that the extension of a spring y is proportional to the force, F, applied to it.

${\displaystyle y=f(F,k)=kF\!}$

constitutes the model, where F is the independent variable. In order to estimate the force constant, k, we conduct a series of n measurements with different forces to produce a set of data, ${\displaystyle (F_{i},y_{i}),\ i=1,\dots ,n\!}$, where y_i is a measured spring extension.^[14] Each experimental observation will contain some error, ${\displaystyle \varepsilon }$, and so we may specify an empirical model for our observations,

${\displaystyle y_{i}=kF_{i}+\varepsilon _{i}.\,}$

There are many methods we might use to estimate the unknown parameter k. Since the n equations in the m variables in our data comprise an overdetermined system with one unknown and n equations, we estimate k using least squares. The sum of squares to be minimized is

${\displaystyle S=\sum _{i=1}^{n}(y_{i}-kF_{i})^{2}.}$^[12]

The least squares estimate of the force constant, k, is given by

${\displaystyle {\hat {k}}={\frac {\sum _{i}F_{i}y_{i}}{\sum _{i}F_{i}^{2}}}.}$

We assume that applying force causes the spring to expand. After having derived the force constant by least squares fitting, we predict the extension from Hooke's law.

Uncertainty quantification

In a least squares calculation with unit weights, or in linear regression, the variance on the jth parameter, denoted ${\displaystyle \operatorname {var} ({\hat {\beta }}_{j})}$, is usually estimated with

${\displaystyle \operatorname {var} ({\hat {\beta }}_{j})=\sigma ^{2}\left(\left[X^{\mathsf {T}}X\right]^{-1}\right)_{jj}\approx {\hat {\sigma }}^{2}C_{jj},}$

${\displaystyle {\hat {\sigma }}^{2}\approx {\frac {S}{n-m}}}$

${\displaystyle C=\left(X^{\mathsf {T}}X\right)^{-1},}$

where the true error variance σ² is replaced by an estimate, the reduced chi-squared statistic, based on the minimized value of the residual sum of squares (objective function), S. The denominator, n − m, is the statistical degrees of freedom; see effective degrees of freedom for generalizations.^[12] C is the covariance matrix.

Statistical testing

If the probability distribution of the parameters is known or an asymptotic approximation is made, confidence limits can be found. Similarly, statistical tests on the residuals can be conducted if the probability distribution of the residuals is known or assumed. We can derive the probability distribution of any linear combination of the dependent variables if the probability distribution of experimental errors is known or assumed. Inferring is easy when assuming that the errors follow a normal distribution, consequently implying that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.^[12]

It is necessary to make assumptions about the nature of the experimental errors to test the results statistically. A common assumption is that the errors belong to a normal distribution. The central limit theorem supports the idea that this is a good approximation in many cases.

• The Gauss–Markov theorem. In a linear model in which the errors have expectation zero conditional on the independent variables, are uncorrelated and have equal variances, the best linear unbiased estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.^[15]
• If the errors belong to a normal distribution, the least-squares estimators are also the maximum likelihood estimators in a linear model.

However, suppose the errors are not normally distributed. In that case, a central limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.

Weighted least squares

"Fanning Out" Effect of Heteroscedasticity

A special case of generalized least squares called weighted least squares occurs when all the off-diagonal entries of Ω (the correlation matrix of the residuals) are null; the variances of the observations (along the covariance matrix diagonal) may still be unequal (heteroscedasticity). In simpler terms, heteroscedasticity is when the variance of ${\displaystyle Y_{i}}$ depends on the value of ${\displaystyle x_{i}}$ which causes the residual plot to create a "fanning out" effect towards larger ${\displaystyle Y_{i}}$ values as seen in the residual plot to the right. On the other hand, homoscedasticity is assuming that the variance of ${\displaystyle Y_{i}}$ and variance of ${\displaystyle U_{i}}$ are equal.^[10]  

Relationship to principal components

The first principal component about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the ${\displaystyle y}$ direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally.

Relationship to measure theory

Notable statistician Sara van de Geer used empirical process theory and the Vapnik–Chervonenkis dimension to prove a least-squares estimator can be interpreted as a measure on the space of square-integrable functions.^[16]

Regularization

See also

References

Further reading

• Björck, Å. (1996). Numerical Methods for Least Squares Problems. SIAM. ISBN 978-0-89871-360-2.
• Kariya, T.; Kurata, H. (2004). Generalized Least Squares. Hoboken: Wiley. ISBN 978-0-470-86697-9.
• Luenberger, D. G. (1997) [1969]. "Least-Squares Estimation". Optimization by Vector Space Methods. New York: John Wiley & Sons. pp. 78–102. ISBN 978-0-471-18117-0.
• Rao, C. R.; Toutenburg, H.; et al. (2008). Linear Models: Least Squares and Alternatives. Springer Series in Statistics (3rd ed.). Berlin: Springer. ISBN 978-3-540-74226-5.
• Van de moortel, Koen (April 2021). "Multidirectional regression analysis".
• Wolberg, J. (2005). Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments. Berlin: Springer. ISBN 978-3-540-25674-8.

External links

• Media related to Least squares at Wikimedia Commons