Residual sum of squares

src: i.ytimg.com

In statistics, the residual sum of squares (RSS), also known as the sum of squared residuals (SSR) or the sum of squared errors of prediction (SSE), is the sum of the squares of residuals (deviations predicted from actual empirical values of data). It is a measure of the discrepancy between the data and an estimation model. A small RSS indicates a tight fit of the model to the data. It is used as an optimality criterion in parameter selection and model selection.

In general, total sum of squares = explained sum of squares + residual sum of squares. For a proof of this in the multivariate ordinary least squares (OLS) case, see partitioning in the general OLS model.

Video Residual sum of squares

One explanatory variable

In a model with a single explanatory variable, RSS is given by:

${\displaystyle RSS=\sum _{i=1}^{n}(y_{i}-f(x_{i}))^{2}}$

where y_i is the i ^th value of the variable to be predicted, x_i is the i ^th value of the explanatory variable, and ${\displaystyle f(x_{i})}$ is the predicted value of y_i (also termed ${\displaystyle {\hat {y_{i}}}}$). In a standard linear simple regression model, ${\displaystyle y_{i}=a+bx_{i}+\varepsilon _{i}\,}$, where a and b are coefficients, y and x are the regressand and the regressor, respectively, and ? is the error term. The sum of squares of residuals is the sum of squares of estimates of ?_i; that is

${\displaystyle RSS=\sum _{i=1}^{n}(\varepsilon _{i})^{2}=\sum _{i=1}^{n}(y_{i}-(\alpha +\beta x_{i}))^{2}}$

where ${\displaystyle \alpha }$ is the estimated value of the constant term ${\displaystyle a}$ and ${\displaystyle \beta }$ is the estimated value of the slope coefficient b.

Maps Residual sum of squares

Matrix expression for the OLS residual sum of squares

The general regression model with 'n' observations and 'k' explanators, the first of which is a constant unit vector whose coefficient is the regression intercept, is

${\displaystyle y=X\beta +e}$

where y is an n × 1 vector of dependent variable observations, each column of the n × k matrix X is a vector of observations on one of the k explanators, ${\displaystyle \beta }$ is a k × 1 vector of true coefficients, and e is an n× 1 vector of the true underlying errors. The ordinary least squares estimator for ${\displaystyle \beta }$ is

${\displaystyle X{\hat {\beta }}=y\iff }$

${\displaystyle X^{T}X{\hat {\beta }}=X^{T}y\iff }$

${\displaystyle {\hat {\beta }}=(X^{T}X)^{-1}X^{T}y.}$

The residual vector ${\displaystyle {\hat {e}}}$ = ${\displaystyle y-X{\hat {\beta }}=y-X(X^{T}X)^{-1}X^{T}y}$, so the residual sum of squares is:

${\displaystyle RSS={\hat {e}}^{T}{\hat {e}}=\|e\|^{2}}$,

(equivalent to the square-root of the norm of residuals); in full:

${\displaystyle RSS=y^{T}y-y^{T}X(X^{T}X)^{-1}X^{T}y=y^{T}[I-X(X^{T}X)^{-1}X^{T}]y=y^{T}[I-H]y}$,

where H is the hat matrix, or the projection matrix in linear regression.

src: slideplayer.com

Relation with Pearson's product-moment correlation

The least-squares regression line is given by

${\displaystyle y=ax+b}$,

where ${\displaystyle b={\bar {y}}-a{\bar {x}}}$ and ${\displaystyle a={\frac {S_{xy}}{S_{xx}}}}$, where ${\displaystyle S_{xy}=\sum _{i=1}^{n}({\bar {x}}-x_{i})({\bar {y}}-y_{i})}$ and ${\displaystyle S_{xx}=\sum _{i=1}^{n}({\bar {x}}-x_{i})^{2}.}$

Therefore,

${\displaystyle RSS=\sum _{i=1}^{n}(y_{i}-f(x_{i}))^{2}=\sum _{i=1}^{n}(y_{i}-(ax_{i}+b))^{2}=\sum _{i=1}^{n}(y_{i}-ax_{i}-{\bar {y}}+a{\bar {x}})^{2}}$

${\displaystyle =\sum _{i=1}^{n}(a({\bar {x}}-x_{i})-({\bar {y}}-y_{i}))^{2}=a^{2}S_{xx}-2aS_{xy}+S_{yy}=S_{yy}-aS_{xy}=S_{yy}(1-{\frac {S_{xy}^{2}}{S_{xx}S_{yy}}})}$

where ${\displaystyle S_{yy}=\sum _{i=1}^{n}({\bar {y}}-y_{i})^{2}.}$

The Pearson product-moment correlation is given by ${\displaystyle r={\frac {S_{xy}}{\sqrt {S_{xx}S_{yy}}}};}$ therefore, ${\displaystyle RSS=S_{yy}(1-r^{2}).}$

src: www.researchgate.net

See also

• Sum of squares (statistics)
• Squared deviations
• Errors and residuals in statistics
• Lack-of-fit sum of squares
• Degrees of freedom (statistics)#Sum of squares and degrees of freedom
• Chi-squared distribution#Applications

src: slideplayer.com

References

• Draper, N.R.; Smith, H. (1998). Applied Regression Analysis (3rd ed.). John Wiley. ISBN 0-471-17082-8. 

Source of the article : Wikipedia