Mallows's C_p

In statistics, Mallows’s C_p,[1][2] named for Colin Lingwood Mallows, is used to assess the fit of a regression model that has been estimated using ordinary least squares. It is applied in the context of model selection, where a number of predictor variables are available for predicting some outcome, and the goal is to find the best model involving a subset of these predictors. A small value of C_p means that the model is relatively precise.

Mallows’s C_p has been shown to be equivalent to Akaike information criterion in the special case of Gaussian linear regression.[3]

Definition and properties

Mallows’s C_p addresses the issue of overfitting, in which model selection statistics such as the residual sum of squares always get smaller as more variables are added to a model. Thus, if we aim to select the model giving the smallest residual sum of squares, the model including all variables would always be selected. Instead, the C_p statistic calculated on a sample of data estimates the mean squared prediction error (MSPE) as its population target

${\displaystyle E\sum _{j}({\hat {Y}}_{j}-E(Y_{j}\mid X_{j}))^{2}/\sigma ^{2},}$

where ${\displaystyle {\hat {Y}}_{j}}$ is the fitted value from the regression model for the jth case, E(Y_j | X_j) is the expected value for the jth case, and σ² is the error variance (assumed constant across the cases). The MSPE will not automatically get smaller as more variables are added. The optimum model under this criterion is a compromise influenced by the sample size, the effect sizes of the different predictors, and the degree of collinearity between them.

If P regressors are selected from a set of K > P, the C_p statistic for that particular set of regressors is defined as:

${\displaystyle C_{p}={SSE_{p} \over S^{2}}-N+2(P+1),}$

where

• ${\displaystyle SSE_{p}=\sum _{i=1}^{N}(Y_{i}-Y_{pi})^{2}}$ is the error sum of squares for the model with P regressors,
• Y_pi is the predicted value of the ith observation of Y from the P regressors,
• S² is the residual mean square after regression on the complete set of K regressors and can be estimated by mean square error MSE,
• and N is the sample size.

Alternative definition

Given a linear model such as:

${\displaystyle Y=\beta _{0}+\beta _{1}X_{1}+\cdots +\beta _{p}X_{p}+\varepsilon }$

where:

• ${\displaystyle \beta _{0},\ldots ,\beta _{p}}$ are coefficients for predictor variables ${\displaystyle X_{1},\ldots ,X_{p}}$
• ${\displaystyle \varepsilon }$ represents error

An alternate version of C_p can also be defined as:[4]

${\displaystyle C_{p}={\frac {1}{n}}(\operatorname {RSS} +2d{\hat {\sigma }}^{2})}$

where

• RSS is the residual sum of squares on a training set of data
• d is the number of predictors
• and ${\displaystyle {\hat {\sigma }}^{2}}$ refers to an estimate of the variance associated with each response in the linear model (estimated on a model containing all predictors)

Note that this version of the C_p does not give equivalent values to the earlier version, but the model with the smallest C_p from this definition will also be the same model with the smallest C_p from the earlier definition.

Limitations

The C_p criterion suffers from two main limitations[5]

1. the C_p approximation is only valid for large sample size;
2. the C_p cannot handle complex collections of models as in the variable selection (or feature selection) problem.[5]

Practical use

The C_p statistic is often used as a stopping rule for various forms of stepwise regression. Mallows proposed the statistic as a criterion for selecting among many alternative subset regressions. Under a model not suffering from appreciable lack of fit (bias), C_p has expectation nearly equal to P; otherwise the expectation is roughly P plus a positive bias term. Nevertheless, even though it has expectation greater than or equal to P, there is nothing to prevent C_p < P or even C_p < 0 in extreme cases. It is suggested that one should choose a subset that has C_p approaching P,[6] from above, for a list of subsets ordered by increasing P. In practice, the positive bias can be adjusted for by selecting a model from the ordered list of subsets, such that C_p < 2P.

Since the sample-based C_p statistic is an estimate of the MSPE, using C_p for model selection does not completely guard against overfitting. For instance, it is possible that the selected model will be one in which the sample C_p was a particularly severe underestimate of the MSPE.

Model selection statistics such as C_p are generally not used blindly, but rather information about the field of application, the intended use of the model, and any known biases in the data are taken into account in the process of model selection.

See also

• Goodness of fit: Regression analysis
• Coefficient of determination

References

1. Mallows, C. L. (1973). "Some Comments on C_P". Technometrics. 15 (4): 661–675. doi:10.2307/1267380. JSTOR 1267380.
2. Gilmour, Steven G. (1996). "The interpretation of Mallows's C_p-statistic". Journal of the Royal Statistical Society, Series D. 45 (1): 49–56. JSTOR 2348411.
3. Boisbunon, Aurélie; Canu, Stephane; Fourdrinier, Dominique; Strawderman, William; Wells, Martin T. (2013). "AIC, C_p and estimators of loss for elliptically symmetric distributions". arXiv:1308.2766 [math.ST].
4. James, Gareth; Witten; Hastie; Tibshirani (2013-06-24). An Introduction to Statistical Learning. http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf: Springer. pp. 211. ISBN 978-1-4614-7138-7.CS1 maint: location (link)
5. Giraud, C. (2015), Introduction to high-dimensional statistics, Chapman & Hall/CRC, ISBN 9781482237948
6. Daniel, C.; Wood, F. (1980). Fitting Equations to Data (Rev. ed.). New York: Wiley & Sons, Inc.

Further reading

• Chow, Gregory C. (1983). Econometrics. New York: McGraw-Hill. pp. 291–293. ISBN 978-0-07-010847-9.
• Hocking, R. R. (1976). "The analysis and selection of variables in linear regression". Biometrics. 32 (1): 1–50. CiteSeerX 10.1.1.472.4742. doi:10.2307/2529336. JSTOR 2529336.
• Judge, George G.; Griffiths, William E.; Hill, R. Carter; Lee, Tsoung-Chao (1980). The Theory and Practice of Econometrics. New York: Wiley. pp. 417–423. ISBN 978-0-471-05938-7.