If you are looking for overall summary of how well your model fits the observed data, then an alternative is to work directly with the likelihood ratio statistic, sometimes expressed on an additive scale as -2 ln likelihood. For categorical data this can be simply calculated as the G^2 statistic
However, if your interest is more in identifying outlying individual observations, then a calculation of residual values for each can be useful (e.g., Pearson residuals, or deviance residuals), particularly if calibrated as studentized values. Pearson residuals form the components that make up the Pearson X^2 statistics, while deviance residuals combine to form -2log likelihood, known as the deviance.
Finally, one can reduce over-fitting of a model by using a training sample of observations to estimate the model and then a separate testing set to evaluate the fit of the model (which seems to be something along the lines which you have described). The Prediction Error Sum of Squares is a summary measure of the fit of a regression model to the set of observations that were not themselves used to in estimating the model. It is the sums of squares of the prediction residuals for those observations.
Nick
Edited: 23 Dec 2012, 6:42 a.m.
