Problems with CR! Part I
I found out about a blog written by Michael Karesh of TrueDelta via allpar here. The original blog by Michael Karesh can be found here.
Michael in his blog discusses the “anomalies” that exist in CR’s reporting of the “reliability” of each vehicle. The majority of the vehicles discussed in Michael’s blog consists of vehicles that have different bodystyles of the same basic car structure, e.g., Dodge Magnum and the Chrysler 300. As Michael states in his blog these vehicles “share the great majority of their parts, including powertrains. This is especially the case if both are made in the same plant.”
Michael then reports on such “anomalies” such as the V8 Magnum scoring 55 points higher then the V8 300 yet the V6 Magnum scores 25 points worse then the V6 300. Michael goes on to discuss a wide variety of vehicles where this same “anomaly” or situation occurs.
Michael runs a sit, named TrueDelta where his analysis not only includes the mean or average number of successful repair trips per year but he also includes the standard deviation (error bars or confidence intervals) of his data. This type of data reporting is crucial to any analysis undertaking. An example of his data reporting can be found here at his website.
From Michael’s analysis it is easy to see, with no questions asked that the “reliability” or the number of trips per year that the 06 Chevy HHR will have is larger then the average number of trips per year a 06 VW Jetta will have. Error Bars Included!!! Similarly, his analysis shows that one cannot say that the 06 Chevy HHR is more “reliable” then an 06 Charger/Magnum/300 or visa versa.
One possible reason why CR does not discuss their error bars or standard deviations from their means is due the possible large values of their error bars. People report to CR about a number of items about their vehicle. They report on the vehicles cooling, suspension, transmission, engine, etc. For each vehicle, each one of these sub-classification, e.g., transmission, of the vehicle will have a distribution of numbers related to the “reliability” or happiness of the consumer. (CR’s data is obtained by polling information). To calculate the average CR would add all of the data and divide by the number of data points. To calculate the error of their average or mean CR would (or do, who knows) would calculate a root mean squared uncertainty. Each data point would be subtracted from their calculated mean then squared, summed, divided by the number of data points (or N-1) then the square root of this number obtains the uncertainty or the standard deviation. (Another possible method is to fit the data to a Gaussian then the standard deviation would just be sigma = sqrt[ - ^2]). In summary each item that CR reports, e.g., transmission or engine, should have a mean and a standard deviation.
For example if CR asks four owners of a Chrysler 300 to rate their suspension of their vehicle rating the suspension from 0 to 10.
Person A may say 7
Person B may say 3
Person C may say 8
Person D may say 10
CR would report the suspension of the 300 to be (7+3+8+10)/4 = 7. To calculate the error of their reported average we would calculate {[(7-7)^2+(3-7)^2+(8-7)^2+(10-7)^2]/4}^(1/2) = 2.5 (If we used N-1 for the normalization constant the error would be 2.9).
It would be wrong for me to just report that the average 300 customer reports that the suspension of their vehicle rates a 7 out of 10. We must include the variance of the data and state that the average 300 customer rates their suspension to be 7 +/- 2.5 out of ten.
Then CR must then give an average of all of the sub-classifications of their car….I take it this is where they rate their reliability of the vehicle. There they will again find the average of each of the sub-classification, but to find their uncertain here they must add the square of their uncertainties to find the “average uncertainty.” Here is were the their uncertainties will grow.
