Problems with CR!
I found out about a blog written by Michael Karesh of TrueDelta from a news article here at allpar. The original blog by Michael Karesh can be found here.
In Michael’s blog, he discusses the “anomalies” that exist in CR’s reporting about the “reliability” of particular vehicles. The majority of the vehicles discussed in Michael’s blog consists of vehicles that have different bodystyles of the same basic car structure, e.g., Dodge Magnum and the Chrysler 300. As Michael states in his blog these vehicles “share the great majority of their parts, including powertrains. This is especially the case if both are made in the same plant.”
Michael then reports on such “anomalies” such as the V8 Magnum scoring 55 points higher then the V8 300 yet the V6 Magnum scores 25 points worse then the V6 300. Michael goes on to discuss a wide variety of vehicles where this same “anomaly” or situation occurs.
Michael runs a site, named TrueDelta, where his analysis not only includes the mean or average number of successful repair trips per year but he also includes the standard deviation (error bars, uncertainties, variance, confidence intervals) of his data. This type of data reporting is crucial to any analytic undertaking. An example of his data reporting can be found here at his website.
From Michael’s analysis it is easy to see, with no questions asked, that the “reliability” or the number of trips per year that the 06 Chevy HHR is larger then the average number of trips per year of a 06 VW Jetta. Error bars are included!!! Similarly, his analysis shows that a 06 Chevy HHR is not more “reliable” then an 06 Charger/Magnum/300 or visa versa.
From surveys, people report to CR about a number of items about their particular vehicle. They report on the vehicles cooling, suspension, transmission, engine, etc. For each vehicle, each one of these sub-classification, e.g., transmission, will have a distribution of numbers related to the “reliability” or happiness of the consumer. (CR’s data is obtained by polling information of their subscribers). To calculate the average CR would add all of the data and divide by the number of data points. To calculate the error of their average CR would (or does but who knows since they do not publish these results) calculate a root mean squared uncertainty. This uncertainty is found when each data point is subtracted from their calculated mean, then squared, then summed, then divided by the number of data points (or N-1), then finally the square root of this number obtains their uncertainty or standard deviation. Another possible method is to fit the data to a Gaussian then the standard deviation would just be the second moment of the distributions or sigma = sqrt[{x^2} - {x}^2]). (These should be averages in x, but I cannot place the “average” symbols here). In summary each item that CR reports, e.g., transmission or engine, should have a mean and a standard deviation associated with the mean.
For example if CR asks 4 owners of a Chrysler 300 to rate their suspension of their vehicle; rating the suspension from 0 to 10 they may be a variety of answers. Lets say for illustrative purposes the responses that they get are as follows
Person A may say 7
Person B may say 3
Person C may say 8
Person D may say 10
CR would report the suspension of the 300 to be (7+3+8+10)/4 = 7. To calculate the error of their reported average they would calculate {[(7-7)^2+(3-7)^2+(8-7)^2+(10-7)^2]/4}^(1/2) = 2.5 (If we used N-1 or 4-1=3 for the normalization constant, the error would be 2.9).
It would be wrong for CR to just report that the average Chrysler 300 customer rates their suspension of their vehicle as 7 out of 10. They must also report the variance of their data and state that the average 300 customer rates their suspension to be 7 +/- 2.5 out of ten.
Then CR must then give an average of all of the sub-classifications of each vehicle. (I take it this is where they rate or calculate the reliability of the vehicle). To accomplish this they will find an average of each of the sub-classification. To find their uncertainty on their “reliability” of the vehicle they must add the square of each of their uncertainties of the sub-classifications to find the “average uncertainty” for each vehicle.
For example, Vehicle A was surveyed and the average and uncertainty of each of the 5 classification or sub-classes was found to be the following (for illustrative purposes)
Engine: 5 +/- 3
Transmission: 7 +/- 3
Suspension: 6 +/- 4
Cooling Ability: 7 +/- 3
Noise Levels: 5 +/- 4
The average score for the overall vehicle would be (5+7+4+7+5)/5 = 5.6 (With equal weighting for each of the sub-classifications).
We wouldn’t add their uncertainties like this (2+3+2+3+4)/5 = 2.8 (This is wrong).
You add the squares of their uncertainties, then take the square root such that [(2^2+3^2+1^2+3^2+4^2)/5]^(1/2) = 2.898.
So the “reliability” of Vehicle A is found to be 5.6 +/- 2.9.
The same survey for Vehicle B might show:
Engine: 6 +/- 4
Transmission: 8 +/- 2
Suspension: 5 +/- 5
Cooling Ability: 7 +/- 3
Noise Levels: 4 +/- 3
Which would have a “reliability” if 6 +/- 3.5.
CR, in my opinion, would rate Vehicle B higher or more “reliable” then Vehicle A. Sure the average is higher along with the upper bounds of the error bars (6+3.5=9.5 versus 5.6+2.9=8.5) but the lower end of the error bars puts vehicle B at 2.5 (6-3.5=2.5) versus 2.7 for vehicle A (5.6-2.9=2.7).
However, it is my opinion that it would be silly or not prudent to say that Vehicle B is more “reliable” then Vehicle A due to the larger area of overlap between the error bars associated with each vehicle. A reduction in the error bars would be needed before a concrete discussion was made about the relative reliability of either vehicle. One may reduce the size of the error bars by increasing the number of data points.
In order to calculate the reliability of brands the average reliability of each of the brands vehicles should then be calculated. In order to accomplish this CR would have to make a decision on how to weight each vehicle in their calculations.
For illustrative purposes, if brand X produces 3 Vehicles A, B, and C.
Vehicle A has a calculated “reliability” rating of 5 +/- 2.
Similarly Vehicle B has 7 +/- 1 and Vehicle C has 8 +/- 1.
Assuming equal weighting for each vehicle, brand X would have an average reliability rating of 6.7 +/- 1.4.
However, if brand X sold 10,000 units a year comprised of 9,000 units of Vehicle A, 750 units of Vehicle B, and 250 units of Vehicle C. You would think that you should included the weights of these units into your calculation for the “reliability” of brand X. Assuming this, brand X would have a “reliability” rating of 5.23 +/- 0.69 (instead of the equal weighting of the vehicles found to be 6.7 +/- 1.4).
CR doesn’t state how they weight each vehicle when they calculate an average for each brand. Obviously this is important information since it dramatically changes the results.
In summary, I want to congratulate Michael Karesh of TrueDelta for finally creating a second option, relative to CR, for consumers to obtain information regarding manufactures of automobiles. His analysis has been open and honest. It contains the most BASIC ingredients, ones which are lacking in CR reporting, regarding statistical analysis of data collection.
I used to like CR as a rough guide to reliability and for their ability to evaluate what would not occur to me. I could eliminate from consideration the Yugos of the lot and choose from the rest what is most important to me. Recently, I’ve not been able to make much sense of their ratings.
Your complex analysis might work but for the GIGO principle. Who are the people responding to the survey and how objective are they? Discounting positive or negative shills, there remains the question of whether it is the more satisfied or dissatisfied car owners that are more disposed to respond to the survey.
Yes, but all surveys are biased by the fact that as a voluntary response, the people most likely to respond feel most strongly either good or bad. I would guess that most are dissatisfied with the vehicle.
Huh? The surveys ask if you had a problem, not if you like the vehicle.
Producing things like “error bars” shows the confidence you have with your “mean” or “average.”
That’s the point. So as long as you, like Michael Karesh, are trueful in displaying your uncertainties, the buying public will be able to make better choices because of the honesty of the reporter.
“Huh? The surveys ask if you had a problem, not if you like the vehicle.”
That leaves open the questions of 1) who knows of the survey and 2) of they who know, who responds.
I don’t discount the surveys but can consider them but a rough guide of reliability. People are in general quirky and with regards cars…fuhgetaboutit.
If you want a good feel for a car, go to http://www.carsurvey.org, disreguard the hi and the low and you’ll have a pretty good idea weather the car is a peice of junk or ok
The link won’t work with the ending comma in the URL.
http://www.carsurvey.org
I do agree that website like carsurvey.org are great.
In a couple of clicks you can find many different blogs about ones car. For some vehicles there are more then one blog and for other vehicles there are are hundreds.
So, I went to the site and checked out a blog for a Commander Limited. IMO, the blog was a bit biased towards the vehicle. The person obviously just bought it, was very excited about it, and rated it 10/10 in every category he mentioned.
But is this persons comments about the vehicle wrong? No.
Will he feel the same way about his Commander 6 months from now? Maybe.
Anyway, the point I was trying to get at with this blog was that CR distorts its reporting about the vehicles they survey. To me this is not only wrong, it is inexcusable. Your first year science teacher (high school or college) would mark you down a grade letter if you did that in his/her lab.
CR comments are ALWAYS foreign car sympathetic. USA brand vehicles are as good, and probably better, than anyones; but take a pounding from the prideful owners of the imports who have to rationalize their purchase decisison as related to “superior quality issues”. To do otherwise would be just unpatriotic. So after we have sold our domestic auto makers out of business with CR leading the misinformation war, what will we drive? Duh! Chinese anyone?
Claiming to be wise…they became fools. What a great line.
I actually enjoyed this years auto report becuase it shows how bad they actually are doing. They have some cars with horrible ratings, yet they recommend them…HUH! Yes they seem to throw in some weird data that they have yet to disclose that bumps certain cars into the category. My guess is they want all american companies to go belly up…oops, butnot FORD…notice how they recommended them over GM and Chrysler as if to help them and only when they are in real bad shape. I’ve argued with some that some owners don’t go into shops reguarly and some import owners get everything done even if it’s not needed. Which tells me that one group is spending more than they should to keep themselves happy
This years issue was an eye opener. Goes back to the basic sense of avoid the first year of a model and buy what you like.
Consumer Reports is facilitating Consumer Fraud based on snobbishness and greed.
Their “reliability figures” at one time were very useful, and statistically valid. In the 1970s the typical American car had around 750 defects per 100 vehicles or 7.5 defects per vehicle, on average. The typical imported Japanese quality leader Toyota had only 350 defects per hundred vehicles, or 3.5 defects per vehicle on average.
Thirty five years later, every manufacturer has improved quality. American manufacturers had to discard all their old, proven but over sized technology and replace them with new down-sized versions. Some of which frankly, were not very good, leading to a reason for a good part of the quality differences seen then of 350 versus 750 defects per hundred vehicles.
Today Toyota, the perceived quality leader has around 110 defects per vehicle or 1.1 defects per vehicle, on average. The typical American vehicle has around 120 defects per hundred or 1.2 defects per vehicle, on average. Is there still a valid statistical basis to project reliability difference of greater than 60% or more, that Consumer Reports insists that there is?
NO! Since most of the difference now is due to the fact that Toyota starts building the first year versions of its new models in Japan and so Americans don’t see the new car teething problems. With those “teething pains” gone that biases the Toyota statistics, as slightly better.
Does CR understand this? If they do so, they ignore it, so the answer is NO. They insist that still are dramatic differences in quality even when comparing second year production models which presumably have the ‘teething pains’ resolved for all manufacturers, and these in fact do show, no statistical differences.
They persist with this nonsense when they evaluate Used Cars. Their wrong quality ratings help set resale values, driving down the value of essentially equal vehicles which don’t have their wrongly assigned superior quality ratings.
They still recommend that their consumers waste their money on hater over reported initial quality vehicles, instead of purchasing “more car for the money”, as they continue the scam and urge their readers to overpay for the quality differences which just are not statistically there. Consumer Reports has its consumers needlessly overspending to purchase perceived “quality” that is not statistically there at all, except in their imaginations and unsupported by proper statistics.
The question is WHY ???
If I were a cynical ambulance-chasing lawyer, I could argue they do that to preserve their “consumer reliability studies” and the reason why consumers purchase their magazines in the first place. So its G-R-E-E-D. Million of $ is in the balance, a class action suit is more than OVERDUE !