Sorry for the contentious title, but I figured ' why so little blind testing?' might not get the same response. 
What this is about is reviews of most home theater equipment that we read in magazines, etc (though it could apply to many other reviews). Let's take a typical example. Reviewer gets a new DVD player from Company X to review. Chances are that player is provided free on extended loan, reviewer will no doubt have enjoyed hospitality of the said company at one of more corporate events, etc. This instantly raises the possibility of bias, but let's assume the reviewer is incorruptable. He or she then plugs the player into a home theater system and sees how it performs. Okay, if s/he does some technical tests, these have an objectivity to them, but what about the subjective impressions? The reviewer knows what they're reviewing, will have all sorts of preconceptions (not least based on price), and these will all colour his or her judgment.
I teach psychology, and if the rawest of raw freshmen produced a study like that, I would personally lynch them as a warning to other students not to waste my time. The whole home theater review process is just so plain wrong that it would never be countenanced in the behavioural sciences. So why on earth is it allowed to continue in nearly all magazines, review sites, etc? I find it amusing and ironic that magazines that will produce streams of technical data about response rates, frequency responses and goodness knows what else can make such an elementary mistake.
There is of course a way round it, called blind testing. In the case of a DVD player, the reviewer would have an assistant who would hide the player and all the reviewer would see would be the picture produced on the screen. This would then be compared against several other players (again, their identities hidden) and each would be graded separately. Only at the end of this blind appraisal would the reviewer find out their grading of the DVD player under assessment. Thus, any subjective opinions would be based purely on the product and not on the bias of expectation, etc.
But as things stand, we have an assessment process that is so unreliable it's a farce. And in the process the consumer stands a good chance of getting ripped off.
So why don't critics do blind testing? Might it be that if they did we'd find that there's far less difference between products than they'd like to admit?
What this is about is reviews of most home theater equipment that we read in magazines, etc (though it could apply to many other reviews). Let's take a typical example. Reviewer gets a new DVD player from Company X to review. Chances are that player is provided free on extended loan, reviewer will no doubt have enjoyed hospitality of the said company at one of more corporate events, etc. This instantly raises the possibility of bias, but let's assume the reviewer is incorruptable. He or she then plugs the player into a home theater system and sees how it performs. Okay, if s/he does some technical tests, these have an objectivity to them, but what about the subjective impressions? The reviewer knows what they're reviewing, will have all sorts of preconceptions (not least based on price), and these will all colour his or her judgment.
I teach psychology, and if the rawest of raw freshmen produced a study like that, I would personally lynch them as a warning to other students not to waste my time. The whole home theater review process is just so plain wrong that it would never be countenanced in the behavioural sciences. So why on earth is it allowed to continue in nearly all magazines, review sites, etc? I find it amusing and ironic that magazines that will produce streams of technical data about response rates, frequency responses and goodness knows what else can make such an elementary mistake.
There is of course a way round it, called blind testing. In the case of a DVD player, the reviewer would have an assistant who would hide the player and all the reviewer would see would be the picture produced on the screen. This would then be compared against several other players (again, their identities hidden) and each would be graded separately. Only at the end of this blind appraisal would the reviewer find out their grading of the DVD player under assessment. Thus, any subjective opinions would be based purely on the product and not on the bias of expectation, etc.
But as things stand, we have an assessment process that is so unreliable it's a farce. And in the process the consumer stands a good chance of getting ripped off.
So why don't critics do blind testing? Might it be that if they did we'd find that there's far less difference between products than they'd like to admit?
.
