Double Blind Testing of HT Components - Necessary for you? (1 Viewer)

[John Morris]

With the growing published reviews of the Outlaw 950 and the Rotel 1066, among other new preamp-processors, it has been suggested by some that any sonic comparison of two or more of these units is not valid without Double Blind, Crossover, Randomized Testing. Others think that simply carefully listening to each preamp-processor in the cozy confines of your own HT is enough to make a decision on which sounds best.
Which camp do you fall into, and why?
1 - Double Blind Testing is usually the only way I can decide.
OR
2 - Most times I can decide after careful listening in my HT.
This should be interesting...

 

[george king]

Number 2. Let the flames begin, but DBTs are not particularly useful for audio equipment.

 

[Bob-N]

Daring today, aren't we?
There was a thread a couple of weeks ago that degenerated into the DBT (amps all sound the same?) debate/debacle. Nothing was accomplished but hurt feelings and bandwidth.
However, to start the ball rolling, I'm in the latter catagory. Give me time in my system and if there's a difference, I'll pick it up even if sighted.
My explaination for this is that I have done quite a bit of sighted, side by side comparison speaker listening using the same equipment, same room and I can tell what I like and don't like given a good hour or two for each speaker pair listening to familiar/favorite material. My dealer carries 10 different brands of speakers, each with at least 2 families. I probably listened to about 20 different pairs of speakers and got a good idea of what "sound" I liked. Then I pruned it down from there.
I wouldn't anticipate it being much different with listening to pre/pros using the same material, equipment and room.
Of course, YMMV.
Bob

 

[Jed M]

I would go with 2 as well. I say for two simple reasons, 1. I trust my ears and can tell if I like something, and 2. I don't have the time in my life to Double blind test everything I want to buy. If that was the case I would do it for grocery products and cars. Sometimes you have to rely on yourself to figure it out.

 

[John Morris]

I didn't want to voice my opinion first... but I just can't hold it in anymore. I pick #2 also. Why?
First: True Randomized, placebo controlled, double blind, crossover testing is virtually impossible to design and achieve when testing two different components. Why? Even if you could build three identical rooms where the only difference at all is the tester, the intersubject/intertester variability is so great as to make reaching a significant p value virtually impossible without probably testing 1000s of people on each of the three choices: preamp#1, preamp#2 and Placebo. Oh yeah, and how do you design a placebo unit...a white noise generator? And how do you account for the infinite differences between each of our systems? Bad methodology and/or a bad protocol will make any results null and void and a waste of time and effort. So how do I make my choices?
Well, IMO, you need to think about your base hypothesis BEFORE you think about which protocol and methodology you need to employ for testing. Are you trying to determine if one pre-pro's sound is superior to the other pre-pro's sound, period? Or are you just trying to determine which unit each one of us, prefers Individually? If the latter is your goal, then the faceplate, weight, cost and color of the on/off button MAY all be considered based on your personal priorities. If your friends all have Krell's, and you've always wanted a Krell, and you'd be disappointed every time you look at your system unless a Krell is in the rack, then the Outlaw 950 is not for you regardless of good it sounds in comparison to the Krell.
IMO, other's reviews and opinions can only serve as a guide as to where you might start your own personal search. IMO, the only way to determine which pre-pro is the one to buy, is to take the pre-pros home and listen to them in YOUR system. Play the music and movies YOU like and Carefully take notes. Then, switch out pre-pros and do it again. At this point, you'll probably note any major differences like a lower noise floor. Still, switch back to the original pre-pro and do it again. In my experience, if there is any discernable sonic differences in the units, you WILL hear it by the time you've switched for the second time. Things will pop out at you. After you've carefully notated these things, switch back again and look for those things to happen, or not happen, at exactly the same point on the same disc, with the other unit. At this point, sit down and consider other factors such as features, cosmetics, cost and yes, even snob appeal. If you determine that you cannot make a decision based on what you've heard, then it may come down to which unit costs less; or what unit looks the best in your rack. It is a personal decision for EACH of us to make. Our purchases have to make EACH of us happy, and NO ONE else... unless of course, they are paying for the unit!
Of course, this is only my opinion and valid for me and how I'm gonna pick my next pre-pros. YMMV!

 

[Roger Kint]

I must be deaf in the audiophile world. I pick #1 Dbl Blind Test. I don't think I can hear the differences and it would be easier for me to decide on the spot in real-time. If I take a unit home, listen to it, I might say yeah it sounds really good. Take another unit home, listen to it yah this sounds good too, another unit home, sounds really good too. Darn they all sound good as the next unit. I can't tell. I guess I don't have a good audio memory or my ears are not fined tuned like some super hearing humans out there.

Ideally I'd like a DBT in my HT. For me, that solves everything.

 

[brian a]

Merc,

I think the problem that you're running into is people misusing the term DBT. I believe what most of these people have suggested is that you do exactly what you just listed only you have someone else do the switching without you knowing which unit is playing. Then you just pick the one that sounds better. While not a true DBT, it seems like a reasonable test. This is espically true for folks who claim a night and day difference between units.

I actually have no opinion on this one either way. If you're happy with your gear, you're happy with your gear. I do however think that instead of attacking people's misuse of the DBT term, people who are against this process should say what they feel is wrong with using it to determine which equipment sounds better to them.

 

[John Morris]

I think the problem that you're running into is people misusing the term DBT.

brian: So true. In fact for it even to use the D in DBT the person who does the switching must also not know which unit he/she is switching to... So, I guess you'd have to take both pre-pros that you want to compare and send them in to a custom shop for placement into a common outside case with common backplate. Otherwise, if the switcher knows which unit he/she chooses, and you know that he/she knows, your guessing as to the testing strategy that the switcher employs can also supposedly color your opinions just as much as simply knowing which unit is which. Also, without using a placebo control group, or without reaching a significant p value(usually requires a very high number of testing opinions), the opinion you may reach cannot be determined to be due to the sound any more than that opinion as reached by pure chance. Does that make sense?

 

[Gregory F]

Double blind testing cars can be very dangerous! Please do not try this at home, or at least in the DFW, TX area where I live.
That said, I am sure that I don't need a double blind test to tell the difference between the Firebird I used to drive and the Corvette Z06 I now drive. They both had 4 wheels and more performance (top speed) than I care to use, but I am sure I could tell the difference. I am also sure that I am tell you which one I prefer.
Can I tell the difference between the sound of amps. I don't know as I have tried very hard, but I do know that my speakers determine which amps I buy. I have Apogee Caliper Signatures and Martin Logan Scripts adn a ML Theater(? the big center channel) and I can not drive these with just any reciever/amp. So the same measured frequency response with an 8 ohm resistive load just does not mean that that the amp will sound the same with my speakers.
By the way how do measure the ability of a speaker to resolve detail in music?
So mark me in the 2 category.
I am influenced by image but I really like bang for the buck. I won't buy any Mercedes I have look at. The performance to price doesn't get me there. If you have ever been to Germany and ridden around in the Mercedes taxi cabs, you would not be as impressed with the mark. It's a german Chevrolet .
Sorry for the ramble.
Greg

 

[KeithH]

Oh, no. Here we go again. Let the games begin! I fall into camp 2. There is an assumption made by many believers in DBTs that people who choose one component over another in a sighted comparison automatically have a bias going in. :rolleyes

 

[John Morris]

Guys: Please, let's try to keep this thread on topic.
We are talking ONLY about what is necessary for you to decide on which of two preamp-processors sound best and why and why the other choice doesn't work for you... in your deciding which of two units sound best. That is all.
Please, we are not talking about testing speakers or amps or, god forbid, cables.
Thank you all very much for your cooperation!

 

[george king]

John,

Given your last clarification, I guess I am missing the point of the thread. The absolute best that a properly done DBT can do is determine whether two or more components sound "different." The assessment of "better" is not terribly amenable to a DBT at all (although a SDT type analysis might work).

Am I missing something?

 

[Roger Kint]

Can I get further clarification on the original question?

Is it a DBT at the store vs. sighted test at home?

It that was the case, I'd take the sighted test at home because there are too many uncontrolled and unknown variables at the store.

Or is it DBT vs no DBT? IOWs, to see who thinks their ears are as sharp and their judging unbiased with their eyes open or closed. IMHO, I would think that removing one sense would make the others more acute and therefore a more accurate judge. And with eyes closed there is no question about visual aesthetic bias. So I believe blind testing is superior, all other things being equal/same.

 

[John Morris]

George: ABSOLUTELY! That is another reason (reason #101?) why I think a DBT is not appropriate for making purchase decisions nor for even determining which of two preamp-processors sound best. Ideally, a DBT uses double blinded placebo controlled(if ethically possible)protocols and efficacy is judged by the investigator who looks at subjectively measurable outcomes which are previously established as determining success. How can you possibly do that in order to choose which preamp-processor is best? I guess you could have a panel watch the subject from behind a two way mirror and count the number of smiles and grimaces they exhibit during each testing session... and then repeat that a few hundred times to achieve p value significance... :rolleyes
Maybe, in fact, as brian suggested, most DBT folks are talking about just comparing the two units during the same approximate time period connected to the same other gear?

 

[John Morris]

Roger: To further clarify,
maybe, all this question really comes down to is:
Can you fairly decide which of two preamps sound best to you if you switch back and forth between two preamps, while carefully noting any differences between them?
Or
Does simply knowing which unit you are listening to, make you unable to clearly determine which sounds best to you?
Dang, now you guys have me confused too!

 

[chung]

There is a debate ongoing at the rec.audio.high-end newsgroup. Do a google group search and you can see the viewpoints on both sides.
I think a lot of people don't understand what DBT means. The website www.pcabx.com contains valuable info, IMO. The primary purpose of DBT is to validate that perceived differences exist under well-controlled conditions. Whether one uses DBT to make buying decisions is a separate issue altogether. There are a lot of factors going into buying decisions: price, aesthetics, reliability, performance, ergonomics, prestige, etc. I don't think even the strongest DBT proponents propose making buying decisions based solely on DBT results.

 

[KeithH]

If one wants to compare two different HDTVs, should a double-deaf test (DDT) be conducted? Bah-doomp-chishhhh!

 

[Evan S]

I think, keeping this overly simplistic....
If I were to sit down, blindfolded and had some sort of switcher box that allowed someone in the room to instantaneously switch between my Anthem AVM-20 and say either the new Rotel-1066 or Outlaw 950, I would have to be extremely arrogant to be able to instantaeously tell you which one was which.
A good processor should impart very little of it's own sonic characteristics onto the finished product and therefore, I think without doing the test for hours on end, I would be hard pressed to tell the differences, and even if differences could be ascertained, I highly doubt I could make a blanket statement like..."Yep, that's definately the Outlaw."
Speakers on the other hand.....

 

[Saurav]

Nothing about a DBT says you have to tell anything instantly. You can switch back and forth as many times as you want, for as long as you want. In fact, you even know which is which, in a manner of speaking - you can hear choices A, B, and X. You already know, in advance, that A is the Outlaw and B is the Rotel (or whatever). The DBT test allows that. All you have to do is tell if X is A or B. You're allowed to switch back and forth between A, B and X as many times as you want.

So, you don't even have to rely on memory to tell if the current unit you're listening to is A or B, you can listen to A and B in a sighted way, i.e., know which is which, in every test.

 

[John Morris]

A good processor should impart very little of it's own sonic characteristics onto the finished product and therefore, I think without doing the test for hours on end, I would be hard pressed to tell the differences, and even if differences could be ascertained, I highly doubt I could make a blanket statement like..."Yep, that's definately the Outlaw."

Evan: Wow, I agree with this too, usually.