Double Blind Testing of HT Components - Necessary for you? (1 Viewer)

[Saurav]

That's been my big question with DBTs the switcher box. If you take that out of the picture, then you're manually swapping components, and that takes too long to be meaningful.

Here's an analogy - can most people tell the difference between a 50Hz tone and a 100Hz tone? Yes, they can, those are a full octave apart. Can most people tell the difference between a 10,000Hz tone and a 10,050Hz tone? I think that would be much harder to differentiate, as that's probably less than a note apart. Human response to a lot of things is logarithmic - a doubling of volume is perceived as a linear increase in loudness, a doubling of frequency is perceived as a linear increase too - every octave sounds the same 'distance' away from the previous one.

So, I wonder if something like that can be a factor, that the act of adding the switch box in the middle can mask the difference between the components and make it harder to hear - the difference is still there, in the same magnitude, but it's been shifted to another point on a scale. Just speculating...

 

[John Morris]

Nothing about a DBT says you have to tell anything instantly. You can switch back and forth as many times as you want, for as long as you want. In fact, you even know which is which, in a manner of speaking - you can hear choices A, B, and X. You already know, in advance, that A is the Outlaw and B is the Rotel (or whatever). The DBT test allows that. All you have to do is tell if X is A or B. You're allowed to switch back and forth between A, B and X as many times as you want.

Saurav: Well, if that is the A/V definition of a Double Blind Test, it is mostly unrelated to any pharmaceutical Double Blind Test that I've ever heard of...? Why would you even do this test? What purpose would it serve? As far as I'm concerned, if you CAN pick out a specific preamp-processor by it's sound, THAT would probably NOT be a preamp I'd ever want to buy.

I am currently doing some tube rolling in my new Tubed Headphone amp, and although I love the weighty midrange, I miss the clear crisp unrolled off upper treble and the quick bass that I get from SS amps. Still, at this time, I can easily identify one of the three tubes that I'm currently rolling, under a DBT as you describe it, just by playing back parts of 3 seperate CD and SACD cuts and listening for the only tube of the three to get each detail better than the other two. However, if you just inserted one tube, and said what tube is this, hell, I could guess but that'd be all it would be... a guess.

 

[Martice]

DBT with Preamps should be the least of your worries. If you sample a unit in the store and you find that it sounded better in the store than it does in your house, it may not only be the room but the associated equipment as a whole. DBT one piece and you may as well DBT the whole thing if you really want to get to the bottom of things.

I remember going to one HTF members home to sample his SVS system and although the subs were wonderful, his system totally out performed mine in all aspects that I could think of and get this, my system probably cost 2 to 3 times as much as his. Since then I've been striving to know every controllable (real) difference making tweak available to me regarding my system. On paper my system was impressive but in actuality I was embarrassed although he never heard my system. If I were to be DBT'd using both our systems and I was told that one of the systems was worth about so and so thousands of dollars and the other was worth 2-3 times as much guess what, I would've picked his system over mine I'm sure. It was that obvious to me.

I think what's more important than DBT is how synergistic and how properly calibrated your system is. Meaning how well do the many parts of your system work with each other. You can say that this pre amp is quiet and how neutral it is but if your quiet pre amp is up to performing but the rest of your system is not, what good is a DBT going to do when the pre amp will never sound as good as advertised because the rest of your system is one big weak link?

 

[Saurav]

However, if you just inserted one tube, and said what tube is this, hell, I could guess but that'd be all it would be... a guess.

Exactly. You seem to be saying pretty much what I'm saying, so where exactly do you disagree with me?

Remember, in any single DBT test, you have A, B and X. You know what A and B are, you're supposed to try and find out what X is. If all you do is play a series of X's and ask the listener to tell what it is, then it's similar to what you said, put in a tube and try to tell which one it is without any comparisions. That isn't a DBT, and if anyone thinks that's what a DBT involves, I think they're mistaken. A DBT, in your case, would be to put in one known tube, listen to it, put in the other known tube, listen to it, then put in a third tube which is either the same as the first or the second (but you don't know which), and then try and tell if it sounds the same as the first one or the second one. And in theory, you should be able to switch instantaneously, which would be pretty hard to rig up in practice.

 

[Saurav]

I think what's more important than DBT is how synergistic and how properly calibrated your system is. Meaning how well do the many parts of your system work with each other. You can say that this pre amp is quiet and how neutral it is but if your quiet pre amp is up to performing but the rest of your system is not, what good is a DBT going to do when the pre amp will never sound as good as advertised because the rest of your system is one big weak link?

That's a very valid point. However, it's not relevant to what a DBT is trying to achieve. A DBT is not meant to decide what is better. It's only used to determine if there is a difference. Think of it this way - let's say you try two components in your system and in your room and you cannot hear a difference at all. Which is better for you? There are many other factors which will decide that - cost, ergonomics, etc. All the DBT has told you is, neither one sounds different from the other. So, you can pick the one you want with the reassurance that you won't be picking the one that sounds worse, because you tried and couldn't tell them apart.

If, on the other hand, you can tell them apart - all you've established is that they sound different. Which one is better? That's still a completely different question, and one that is totally not answered by the results of your DBT.

 

[John Morris]

Exactly. You seem to be saying pretty much what I'm saying, so where exactly do you disagree with me?

Sauruv: I'm not sure I do. But if I do it is in the reason why you need to include an X? Why not just do A then B, then A, then B, then A, then B etc(as long as it takes you to either decide or say they are both sonically equivalent) and if you can determine which you think sounds better, then your job is done. If you think they both sound the same, then your job is done too... you buy the cheaper one. It seems to me that X only exists to try to humiliate and stress the tester. Of course, using the X in a double blind situation will also modify what the tester actually hears due to that imposed stress. The stress involved in the public challenge will modify your ability to hear and how you interpret what you hear. Who listens to music under this type of stressful situation. This part I do know... my original degree(22 yrs ago) is in Industrial Psychology and my senior thesis was on attempting to design a HUD which would provide the most accomodating focal length for a pilot flying in complete darkness under high stress. Unfortunately, we found that a human's dark focus changes non-predictably and variably from the tip of his/her nose to infinity so no set focal lenght could be designed for each pilot. In other words, stress does alter the ability to see at night. It probably also alters your ability to hear and concentrate.

Anyway, if any stress is imposed to the subject making the decisions, his/her ability to make that decision may be impaired and the test is null anyway. Does that make sense?

 

[Shawn Fogg]

John,
" I'm not sure I do. But if I do it is in the reason why you need to include an X? Why not just do A then B, then A, then B, then A, then B etc"
Because that doesn't eliminate bias. When you are listening to X you don't know what you are listening to so any personal biases can't effect the outcome. The only thing you can use to determine if X is A or B is your ears. For some reason only using your ears tends to scare the heck out of some audiophiles.
"using the X in a double blind situation will also modify what the tester actually hears due to that imposed stress. The stress involved in the public challenge will modify your ability to hear and how you interpret what you hear. "
You can do double blind ABX tests alone by yourself in your own system and take as long as you want to do it. There is no stress in this setup, it is an exploration. It is also a big eye opener.
Another important point not yet made is the need to level match the components to 0.1dB. You should do this in any comparison (sighted or not) otherwise the volume differences can and will influence the outcome of the tests. Listen to the Outlaw in 2 channel with it 0.5dB louder then the Rotel and you will prefer the Outlaw. Make the Rotel 0.5dB louder then the Outlaw and you will prefer the Rotel. And the thing is the differences you will attribute to the two units will be in areas like clarity, soundstaging and imaging. You will not attribute it to volume. That is assuming you aren't letting other biases get in the way of what you here. Of course this means that like 99% or more of the comparisons you read about are invalid because the people performing the comparison didn't control this very important variable. Before anyone jumps down my throat for saying this I suggest you try it yourself. To level balance that closely you need a multi-meter and a test CD with a 1kHz test tone.
For more info see:
Link Removed
http://ackthud.com/shawnfogg/pics/temp/ABX.pdf
http://ackthud.com/shawnfogg/pics/temp/ABX_manual.pdf
Shawn

 

[Michael Lee]

#2 definitely.
Do you have any idea how hard it is to get 2 blind people in my home theater at the same time?!?! It is hard enough to get one !

 

[Justin Doring]

Number 2. I don't blindly test gear.

For example, if I was shopping for a pre-amp, I would try a number of them in my system (without changing anything else, of course), and pick the one which sounded best (all else being equal, which is, of course, unlikely). If I could discern no difference (highly unlikely), I'd pick the cheapest one (all else being equal, which is, of course, unlikely).

 

[Dave Moritz]

I would lean toward no 2, the concept of double blind test would suggest that the reveiws otherwise are tanted by opinions and bias. I feel that if you sit in the same room it is easier to evaluate equipment. Because you do not have the problem of direrances in room accoustics to affect the evaluation. I beleive it levels the playing field so it is easier to figure out what dirrerances there are between equipment.

 

[Todd Hochard]

I barely can find the time to watch a movie on my system these days. How in the world would I find the time for a DBT?

 

[John Morris]

Because that doesn't eliminate bias. When you are listening to X you don't know what you are listening to so any personal biases can't effect the outcome. The only thing you can use to determine if X is A or B is your ears. For some reason only using your ears tends to scare the heck out of some audiophiles.

Shawn: Either I'm not being clear or I still don't understand what you are talking about... because why do I need to prove I know X when I hear it? Why do I need to know that? What does that matter to me? All I want to know is if I like A, or B better. I guess I just don't feel the need to try to display any sense of "golden earedness" or exacting audible memory to anyone. I still don't understand why anyone feels the need to display this ability? If you can ascertain through careful A/B listening that you like A better than B, what more is to be gained? Even if I could not identify X as being A, it still would not change my opinion of how A compares to B.I just don't get this X thing at all...

And, with regard to bias: it only exists for someone else. Any bias that you have when you are making a buying decision for yourself is irrelevant to you. Instead, it is a part of who you are, what you like, and even may have helped shape how you hear. Now, if Rotel is paying me or feeding my children, then I can see how intentional bias toward Rotel may be significant. However, if I am searching to buy the best sounding, within my budget, preamp-processor, any previous biases based on experience may actually be helpful in making my choice.

Finally, how do you set up this ABX thing in a blinded fashion in a typical crowded HT rack? Do you use a switchbox? If so, did you do an ABX comparison of the switchbox to see if you could identify it with each prepro and/or without it? If so, did you also do an ABX to see if you could also identify when you used it and when you didn't?

Finally, I did actually learn something in this thread that I am definately gonna use. Using the 1kHz calibration tone instead of a test CD is something I'm gonna do from now on. I not gonna use a multi-meter though; instead I'll continue to use my RS sound meter to calibrate to only within 1dB.

 

[Ian Montgomerie]

Unlike many people here, I have enough experience with scientific testing methods to have lost the arrogance that when I think I hear (or otherwise detect) some difference, that it is actually real. Double-blind testing is great, except that virtually nobody does it. Even a simple blind A/B test in the comfort of one's own home can be great at puncturing illusions that one component is "obviously" far superior than another, but it is quite a pain in the ass to carry out - because you have to get at least two expensive pieces of audio equipment in the same location to try it out. Hard to do when, say, considering mail-ordering some new stuff.

I find the negative reactions to double-blind testing or "engineering" stats in the "audiophile" community extremely amusing. So many people will look at two components which objective, scientific measurements cannot distinguish, and which self-declared audiophiles cannot distinguish under blind listening conditions, but claim that the one that costs 100% more is "obviously" better from their purely subjective non-blind listening, and proceed to pay for it.

What reading audio boards has throroughly convinced me of, is that many audiophiles, who spend tens of thousands of dollars on audio equipment, are spending _well_ beyond the point of diminishing returns. Many are spending at the point where a 100% increase in price will buy you only 1% in perceivable quality. Some are spending at the point where a 100% increase in price won't buy you any perceivable quality. (Where I definitely do define a quality difference as a _difference_, i.e. you would be able to detect a difference at better-than-chance levels under blind conditions).

Convinced me quite handily that a lot of my money would be better spent on other things. "Don't spend beyond the point of diminishing returns" is a philosophy I apply to all my stuff - and it means that I have a heck of a lot of great stuff. Of course, it does take a certain non-competitive, non-status-oriented mindset.

A lot of people want to have THE BEST at something (or at least THE BEST that they could possibly afford). They don't just want to have something great. It drives them nuts to know that however good the thing they have is, they could have paid a little more or bought a different brand and got something a little bit better. Or at least that they think is a little bit better. And thus, living in Silicon Valley, I've met large numbers of people who are paying more to lease their new BMW than they are paying in rent (and this in an area where housing prices are sky-high). And yet they are a pretty average driver, staying around the speed limit and never taking a fast turn. And they think their car is just THE BEST, even if they have never driven other performance cars, or never pushed any car to the limit.

And don't let me get started on the people who think spoilers and 18 inch wheels will give their car TOTAL PERFORMANCE.

A lot of people say "well who cares if the speakers can't be differentiated, I subjectively like the more expensive ones better, and I'm willing to pay for that purely subjective feeling". Well, in a sense you can't argue with that. In a sense, you couldn't argue with someone who says "I just subjectively like to make a big pile of hundred dollar bills, douse it with kerosene, and burn it to fine ash, just for the feeling of it".

But then I notice that often the same person saying this is someone who was arguing last week in another thread for the superiority of some brand of audio equipment over some other brand, and "forgetting" to mention "this is just my purely subjective feeling that could be impossible to replicate under testing, so I have no basis on arguing with your purely subjective feeling if it is totally different". And trying to rationalize their argument with objective measurements, engineering properties, and so on.

 

[Shawn Fogg]

John,

".. because why do I need to prove I know X when I hear it? Why do I need to know that? What does that matter to me?"

Apparently it doesn't matter to you. To me it confirms if I'm hearing what I think I'm hearing. If I like 'A' better then 'B' and can reliably ID 'X' I are hearing what I think I am. If I can't ID 'X' things are a lot more murky.

"Any bias that you have when you are making a buying decision for yourself is irrelevant to you. Instead, it is a part of who you are, what you like, and even may have helped shape how you hear. "

It isn't irrelevant to me at all. Bias has nothing to do with what your sound preferences are, of course you should make your choice based on preference. Bias would be me not giving an active pre-amp a fair listen because I'm a passive pre-amp kind of guy. Or SS vs. tubes or brand X over brand Y. I don't want to let those prejudices influence my appraisal of their sound.

"if I am searching to buy the best sounding, within my budget, preamp-processor, any previous biases based on experience may actually be helpful in making my choice."

And it could just as easily hurt you. Thats why you eliminate it.

"inally, how do you set up this ABX thing in a blinded fashion in a typical crowded HT rack?"

Read the links I already posted.

"I not gonna use a multi-meter though; instead I'll continue to use my RS sound meter to calibrate to only within 1dB."

That is not good enough to eliminate volume differences from coloring the outcome of what you hear.

Shawn

 

[John Morris]

Well, I went back and re-read most of the posts in this thread and maybe I think I found something that would make the X make sense to me... I think? That is that once you play the X, you can still go back and compare the X to A and the X to B until you match X to A or B. Is that correct? So, once you decide you like A or B better and then listen to X, you can then compare X to A and B all over and over again till you match it correctly. I still don't know why you'd do that, since you are still gonna know what X is once you hear A and B again and therefore if any bias existed, it still will exist?

 

[Philip Hamm]

I've never DBTed anything and I doubt I ever will. However, DBTs as research tools are interesting and useful and I'll gladly read about them.

 

[John Morris]

Ian: Reading your post I believe has given me some insight into why some folks believe that you must do DBT ABX testing. But, I'm not comparing a Krell prepro and a Tag. I'm thinking in terms of prepros in the cost range of Rotel, Outlaw and Ref 30. I also almost always try to do the A/B thing. My audible memory is just not specific enough for me to do comparisons between two pieces of gear I've heard weeks apart, that are comparable in quality. For example, when I A/B'd the A prepro versus the B prepro, it was easy to decide which I liked better because the difference was due to noise floor. It was not a minor difference. One prepro has audible noise when I placed my ear a foot from the front speaker, the other one doesn't. When listening to music, I could then carefully listen and hear that that noise floor impacted the clarity of the music coming through it. I absolutely knew which pre-pro I preferred. I didn't need to do an X to validate what I was hearing.

Then again, if two pre-pros are so close as to me being unsure of which I like better after A/B'ing, then I simply choose the cheaper one or the one with better features and looks. Once again, it is simple... at least for me?

 

[Shawn Fogg]

John,

"That is that once you play the X, you can still go back and compare the X to A and the X to B until you match X to A or B. Is that correct?"

Yes. A true DB ABX test consists of many trials. At the start of each trial the ABX box randomly picks X to be A or B. Obviously it doesn't tell you which it is.

Then you are free to switch instantly between A, B and X as many times as you like and for as long as you like. After you have decided if X is A or B you enter your answer into the machine. This completes this trial and moves you on to the next trial which is the same thing all over again. After you completed a certain number of trials you can end the overall test and find out how you did.

You perform multiple trials in a test to see if your answers reach a point of statistical signifigance. The 5% rule is usually used. If your score on picking X correctly could only happen by random 5% or less of the time it is safe to say you heard what you think you did.

"since you are still gonna know what X is once you hear A and B again and therefore if any bias existed, it still will exist? "

Not at all. During the trials you only THINK you know what X is based on comparing it to A and B. If the difference you think you are hearing is true and actual then you will properly ID X. If it is all in your head (because of bias or whatever) you will most likely fail to correctly ID X a statistically valid number of times.

There have been ABX tests performed where A and B are the exact same product... so of course X was too. Those taking the test would swear they were hearing differences between A and B and that they were correctly IDing X. Of course when the results were tabulated their answers were totally random. That is the point of X, it is the control.

Shawn

 

[John Morris]

Anyway, although I've now gained a better understanding of A/V DBT ABX, I still have to ask, back on topic, who feels like they have to do DBT in order to make a decision on a pair of pre-pros like the Outlaw 950 and the Rotel 1066?

 

[chung]

John:
The X is simply one of the units under test whose identity is not disclosed to you (and the tester). Without X, the tester would then have to tell you: this is A, this is B, and so on. You may then believe that A and B are always different. Now when there is an X, you have to use your ears to tell whether X is the A or B that you listened to earlier. If you can identify X correctly and reliably, then indeed there is a difference between A and B. If you cannot, then the conclusion is that you cannot tell A from B. Remember you can take as long as you want to accustom yourself to A and B, with the music or test tones that you prefer.
What happens a lot of times when golden-ears take the DBT is that when they listen to A and B, they can clearly state the difference between A and B (A has a wider soundstage, B is darker, and so on), Now when X is played, they often guess wrong. A lot of golden-ears do not like DBT's for that reason
If like in your example, you can identify one of the units by the noise floor, then there is clearly a perceived difference. The DBT does not tell you whether A or B is superior in this case.