Craig Chase
Gear Guru
Speaker testing, especially when doing an "A" versus "B" speaker test, has been one of the more controversial aspects of this hobby for not just years, but for decades. Most speaker tests involve the reviewer taking a pair of speakers, putting them in his system, listening to some tracks, then comparing them to other speakers to which he listened in the past.
In most reviews, if we are lucky, the reviewer has done some relatively quick swap outs of the speakers being compared, but they are still not instant "A" versus "B", and there are no scientific controls on the test.
It used to be the reviewer would take the time to do a proper test. Unfortunately, this was a long time ago. J. Gordon Holt, founder of Stereophile Magazine, and a pioneer in blind speaker tests, had this to say about this hobby in the 1980's:
The above quote is an excerpt from a fairly easy to understand page as written by Sean Olive in 2009. In this, Mr. Olive posts a graph showing what happens when even experienced listeners compare speakers under both blind test and sighted test conditions. The results are not surprising: When the bias of a sighted test is removed, the results are always different.
I am in my 38th year as an audio enthusiast, and discovered this year just how biased I can be in a sighted test. Last spring, when comparing the $6995 Legacy Audio Signature SE's to the $2660 Axiom M-100's, I found the Legacy's to be the superior speaker when I knew which speakers were playing, and when I was swapping them out by switching wires.
Out of 6 people listening, all 6 of us preferred the Legacy speakers.
When I purchased a pre-amp level switch box that allowed instant A/B switching between the two pairs of speakers, level matched and under blind conditions, the results changed. We found that 3 of us preferred the Axiom M-100's, Two preferred the Signature SE's and one found them to be different, but did not have a preference.
Under these blind conditions, the results were always much closer. I had not seen the above referenced Sean Olive page before these tests were done. In fact, while I had known of Floyd E. Toole's preference for blind speaker testing, I had not read his material until the last couple of months.
Toole, as a relentless pursuer of "what works", would take the testing even further - a true double blind test with the listener not having ANY idea what the speakers are in the test. I have done several such tests this year, and had to post the results before knowing which speakers were being auditioned. It is a most difficult test to take: You are putting it all out there, and for all you know, you are picking a pair of Bose 301's over a pair of B&W 805 Diamonds.
It was with pleasure that I found out my results, under double blind test, were remarkably consistent, and that they were consistent with very well trained audio engineer's results. Without knowing it, I was hearing the same speakers, but instead of an AB to AB set up each time, it was AB and AB ... then BA and BA.
In the testing done between the Legacy Signature SE's and the Axiom M-100's, none of the listeners knew which speakers were being auditioned. Of course, I did. I made no comments when they were listening. All the listeners were music lovers. One has had extensive training as an opera singer. While this was not a true double blind test for the listeners (as I was present), it was as close as one can get in a domestic environment.
What was fascinating under the blind tests is the consistency in the remarks between the speakers. The Legacy's were found to be "more forward" by every listener, while the Axiom's were found to be "more warm". These differences were much less noticed under the blind conditions, as Sean Olive predicts will "always" happen when comparing sighted tests to blind tests.
We are all subject to being biased by sight. I have to smile when I see this sentence typed by a listener ... "I don't need a blind test to know which speaker is better, I can easily hear the difference". I smile because, if this is correct, this listener should embrace the blind test, as his ears will pick the same speaker every time.
To make sure I am giving a lot of disclosure here, I am as biased as is anyone in a sighted test. Bias does not mean evil. It means our eyes are doing a lot of the listening for us.
If you see a review from me, comparing two speakers, I can promise you this:
1. The listening will be done blind.
2. The Speakers will be done in AB AB .. then BA BA for positioning, so they are both auditioned in the same location. In other words, each test will have two rounds, with the speakers repositioned to make sure the room effect is minimized.
3. The order of the speaker in terms of which is first and which is second will be done by another party. The speaker wires are listed as "A" and "B", with 4 channels being used in total. Let's say I am going to compare a pair of Bose 301's to a pair of PSB Imagine B. It's possible the Bose will be the "A" pair of speakers in both tests, just in a different position.
4. All results will be discussed as based on the blind test.
Once the test is done, and the write up is complete, more analysis will be done. As an example, in the case of the Legacy Signature SE's, I found it to be in the same class as the B&W 802 Diamonds. The caveat on this is I have auditioned the Diamonds at a high end store, and the Legacy's at home. They share that trait I have dubbed the "audiophile sound" in a speaker in which the midrange and tweeter are a bit forward in contrast to the bass.
The Axiom's are closer to the Magnepan 1.7's (Magnepan also does extensive blind listening in their speaker development). They are a bit more neutral than the Legacy's ... but other listeners have found the Legacy's to be the neutral speaker with the Axiom's to be a bit on the "warm" side.
I auditioned the B&W's and the Maggie 1.7's at the same store, and it was a varied preference there as well. Some liked the more forward sound of the B&W's while others the Maggies.
This brings us to this thought ... even under blind conditions, we will have different preferences. If you are spending your hard earned money on speakers, TRY to do some comparisons under blind conditions. Get the salesman to put a couple pair of speakers into the system and let you switch back and forth between them with him out of the room.
You don't need the salesman yapping in your ear about all the stuff "Acme" speaker is doing better than "The Other" speaker.
If you are deciding between a pair of ID speakers versus a brick and mortar, order a pair of ID speaker and take them to the store for an audition (unless the store will let you try their speakers at home).
Even if all you can do is listen to one pair for a specific time period, and have a friend switch them, if it's blind, it's your ears making the decision.
And that is the important part ... YOUR ears.
In most reviews, if we are lucky, the reviewer has done some relatively quick swap outs of the speakers being compared, but they are still not instant "A" versus "B", and there are no scientific controls on the test.
It used to be the reviewer would take the time to do a proper test. Unfortunately, this was a long time ago. J. Gordon Holt, founder of Stereophile Magazine, and a pioneer in blind speaker tests, had this to say about this hobby in the 1980's:
The above quote is an excerpt from a fairly easy to understand page as written by Sean Olive in 2009. In this, Mr. Olive posts a graph showing what happens when even experienced listeners compare speakers under both blind test and sighted test conditions. The results are not surprising: When the bias of a sighted test is removed, the results are always different.
I am in my 38th year as an audio enthusiast, and discovered this year just how biased I can be in a sighted test. Last spring, when comparing the $6995 Legacy Audio Signature SE's to the $2660 Axiom M-100's, I found the Legacy's to be the superior speaker when I knew which speakers were playing, and when I was swapping them out by switching wires.
Out of 6 people listening, all 6 of us preferred the Legacy speakers.
When I purchased a pre-amp level switch box that allowed instant A/B switching between the two pairs of speakers, level matched and under blind conditions, the results changed. We found that 3 of us preferred the Axiom M-100's, Two preferred the Signature SE's and one found them to be different, but did not have a preference.
Under these blind conditions, the results were always much closer. I had not seen the above referenced Sean Olive page before these tests were done. In fact, while I had known of Floyd E. Toole's preference for blind speaker testing, I had not read his material until the last couple of months.
Toole, as a relentless pursuer of "what works", would take the testing even further - a true double blind test with the listener not having ANY idea what the speakers are in the test. I have done several such tests this year, and had to post the results before knowing which speakers were being auditioned. It is a most difficult test to take: You are putting it all out there, and for all you know, you are picking a pair of Bose 301's over a pair of B&W 805 Diamonds.
It was with pleasure that I found out my results, under double blind test, were remarkably consistent, and that they were consistent with very well trained audio engineer's results. Without knowing it, I was hearing the same speakers, but instead of an AB to AB set up each time, it was AB and AB ... then BA and BA.
In the testing done between the Legacy Signature SE's and the Axiom M-100's, none of the listeners knew which speakers were being auditioned. Of course, I did. I made no comments when they were listening. All the listeners were music lovers. One has had extensive training as an opera singer. While this was not a true double blind test for the listeners (as I was present), it was as close as one can get in a domestic environment.
What was fascinating under the blind tests is the consistency in the remarks between the speakers. The Legacy's were found to be "more forward" by every listener, while the Axiom's were found to be "more warm". These differences were much less noticed under the blind conditions, as Sean Olive predicts will "always" happen when comparing sighted tests to blind tests.
We are all subject to being biased by sight. I have to smile when I see this sentence typed by a listener ... "I don't need a blind test to know which speaker is better, I can easily hear the difference". I smile because, if this is correct, this listener should embrace the blind test, as his ears will pick the same speaker every time.
To make sure I am giving a lot of disclosure here, I am as biased as is anyone in a sighted test. Bias does not mean evil. It means our eyes are doing a lot of the listening for us.
If you see a review from me, comparing two speakers, I can promise you this:
1. The listening will be done blind.
2. The Speakers will be done in AB AB .. then BA BA for positioning, so they are both auditioned in the same location. In other words, each test will have two rounds, with the speakers repositioned to make sure the room effect is minimized.
3. The order of the speaker in terms of which is first and which is second will be done by another party. The speaker wires are listed as "A" and "B", with 4 channels being used in total. Let's say I am going to compare a pair of Bose 301's to a pair of PSB Imagine B. It's possible the Bose will be the "A" pair of speakers in both tests, just in a different position.
4. All results will be discussed as based on the blind test.
Once the test is done, and the write up is complete, more analysis will be done. As an example, in the case of the Legacy Signature SE's, I found it to be in the same class as the B&W 802 Diamonds. The caveat on this is I have auditioned the Diamonds at a high end store, and the Legacy's at home. They share that trait I have dubbed the "audiophile sound" in a speaker in which the midrange and tweeter are a bit forward in contrast to the bass.
The Axiom's are closer to the Magnepan 1.7's (Magnepan also does extensive blind listening in their speaker development). They are a bit more neutral than the Legacy's ... but other listeners have found the Legacy's to be the neutral speaker with the Axiom's to be a bit on the "warm" side.
I auditioned the B&W's and the Maggie 1.7's at the same store, and it was a varied preference there as well. Some liked the more forward sound of the B&W's while others the Maggies.
This brings us to this thought ... even under blind conditions, we will have different preferences. If you are spending your hard earned money on speakers, TRY to do some comparisons under blind conditions. Get the salesman to put a couple pair of speakers into the system and let you switch back and forth between them with him out of the room.
You don't need the salesman yapping in your ear about all the stuff "Acme" speaker is doing better than "The Other" speaker.
If you are deciding between a pair of ID speakers versus a brick and mortar, order a pair of ID speaker and take them to the store for an audition (unless the store will let you try their speakers at home).
Even if all you can do is listen to one pair for a specific time period, and have a friend switch them, if it's blind, it's your ears making the decision.
And that is the important part ... YOUR ears.
