Review Scores (1 Viewer)

[Trevor Schell]

Hi!

I subscribe to the DVD ETC and really appreciate the selection of gear that is reviewed.
I am just curious as to why almost every piece scores usually a 95 or higher?

 

[Tim Tepas]

Trevor,
This issue was addressed in a previous thread a while back, so I'll try to give you an accurate & concise answer. First, each piece of equipment is tested against similar units in the same price range. Therefore, inexpensive products can score as good or better than others that are in a higher price point. Conversely, high-end products need to really shine to out perform units from another company. Secondly, the editorial staff seeks out the latest and greatest products for review, in addition to manufacturers offering their best equipment to be scrutinized by the staff. No manufacturer would want a product put through a review process that can't hold its own. We tend to get the best equipment in for review because companies want us to analyze their better gear and we request the stuff our readers would most want from established companies, not cheap third world country junk.

Does this mean everything that passes over our desks is above average (a score of 90)? Of course not. I try my best to find the flaws that aren’t obvious to the casual a/v enthusiast and give a good overview of the pros & cons for the target consumer to make an intelligent buying decision. I’ve given out a few sub 90 scores on products I feel the manufacturers needed to do some redesigning on. Also, only a few products were scored above the "Top Choice" threshold of 95 by me. If we gave everything a high rating because of their name, status or advertising standing, we would loose all credibility with our most important customers- you the reader. I’m sure the companies that sell these items wouldn’t want us to just shine on an inferior product either.

So is every consumer concerned with multi-format playability or a chroma bug-free DVD player? Probably not. Just as some Plasma television shoppers aren’t concerned with brightness levels as much as price. This is why we provide detailed information in the article to back up our impression of what a product deserves as an overall rating. It’s up to our readers to decide what positives and negatives are most important to them in their application.

I hope I’ve helped!

 

[ManojM]

Also, to put it into perspective, we don't review every single piece of gear out there- this would be impossible. We are probably guilty of cherry-picking gear that we suspect will be good or perhaps outperform the competition- I know that I certainly push harder to review pieces that I think are worthwhile. This creates a tendency towards higher scores.

 

[Mika-H]

Could either of you guys respond to my "Bluejeans cable review" thread?

thanks

 

[Jack Briggs]

Mika, kindly as a matter of Forum decorum don't bring attention to another thread if it's not related to this one. Thank you.

 

[Mika-H]

Seeing as this thread was about DVD Etc. reviewing procedure, I thought it was related.

 

[Jack Briggs]

Not to belabor it, but this thread pertains to the scoring system the editors use to rate reviewed products instead of the testing methodology used in assessing cables and interconnects.

Clint et al, care to check out Mika's post? (So there!)

 

[Bob~M]

Hello DVDetc,
I wondered why your magazine seems to like everything! When I saw the first issue of your mag, I thought you were some kind of industry publication; and "advertorial" if you will.

My personal opinion: I often learn more from negative or mixed reviews that positive ones. I think you are doing your readers a disservice by publishing reviews of only the best equipment. Your readers are also looking to you to help them avoid high-profile or heavily promoted gear that doesn't measure up. You can't help your readers separate the wheat from the chaff if you only mention the wheat.

I think DVDetc would be a much more interesting magazine if you simply selected what you wish to review - based on new technology, profile/availability, feature set, etc - and then published your conclusions, good, bad, and ugly. To do anything else is to risk the impression that you're afraid to call 'em as you see 'em because you might risk advertising revenue.

Oh, and please find a proofreader. I'm sure this is just growing pains, but I've never seen a national publication with as many typos as DVDetc.

Best of luck,
Bob

 

[Trevor Schell]

Thanks Tim!

I did not realize the ratings were divided by catagory.
I failed to recognize the different shaped symbols around the rating score.

The score system is unique and does make sense.

Good Job!

 

[Clint Walker]

Hi Bob,

First as this pertains to typos the title of the magazine is DVD ETC. Magazine not DVDetc. Being partial to perfection I thought you may want to know this for future reference.

As timing would have it, we did just replace a copy editor. However, I will point out that with two magazines in the stable, there are over 1.6 million words to be read in a year between two people. A typo is something we will always face be it a probablity of chaos-factor or an acceptable percentage--margin of error.

Getting to your idea that we "seem to like everything" makes me wonder if you missed reading reviews such as those of the Pioneer and Apex, McIntosh and others. Regardless, I don't beleive that people buy magazines to read negative reviews. In fact, if a product fails to meet a minimum criteria, then I am more prone to pull it--why waste the paper?

Perhaps you do not understand our scoring system? We have run many reviews of products that scored "below average" (90).

Interesting how you are not arguing our specific findings on a given product, but rather asking why we don't pick them apart more. Actual performance is more black & white where as nit-picking the remote, menu, finish, etc. is more a matter of opinion. We do not call in products that we know are defunct and likewise, manufacturers will not send us products that they know are defunct.

I hope this adds some understanding to your query.

-Clint

 

[Trevor Schell]

Getting to your idea that we "seem to like everything" makes me wonder if you missed reading reviews such as those of the Pioneer and Apex, McIntosh and others.

The reason the perseption exists is that we are used to seeing
HT gear receive ratings in the 70's and 80's in other magazines. DVD ETC. seems to rate most units in the 90's no matter what the price catagory they fall into.

I understand that you review only the cream of the crop gear , so that may be why we see the higher scores consistantly.

Thanks,

 

[Clint Walker]

I wouldn't say we only review "cream of the crop gear" by any means. We've reviewed everything from $60 Mintek DVD players to Apex televisions, to the higher end gear. Our scoring system is defined in every issue--few take the time to read it or understand it. They are used to other magazines rating systems and don't understand that 90 is the average--anything below 90 is below average--and I personally wouldn't buy anything below average. Some magazines don't even score their reviews!

 

[Trevor Schell]

Our scoring system is defined in every issue--few take the time to read it or understand it.

I was guilty of that myself.
In fact now that I understand the scoring structure, I find I still have to look back at the score symbol descriptions so as to recall what catagory of gear they fall under in many cases.

Perhaps eliminating the Symbol structure and simply stating the catagory or price range of the catagory may enable your customers to better understand the rating system vs what most of us are accustomed to.
EG..High End/High Quality/Mid Quality/Budget Quality

Thoughts?

 

[Clint Walker]

Hi Trevor, The symbol/color-code is what consumers look for. For example, if they now their budget warrants high-end choices, they will look for reviews of Black Diamond components. Entry-level budgets will look for Green Circles and those Leprecahns will always be after me clovers! (Sorry, I couldn't help myself.) I put 3 hours of work into this scoring system--how can it be flawed?! (I'm kidding of course.) Does my madness make any sense? Anyone?

 

[Bob~M]

Clint,
Touche on the name of the magazine! Maybe I only value perfection in others ;-)!

First, I do wish DVD ETC. all the best; it can only be good for everyone to have another healthy mag out there. I fear I was a bit critical. Maybe I can be more constructive (and I've picked up a couple of issues at the newsstand, so I haven't seen all your reviews).

Maybe I am not part of the audience you're after, but I certainly DO read a magazine in part for the negative reviews. It's not a waste of paper for me because I become aware of the pitfalls of poor design and implementation, and know what products to avoid. Manufacturers should be held accountable in the press for releasing shoddy products.

And I would say that in this era where, frankly, most products are at least competetent, buying decisions are often based on what might be considered "nitpicks". I think many buying decisions are based on ergonomics, remote design, menus, etc, the "little things". I would venture that most of the electronics purchases I've regretted have not been because of performance, but because of a lousy user interface or a relatively minor annoyance. I really like a reviewer to "pick apart" the products he or she reviews - it makes me a more informed consumer and a happier post-purchase camper.

There are things I do like very much about your magazine. I like the "let's make sense of this new technology" articles. I like the layout and presentation. Your writing is refreshing and straightforward, and blessedly free of audiodrool rhapsodies and philosophical meanderings of some of your competi...er...peer magazines.

So to sum it up, I'd say do the nitpicking for me, so I don't have to do it myself after buying the product. And never be afraid to be opinionated about what separates the merely competent and good-for-the-money from the outstanding. Vigorously shame the average and praise the best.

So really, all the best! Follow your vision and you'll be fine.
Bob

 

[Clint Walker]

Bob, You're a gentleman and a scholar. I agree that it is the subtle nuances that separate the average from the "worthy of purchase" in today's manufacturing climate. Sometimes taking the "simple" approach leaves us with little option to calling out technical quirks. Though we address important issues (such as the chroma-bug) and have called out every single player with the defect, where others have overlooked, or chosen to ignore this defect all together.

You may also find it interesting that I did not pull editors already of this industry when we launched this publication. I chose instead to seek out "enthusiasts" with a fresh approach (and non-inflated ego) to review gear. One of our writers, Greg Robinson was even recruited off of the HTF.

Some say I was creating a magazine for the people, by the people. I can't disagree.

I will also tell you that the feedback I have received (from members of this forum) continually help us to become a better publication. I have even mentioned this credit in my Fade In (Ed Note) in the past. We will never be on cruise control--we are always working to best ourselves.

A sincere thank you for your feedback.

 

[Greg Robinson]

My ears were ringing, so I thought I'd come see what Clint was saying about me now. Bob, you make an excellent point, and it's one I often think about when reviewing gear. I can't tell you how often I've returned something, or sold it to unsuspecting ebayers (sorry), because of a miniscule annoyance that grows bigger with every passing day. Whether it be a button which is frustratingly absent from a remote, a noisy tray mechanism, or the inability to back up in an on-screen menu - I promise to do my part in revealing these little quirks that we all consider to be not-so-little. Thanks for your support.