DVD Shootout... Unbiased ???

Discussion in 'Playback Devices' started by SeanA, Jun 18, 2003.

  1. SeanA

    SeanA Second Unit

    Joined:
    Feb 16, 2003
    Messages:
    329
    Likes Received:
    0
    Trophy Points:
    0
    In a post (now appears to have been corrupted), someone mentioned that the Denon DVD-2900 was tested in the "DVD Shootout" on the Home Theater Hi Fi website. I checked it out and noticed right away that they have an ad for Denon DVD players and the Faroudja chip. How can this website be completely impartial when they are running ads of this nature ? Is it not surprising that many of the top performers are Denons and Panasonics with the Faroudja chip ? I know the website needs the income from such ads, but in my mind this seriously undermines the credibility of their results.

    Sorry to be pessimistic about the shootout results, but my experience with a Denon DVD-900 makes me wonder. I bought the 900 expecting to have better video performance than my current Toshiba SD-4800. The Denon rated about midway on the shootout and the only two Toshibas tested were at the end of the pack. Well I returned the Denon after just one day. I ran the same scenes from LOTR on both players, and I swear the Toshiba had a cleaner and sharper image. Besides, I think the "chroma bug" is over-blown. I have yet to notice the bug in my Toshiba. And the Toshiba had better controls, menus, remote, etc. I think the sound quality from the Denon was its only edge.
     
  2. Dzung Pham

    Dzung Pham Second Unit

    Joined:
    Mar 10, 2001
    Messages:
    271
    Likes Received:
    0
    Trophy Points:
    0
    I think you're off-base with your remarks about Secrets being biased. The shootout ratings are primarily based on deinterlacing. You have to read the individual reviews to find out the different aspects of video quality. If you read the individual review of the 900, they state that there are problems with the Denon 900's frequency response. However, they only have one box for frequency response, and like ten of them for deinterlacing.

    Anyway, they clearly state their criteria and this was formulated way before Panasonic and Denon dominated the ratings. A couple of years ago, the Camelot dvd player was the only highly rated player out there. You should also note that Secrets slams the new Pansonic dvd-recorder's video performance (which uses the same chipset as their new dvd players). The reason why Denon tends to score well is that they seem to design their players around the ratings. I wish more manufacturers would do the same.

    Anyway, I think if you said that the scoring system could weight things differently, I wouldn't argue with you. There are many different characteristics that you are trying to summarize with one number. Everyone is going to have a different opinion as to what is more important. But to say they're biased towards certain manufacturers is IMO, way off-base.
     
  3. SeanA

    SeanA Second Unit

    Joined:
    Feb 16, 2003
    Messages:
    329
    Likes Received:
    0
    Trophy Points:
    0
    Dzung,

    I appreciate what you are saying but I never did say outright that they are biased, only that it is difficult to have complete faith in the final results when the Denons and the Faroudja-based players are all at the top. And I do realize they are reviewing just the video quality, but that is what was most important to me in a DVD player and what I relied on in purchasing the Denon. Granted, I did make the mistake of assuming the 900 would have near the same video performance as the 1600. I am just not sure now that I want to buy the 1600, even with all the positive comments on this forum. I am not sure I will be able to perceive an improvement in video quality over my Toshiba.
     
  4. Dzung Pham

    Dzung Pham Second Unit

    Joined:
    Mar 10, 2001
    Messages:
    271
    Likes Received:
    0
    Trophy Points:
    0
    Sean, I just don't see where there is much room for subjectivity in their reviews. They have certain tests, and in most cases, the player either passes or fails. Their final score is a function of the number of passes and a weighting value on each particular test. So you can argue that you don't like their formula or their tests but I really doubt that you would obtain a different score if you were to follow their exact criteria. The Secrets Shootout is about as objective as you can get.

    And although you did not say outright that they are biased, you did say "in my mind this seriously undermines the credibility of their results", which is a pretty strong statement. However, I will say that nobody should blindly follow the scoring of the Shootout. It is always best to audition for oneself, as you found in your comparisons with the Toshiba 4800 and the Denon 900.
     
  5. Kevin C Brown

    Kevin C Brown Producer

    Joined:
    Aug 3, 2000
    Messages:
    5,712
    Likes Received:
    0
    Trophy Points:
    0
    Out of all the places where I see DVD players being reviewed: all the HT mags, audioreview.com, ecoustics.com, etc, I view Secrets as the most unbiased, simply because their reviews are *mostly* based on measured criteria rather than wanker phrases like, "the best video in a player of this price point", "deeper blacks," "more vivid colors," "a vibrancy that I just have not seen in past machines," etc. [​IMG]
     
  6. John Kotches

    John Kotches Cinematographer

    Joined:
    Mar 14, 2000
    Messages:
    2,635
    Likes Received:
    0
    Trophy Points:
    0
    Sean,

    I was wondering who was going to come along and be the person that questions the bias/integrity of Secrets. Thanks for participating.

    Would you rather read things like "The color saturation of (insert player of the month flavor here) was so intense that I felt like I was actually in the movie. The image was so 3d that when (insert scene from movie of the month here) I was overwhelmed by a palpable dimensionality heretofore unseen with my system." I know I don't. While such writing does show the skill for putting together sentences, truth be told that don't tell you a damned thing about how the player really performs.

    The staff at Secrets have laid out a suite of tests that have been refined over time as they have gained knowledge. In addition the results are objective and repeatable. You could even run them yourself if you wanted.

    Please cite for me exactly how these objective tests are biased towards Denon and Faroudja. I'd like to hear your reasoning.

    The truth is that the tests aren't biased for or against any manufacturer or chipset. The tests are biased for players that handle common encoding errors well, in addition to correct chroma upsampling (ie no chroma error).

    Of course I'm biased too, since I used to write for Secrets, I am a friend of Stacey's and an acquaintance of Don's.

    Regards,
     
  7. ManW_TheUncool

    ManW_TheUncool Producer

    Joined:
    Aug 18, 2001
    Messages:
    6,015
    Likes Received:
    40
    Trophy Points:
    6,610
    Location:
    The BK
    Real Name:
    ManW
    Well, my post was lost to the DB corruption as well, but basically, I agree w/ the others.

    Now, as others have pointed out (or implied), that does NOT mean the better scoring players are always best for you. There may certainly be cases where a lower scoring player might be best for various personal reasons. Feature set can come into play as well.

    The great thing about the Shootout is that it offers objective info about picture quality. You may disagree w/ the weights, and thus, the scoring, or that it doesn't factor in other aspects of PQ enough, eg. the frequency response curve. But you can still draw a well-informed conclusion based on the objective data and/or even do the tests yourself to see how important they are.

    RE: the ads, well, I'd think if I were making players (or chips) that do great in the Shootout, which is uniquely highly respected in the AV community, I would want to buy ad space on the site too, especially if I were NOT a big, mainstream, mass-market manufacturer, eg. Denon, Faroudja, etc, and want the Shootout to prosper.

    And if I were making players that do badly in the Shootout and don't really care about it (or even hate it) because I don't think my customer base cares either, eg. Sony, Toshiba and now seemingly Panasonic also, then why would I buy any ad space on the site???

    FWIW, I don't own any of the top performing players (and have owned my share of Tosh players), so I'm not exactly a Shootout groupie saying all this either. [​IMG] Well, ok, I do own a Philips 963sa, which has a Faroudja deinterlacer, but that's not the only reason I bought it. :wink:

    _Man_
     
  8. LanceJ

    LanceJ Producer

    Joined:
    Oct 26, 2002
    Messages:
    3,168
    Likes Received:
    1
    Trophy Points:
    0
    Specs are only good to a point, then you have to use your own eyeballs to make the final judgement.

    LJ
     
  9. SeanA

    SeanA Second Unit

    Joined:
    Feb 16, 2003
    Messages:
    329
    Likes Received:
    0
    Trophy Points:
    0
    Well I must say that there are some very good points made here, especially with regard to the effort made by Secrets to create objective tests. I must admit that I only briefly reviewed the test criteria... maybe I need to go back and take a closer look. And maybe I will re-consider that Denon DVD-1600 ?

    But I guess the reason the advertising on their website bugs me is because it does seem like they made some effort to be scientific in their analysis, sort of like a Consumer Reports. I expect the typical ad magazine to make vague reviews because they don't want to tick off any current or potential advertiser.

    One suggestion I do have for Home Theater Hi Fi is to test and show more of other brands in their progressive DVD Shootout. I have not counted the numbers, but it appears the Panny's and Denons take up half of the spots or more for progressive players during 2002 to 2003.
     
  10. greg_t

    greg_t Screenwriter

    Joined:
    Jan 18, 2001
    Messages:
    1,650
    Likes Received:
    0
    Trophy Points:
    110
    QUOTE]One suggestion I do have for Home Theater Hi Fi is to test and show more of other brands in their progressive DVD Shootout.[/quote]

    This has been covered by the secrets guys in numerous threads. They are NOT provided players to test by all the manufactures. They have many times in the past spent their own money to buy players to test. There was a thread not too long ago when someone asked Stacey why they didn't test brand X player. The response was that they were not provided one by the manufacturer, and he already has a closet full of players and won't buy anymore. They obvisouly can't spend all of their own money to test multiple players. Stacey and Don do have full time jobs besides doing the secrets tests.
     
  11. ManW_TheUncool

    ManW_TheUncool Producer

    Joined:
    Aug 18, 2001
    Messages:
    6,015
    Likes Received:
    40
    Trophy Points:
    6,610
    Location:
    The BK
    Real Name:
    ManW
    They do welcome loaner players from individuals as well, but I guess not too many step up to the plate for this.

    _Man_
     
  12. Kevin C Brown

    Kevin C Brown Producer

    Joined:
    Aug 3, 2000
    Messages:
    5,712
    Likes Received:
    0
    Trophy Points:
    0
     
  13. greg_t

    greg_t Screenwriter

    Joined:
    Jan 18, 2001
    Messages:
    1,650
    Likes Received:
    0
    Trophy Points:
    110
    I think that's because audio quality is so subjective. Just look at the ongoing Dolby/DTS debate. For video, however, the secrets guys have clearly outlined definitions and standard testing procedures. Their video testing is far more objective than audio testing could be.
     
  14. John Kotches

    John Kotches Cinematographer

    Joined:
    Mar 14, 2000
    Messages:
    2,635
    Likes Received:
    0
    Trophy Points:
    0
    Kevin said:
     
  15. John Kotches

    John Kotches Cinematographer

    Joined:
    Mar 14, 2000
    Messages:
    2,635
    Likes Received:
    0
    Trophy Points:
    0
    Sean said:
     
  16. Michael TLV

    Michael TLV THX Video Instructor/Calibrator

    Joined:
    Mar 16, 2000
    Messages:
    2,909
    Likes Received:
    1
    Trophy Points:
    0
    Location:
    Calgary, Alberta
    Real Name:
    Michael Chen
    Greetings

    Precisely. Many seem to think that there is some mystery philanthropist handing out money to these people to do this work. Truth is ... it's their time and their money and no one is paying them anything.

    So easy to sit back and complain about this or that ...

    Regards
     
  17. Daniel Mai

    Daniel Mai Stunt Coordinator

    Joined:
    Oct 2, 2001
    Messages:
    225
    Likes Received:
    0
    Trophy Points:
    0
    More echos of supports for Secrets here.

    Sean can take it however he wants to take it, but I find Secrets reviews, shootouts, and Q&A to be very enjoyable and informative reading. I check on their site frequently for anything new and I find it to be the best available.
     
  18. SeanA

    SeanA Second Unit

    Joined:
    Feb 16, 2003
    Messages:
    329
    Likes Received:
    0
    Trophy Points:
    0
     
  19. ManW_TheUncool

    ManW_TheUncool Producer

    Joined:
    Aug 18, 2001
    Messages:
    6,015
    Likes Received:
    40
    Trophy Points:
    6,610
    Location:
    The BK
    Real Name:
    ManW
    Sean,

    In general, how much their objective info translates into your actual appreciation of differences will still depend largely on you. So in the end, just like Lance said above, you must still need to make the final judgement on your own.

    Some people probably do put way too much faith in the Shootout. But we are not machines, so no matter how solid and objective the data is, it must still be filtered through our own personal tastes, preferences and criteria.

    For instance, will you really be all that bothered by the small amounts of deinterlacing artifacts on 95% of the mainstream movie DVDs? If you never bother w/ TV shows on DVD, then that already eliminates a lot of this problem. Certainly, many people are not bothered by average quality deinterlacing.

    And something similar can be said for the chroma bug as well. Some people just don't notice it at all for whatever reasons or may not be bothered by it. And there are other criteria in the Shootout that many people either don't notice or can easily ignore, eg. Y/C delay.

    The best thing to do is to see for oneself what the Shootout is really testing for. Then go from there in deciding how to weigh their results.

    And if you have a great dealer, maybe you can demo all the players that interest you using the various tests outlined in the Shootout on a display comparable to your own. Then you can be quite certain of the results and how much they really matter to you.

    _Man_
     
  20. Kevin C Brown

    Kevin C Brown Producer

    Joined:
    Aug 3, 2000
    Messages:
    5,712
    Likes Received:
    0
    Trophy Points:
    0
    Greg- Yeah, but if you look at some of the past complete reviews on the site, they actually do a nice job looking at freq response, distortion, etc, for audio performance too. For example, for some of the older Pioneer players they've reviewed, they measure and show the fallacy of any improvement in audio quality with any type of Legato processing.

    John- Obviously, I know that. [​IMG] But I think that people not well versed in their criteria, simply look at the ranked colored ratings chart, and miss that it is only video-based.
     

Share This Page