The new enemy of the HD Formats is Noise Reduction!

Re: The new enemy of the HD Formats is Noise Reduction!

Funny, because so often I find that the *trailers* on DVD and Blu-ray Discs look a bit better than the feature film presentation... most often because they have *not* been put through the paces of additional processing that so many techs seem to think benefit the feature presentation. (I'm not suggesting that trailers are processing-free, just that they tend to receive much less video DSP than what gets applied to the full-length feature).

Re: The new enemy of the HD Formats is Noise Reduction!

Quite possibly. Don't get me wrong I'm not a fan of processing for processing sake. Just when it's needed to fix problems that crop up in the process of going from a chemical medium to an electronic one.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

I think you all need to read this to get a better understanding of the process from the original negative in the camera to projected film to home video formats. Read both parts 1 and 2. It helps if you are already familiar with color spaces, MTF, and sampling theory. What I found most interesting was that while the original negative has very significant resolution and contrast, by the time it is put on quality projected film stock and shown in a theater, the color space, contrast, and resolution of the projected image isn't nearly as superior to HD that film buffs assume it is, but it can be superior. And that is assuming high quality prints and not the crap you see in the local multiplex. It also talks about what is necessary for digital projectors to effectively replicate a film like look uniformly. Until 4K digital projectors are the norm, most people for most films will see a more accurate presentation of the director's vision on HDM than at the theater.

Re: The new enemy of the HD Formats is Noise Reduction!

But apparently such a reasonable philosophy doesn't apply to the audio portion given the posts in this thread and the Warner thread.

Re: The new enemy of the HD Formats is Noise Reduction!

What are you talking about? Also, this is not a philosophy so I don't understand where you're going with that comment?

Re: The new enemy of the HD Formats is Noise Reduction!

Quite true. What you see in the theater is almost never even close to the potential that 35mm film has which is something in excess of 4k. What is seen at your local multi-plex is probably closer to 2k or less as it is several generations away from the original camera negative. However even at 2k that is somewhat higher resolution than HD. And HD can't really reproduce all the colors that film can, even when the film is 5 or 6 generations out. HD has a pretty limited color space.

Of course film being an analog medium, numbers like 4k or 2k are really just estimates of it's resolving power.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

I understand Doug, but to be honest I don't think we are talking about the same things here, the DNR and EE I am speaking of is not what you are speaking of in regards to the process to get to what we see on HD/BD etc.

it seems you are talking normal processes and I am talking of abnormal type stuff

-Gary

Re: The new enemy of the HD Formats is Noise Reduction!

I'm entirely unconvinced by any arguments that propose EE and DNR as helpful tools unless one is willing to admit that optimal solutions are not intended in the first place for financial or other reasons or we are talking about special situations with special needs (such as film restoration or salvaging problematic shots).
And let's not confuse EE and DNR with grading, please. They are not the same at all. Of course all scanned material from film needs to be graded. Does it need EE and DNR as well? And what kind of EE and DNR?
Let's see. The amount of detail/sharpness a film has is defined in the negative. So, first, if the detail/sharpness on the negative is considered insufficient the first solution is obviously to wonder why that is and start photographing your films the way you really should if you want more detail/sharpness. There are many options to change the MTF of your camera chain. But let's assume you have that soft negative and have to deal with it. There are many possibilities to do a HD transfer or DI and sharpen the look without adding obnoxious white haloes or ringing all over the place. And it is this which we usually call EE here and abhorr. Use more intelligent sharpening algorithms that know a thing or two about textures and edges, inside and outside, and can do a sharpening job that does not scream 'video EE'. Do that and some sharpening is indeed acceptable and potential beneficial. But again, we do not protest against this. We protest against stone age tools giving every high contrast outline an ugly artificial haloe.
The same goes for film elements that are derived from the negative and often used instead of the negative. MTF goes down and the perceived need to enhance goes up along with it. Again one has to question the decision to not use the negative when sharpness and detail are so crucial. It is the optimal source for this and anything else. But if the negative is unavailable then again use smart tools for sharpening and not stone age ones. And the complaints will go away, as long as the result looks pleasingly natural and not artifical.
Is your imagery too soft because you filtered the hell out of it (during compression or noise processing)? Is it too soft because you used outdated scanning equipment? The proper solution is obvious, is it not?
Now for DNR and noise/grain. The argument was that you want to make things look more like a print so video needs some noise reduction. Well, if the intent is to make the video look like a properly graded print from the original negative then I'm sorry but printing from the negative to a print does not apply the kind of DNR we usually use in the digital domain. Printing reduces HF content over the spectrum (as seen in the MTF) and adds some ADDITIONAL finer grain to the coarser grain of the negative, dithering the latter. It also manipulates color and gray scale and gamma for a pleasing end result (corresponds to the digital grading part which always needs to be done). Especially it does not add any DNR artifacts as we know them from digital tools.
Why is the video too noisy? Is the film element too noisy? Is the telecine/scanner adding noise we don't want? In the first place, ask yourself, why did we shoot the film like this if we think it's too grainy? Could we not have avoided this? If the telecine/scanner adds the noise, why did we use that specific device? Today we have telecines and scanners that add negligible noise to what's on the film elements. Did we try to save money and now have a noise problem?
Did we apply very aggressive grading and brought out a lot of noise? Did we digitally grade beyond what traditional analogue copying can achieve and stretch the original photography beyond the breaking point? If so, why? And, especially, why in more cases than the occasional problematic shot?
Do we find a print from the negative too grainy as well? If so, why the heck did we shoot on (this kind) of film and with this kind of exposure to beging with??
Whatever it is, if the video is too noisy the proper fix is somewhere else in the chain and not a digital sledgehammer filter to erase the sins commited elsewhere.
But let's assume the video or DI is noisy and we simply have to live with it for now. We can't make a better version without fiddling with it in the digital domain. What should we do with it? Certainly not apply bad quality DNR. You can remove noise/grain to some degree with minimal side effects, but it's not cheap and it's not real time. Again, if you want to save money and fast results, you can use real time and artifacty DNR. It's that kind which is criticised here. We simply don't want the film look destroyed. You can shoot the same scene with fast and slow film. You can have it grainier and with little to (almost) no grain. If the DNR would go from one to the other while keeping the film look intact there would be much less to no complaints. But it usually does not do that. Instead it adds smearing or flickering, unnatural snapping in and out of focus, waxy or strangely messed up skin textures and all kinds of anomalies that you do not see on film that is not digitally manipulated. And for what? So some grain haters don't have to use the DNR in their display chains or some studios can get away (maybe) with lower bit rates and less compression artifacts on maxed out discs with tons of supplements and 10 soundtracks on top? That is not what I call HD as in high definition. Or high fidelity to the film source.
Careful high quality grain filtering with minimal side effects for selected shots of a film (especially if a film needs restoration) has its place in the digital mastering world. Factory style DNR or EE of whole films as fast and cheap as possible on the other hand to accomodate bit rate needs or public taste on the other hand is an abomination and perversion of the terms HD and high fidelity. It's completely unaccpetable to everyone who wants to see authentic film in digital form.

Re: The new enemy of the HD Formats is Noise Reduction!

That is true. Better terms would have been excessive EE and excessive DNR, including in the title of this thread ( ).
As said, there's nothing wrong with both techniques, as they need to be used to restore the original image as the director intended/created in cases when it was degraded by another process.

However, it's terribly wrong if they are used in excess, either by laziness or negligence, and the result is visible in a negative way.

I also need to repeat that we must be careful when judging such effects: if the "EE halos" were present in the original film image (e.g. caused by background projection or analog enhancing techniques), then we shouldn't blame the digital transfer/processing of course.


Cees

Re: The new enemy of the HD Formats is Noise Reduction!

Be more specific, please. The strange effects are due to what?
- Grading?
- Compression?
- Scanner/telecine anomalies?
- ?

Re: The new enemy of the HD Formats is Noise Reduction!

I thought I was already pretty specific. First of all I haven't been talking about compression at all. I'm talking strictly about the film transfer process. Grading is simply color and density correction.

There are times in the telecine process even on the best of equipment, and its actually fairly rare, where the grain of the film will interact with the scan lines of the video system. It's actually more pronounced in HD and it can cause strange patterns to show up as the grain slips in between the scan lines. Patterns that don't look anything like normal film grain. In this case a little judicious use of DNR can help the film look more natural.

I'm not talking about using a sledge hammer to make the film look super slick with no grain. I happen to like the look of film grain, but only if it looks the way it is supposed to.

To say that DNR and EE are bad is kind of like saying morphine is bad. Yeah if your addicted to it, but if you are going in for an operation, you might want a little.

Now the other thing is, has anyone ever thought that some of these films might have had extensive use of DNR on the home video version, because the director didn't like the way the video looked and asked for it? Frankly most directors are not very technical and just say "I don't like that, get rid of it." So the technicians do their best to give him or her what they want. This is one of the reasons that so many films lately, for example the Pirates films, don't look much like film at all. Though in that case the processing seems to have been done in the DI.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

I think a director will release his film how he wants it to look, for example Swordfish has processing to remove film grain and boost colors, BlackHawk Down has severe grain, it was the same way in the theater for both, bumped colors and shiny as plastic for Swordfish and grainy as heck for BHD, thankfully they look the same way on BD/HD

is the transfer of film to digital that unreliable of a process, I mean really? that is way you make it sound Doug, like it barely works and is in need of all this help, thats the first I have heard of such struggles in this process, you make it sound like it is a total struggle for a film to even resemble how it looked on the original film showings, that ties into the first comment, how often is a director needed for the stage of transferring the film to digital

again someone needs to explain to me how a DNR filter that doesn't remove high frequency info works?

-Gary

Re: The new enemy of the HD Formats is Noise Reduction!

Gary (and others),

There are always compromises involved w/ these kinds of things. The idea should be to go for the best set of compromises to yield the best presentation possible -- and such can often be worked out on a case-by-case basis. It should not be hard to accept that techniques like EE and DNR (and others) are used to some degree towards that end. As long as they are not excessive and do not produce more/worse problems than they solve, that's what matters most. And really, I'm sure progress w/ various techniques along w/ the technologies involved will change things over time as well.


So yeah, DNR *will* generally remove some HF details. However, that's a gross oversimplification of what it really means in practice -- as Doug tried to explain. The goal should be to apply DNR so that the result is better than not applying DNR at all. Why would anyone want to apply whatever technique/processing (be it EE, DNR, color adjustment, etc) just for the sake of doing it or to yield something worse than not doing so anyway???

Having said all that, really, IMHO, we're not all here seeking to become telecine gurus. We shouldn't need to understand every little detail before we can accept that certain techniques are needed in the process. Yes, it's nice to know more, but at some point, it can all get in the way of our real goal as cinephiles, etc. If you're really all that interested in all the nitty gritty stuff behind the process, perhaps, there are better resources/forums toward that end. Not saying this stuff shouldn't be discussed here of course, but just saying that if you want more definitive answers, etc., I don't know you'll get them here.

For myself, I'm just speaking from what I know from the digital still photography domain as an amateur photog. And having spent some time processing photos and learning from others in that domain, not much surprises me about all this stuff. If you're really interested, but don't want to dive into filmmaking/telecine work -- and I don't suppose it's easy to do as an amateur -- perhaps you might want to consider amateur (still) photography as a serious hobby both to help inform this HT/cinephile hobby and for something else rewarding on the creative side of the equation. Personally, I do find amateur photography, especially w/ all the transition to the digital realm, to be a very nice complement to my interests in the cinema in many ways. And FWIW, that's actually part of what my sig refers to too in case anyone's wondering.

_Man_

Re: The new enemy of the HD Formats is Noise Reduction!

OK. Having said all of the above, let me also add that it sounds like our goal in this discussion is to become informed enough so that the HT/cinephile community can effectively voice our concerns to the studios about the quality of their HDM releases. If we see nasty EE and DNR effects in certain releases, then of course, we should let them know we want better.

FWIW, before I upgraded to a 53" HD RPTV 5 years ago, I hardly noticed any nasty EE artifacts from DVDs on my 32" 4x3 CRT (at too far a viewing distance) even though I had read somethings about them in reviews and such. I guess the studios figured strong doses of EE work well for smaller displays (and for the masses who wouldn't notice such things, especially on smaller displays, anyway). But as soon as I upgraded, I was like what the heck is that?? I still remember wondering if something was wrong w/ my new RPTV when I saw the nasty EE effects on DVDs like Lilo & Stitch and Men in Black 2, especially around sharp objects like credits. Took me a couple days to realize that's probably what it was. And yeah, I don't want that on my DVDs and HDM, but I also don't need to understand all the nitty gritty stuff behind the process to dislike the nasty effects and want the studios to do better than that. Of course, after spending some time in amateur photography over the last few years, I've also become a bit more picky about certain kinds of artifacts too.

_Man_

Re: The new enemy of the HD Formats is Noise Reduction!

I'm sorry but if you have an aliasing problem the solution is not DNR but a proper optical low pass filter in the telecine/scanner optimised for the sampling rate of the scan, preferably using oversampling followed by digital decimation to the target resolution. This does not create any strange grain patterns that are not part of the original negative.
If you have aliasing/moire/pattern issues DNR is not the proper answer.
(Do you have any evidence that there was DNR used on the Pirate films?)
Of course that always makes one wonder if the same director objects to the look of the answer print too concerning grain (non DI case) and why something is ok on print(s) but not on HD. Double standards?
No doubt some of the bad stuff that is going on is approved by the film makers themselves. Which kind of leaves no other option but to vote with one's wallet. They are obviously entitled to make their work look any way they want.

Re: The new enemy of the HD Formats is Noise Reduction!

Several layers here. First different people have vastly different ideas of what is good and bad and different sensibilities in these matters. Second it's a business like any other. Meaning there are pressures of all kinds that can affect profit and survival. Jobs need to be finished in time. Vendors want to sell fancy equipment that needs to be depreciated and used, with all the modern bells and whistles, whether they are any good or not, mature or just the latest fad. End customers have their own ideas, how naive or inappropriate they may be (I want no noise ever/I watch only color films/I want no black bars/I hate mono and accept only multi channel sound/I want lights on when I watch stuff and it must look good/I want crisp images even on small screens and when sitting far away...). And they can impose their views with their purchasing behaviour.

Re: The new enemy of the HD Formats is Noise Reduction!

Gary don't get me wrong. I'm not saying that every single shot will need some kind of patch work. It really doesn't happen all that often. But when it doesn't these are some tools that can help the situation. And I never said that DNR didn't remove SOME high frequency, but when you have a problem shot that is the compromise. You have to decide if the noise is more objectionable than the fix for it.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

Doug: I, for one, appreciate your posts. They give us great insight into the perils of converting film into the digital realm. I have seen some really shoddy transfers and had often wondered, "What on earth were they thinking doing this?"

Again, thanks.

Re: The new enemy of the HD Formats is Noise Reduction!

Yes that is one solution to a potential telecine problem. And again I'm not an expert telecine operator. I only know what has happened when I supervised the transfer of some films that I was the DP on. I'm not at all saying that DNR isn't being abused. I am saying that to dismiss it as a useful tool is wrong.

No frankly I don't have any idea what was done to the pirates films, just that they don't look very film like to me. It didn't look very film like in the theater to me either, and I saw them as a film print not digital projection. It looks to me to be very processed.

I think one of the best High Def releases this last year was Bullitt. Beautiful photography by William Fraker, and beautifully translated to HD. It looks like film. I know its apples and oranges being a film from 68, but to me Bullitt is what film is supposed to look like.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

I know every one is entitled to their opinion and I feel weird going against the grain here...but I actually like NR as long as it only removes the grain and doesn't affect any other aspect of the picture. If the NR makes the picture softer or has any other similar effect on it then I totally agree with you that NR shouldn't be used.

I like the fact that I can watch some movies without it looking like I am seeing it through an hourglass with the sand falling....if I want grainy sand I'll go to the beach.

What if movies in the near future become naturally grain-free? Will you want the grain added digitally just so you can experience that old-school film grain look?

I'll take the movie as-is, film-grain or not. I'm usually against any alteration made to a movie, be it aspect ratio or EE....but if an alteration actually improves the PQ then I wouldn't be as opposed to it.

Re: The new enemy of the HD Formats is Noise Reduction!

Grain free movies will be the result of more and more films being shot digitally. Personally I would and have added grain to digitally produced images. I've even go so far as to add very fine specs of dirt to give it that vintage look (the project was supposed to be a film that was made in the 70s)

Personally I'm not a fan of removing grain just to produce a smooth image. If the film has grain I think we should see it.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

film used to make movies has a natural grain in the image, if you don't like that then you don't like films

-Gary

Re: The new enemy of the HD Formats is Noise Reduction!

It has its uses for sure. The problem is the abuse which is so tempting these days. Now that you can play with the color and contrast so easily the idea that you can do the same and should with noise/grain is poisoning minds. But unfortunately the former is trivial compared to the latter. And tools for the latter are far from mature, completely transparent and fast (when you want good quality).
I suspect the film look and digital camera look is converging due to finer grain stock and the DI process (dust busting). More and more films look like they were shot with clean data cameras. They have little to no grain but are not full of DNR problems. They are superclean like video but have otherwise the film look.

Re: The new enemy of the HD Formats is Noise Reduction!

I guess they are now in some sense following the lead of still photography, which could indeed be a bad thing unfortunately. Seems like almost everyone in the world of still photography are clamoring for cleaner and cleaner images w/ very few folks wanting anything that remotely looks like film grain. Everyone wants noise-free ISO6400 images and pixel peep any minor differences to death between cameras (and brands) even at the consumer/entry level. And so many regularly employ DNR software as a routine nowadays -- though that's admittedly better than most any in-camera DNR. Not saying the noise in digital cameras are quite the same as film grain (and I don't mean aliasing/moire type noise), but most folks look at them the same way anyhow. I must admit though that I do lust after the new Nikon D3 a fair bit myself, but I'm not in a hurry to do what many do to get cleaner images, ie. change brands *and* spend a good deal more for a lower feature, but cleaner, camera body...

_Man_

Re: The new enemy of the HD Formats is Noise Reduction!

I agree there is way too much temptation to "play" with the images, both with color and noise reduction. Frankly I'm getting pretty tired of films that have a "color wash" over whole scenes or in some cases the whole movie. There are parts of say Lord of the Rings where they seem to have almost pulled all the color out but a steel blue. I realise they are trying to create a mood, but it was a bit over the top in my opinion. Personally I like to see mixed light. Blue is fine if its mixed with white light somewhere in the frame. But when the whole image is digitally color graded that way...yuck. It seemed for a while there after the Matrix that everyone had to tint their film green. What the hell was that all about? Fortunately that now seems to be a passing fad.

With pirates, for me anyway, it wasn't just that it looked super clean, but rather the color saturation seemed unnatural to me. Not quite over saturated, but some how not natural. Maybe it is that the newer film stocks are being formulated to look more like a digital image I don't know. I do know that the look doesn't thrill me. Compared to a film like Black Book that is also very clean and super sharp, but the colors look very natural in my opinion.

Doug

Re: The new enemy of the HD Formats is Noise Reduction!

Still frames are different from moving pictures. Removing noise from still frames is strictly a local affair. You don't do temporal filtering because there is no temporal dimension. Many kind of artifacts you will not have on still frames because of that. You can still create a waxy or oversmoothed look, but there will be no smearing or flickering or strange noise patterns in time as on moving pictures. With films the whole issue is far more complicated and potentially destructive to the fabric and foundation of the images.

Re: The new enemy of the HD Formats is Noise Reduction!

I didn't mean the techniques will be quite the same, but more that their mentality (and thus, the results) might be following what's going on in the still photography world. Still, plenty of digital cameras produce what look like digital paintings rather than photos due to in-camera DNR -- in some cases, even base ISO stills can have that nasty look on occasion -- and many folks seem to love it though most enthusiasts do want better than that...

_Man_

Re: The new enemy of the HD Formats is Noise Reduction!

You were too involved in the movie to see the glaringly obvious video shortcomings of HDM because there is more to a movie than pixel perfection. However on the audio side, if the box doesn't say "lossless" then it must suck no matter how good it is. Listening to the soundtrack that is on the disc as part of the overall movie experience isn't as important as bitching about how the box says it isn't lossless and conjecturing on how much better it would have been had they used lossless audio (but not TrueHD because DN screws it all up). Remember the bad old days when 1.5 Mb DTS used to be considered great? What a bunch of fools those people were.

Re: The new enemy of the HD Formats is Noise Reduction!

Chuck,

There have always been folks who said we should be happy with what we get. And there have always been folks who said we should ask the studios to deliver the very best that thay can and not settle for "good enough".

The same point of view you express here was behind the criticism aimed at many on this forum in the late 1990's when enthusiasts were demanding that the studios adopt 16x9 anamorphic encoding even though most viewers only had 4x3 televisions and others in the "be happy" crowd were calling us extreemists, elitists, and snobs were and suggesting that since 4x3 letterboxed DVD looked great, we should just be happy like they were.

We were told that by not wanting to settle for 4x3 lbxed DVDs we cared more about technology than we did about movies.



Seems silly today. About as silly as suggesting that lossless audio isn't worth pushing for will seem about 5 years from now.

Those of you who are happy to enjoy what you get and don't feel the need for anything better, by all means, be happy! But rather than criticize those on this forum who do push for greater transparency, just sit back and enjoy the fruits of their labor, because the discs that you're happy with will look and sound better as time goes by because of their effort.

Re: The new enemy of the HD Formats is Noise Reduction!

Early computer DVD software could take advantage of the 16*9 feature.