How can digital restorations fade?

Re: How can digital restorations fade?

Why not?

Re: How can digital restorations fade?

Thanks again Mr. Harris. Looks like alll this debate based on a poorly written article is moot. nice to know our film heritage is being looked after more responisbly.

Re: How can digital restorations fade?

I think because (partly) there is no digital medium that doesn't have issues with corruptibility.

Re: How can digital restorations fade?

I wish I'd had you guys to back me up a few months ago in the TV shows/TV movies forum, where there was a thread similar to this, except that the prevailing opinion was that digital is forever. They practically sent me out on a rail.

Re: How can digital restorations fade?

Outstanding. Thanks for providing the link!

Re: How can digital restorations fade?

If only...

(digital was forever, that is)

It's a really nice theory, it really is. I love the idea that saving something digitally protects it forever and saves us from all possible future losses and headaches, but it just ain't so. Besides the possibilities for data corruption, there's the simple truth that digital technology is evolving rapidly, and state of the art today will probably be obsolete in a few years. As an example, I have floppy disc backups from my first computer in 1992, but nothing to read them with, and even if I did have a drive for those discs, I don't have the necessary software to open the files. What's to say that won't happen on some level with digitally preserved film files, where even if the storage media itself is in good shape, there's nothing handy to play it on? I imagine that the most high profile and most profitable digital restorations won't have to worry about that, but what about the smaller gems? Will studios put up the money to transfer this material every time there's a new storage medium or an older one becomes obsolete?

I'm not saying a film print or a restored negative is a perfect solution, but the simple fact remains that 35mm film technology allows for a twenty or thirty or forty year old film to be played with much more ease than a data file. But film itself is susceptible to problems of its own.

My own opinion is that the wisest course of action is to create backups across all mediums, so that the digital files are preserved using the best available technology of today, but also so that a new film negative and prints are also saved. I doubt that there will ever be a completely foolproof method where someone can create a backup that can be stored in one way forever and is always in perfect shape. It'd be nice for sure, but I just don't see it happening. For better or worse, someone is always going to have to be keeping an eye on these things...and hopefully there will always be resources dedicated to this along with people who truly care being allowed to oversee it all.

Re: How can digital restorations fade?

The Afterlife Is Expensive for Digital Movies

I'm trying to locate a copy of the report, called "The Digital Dilemma: Strategic Issues in Archiving and Accessing Digital Motion Picture Materials"

Re: How can digital restorations fade?

The hysteria over digital files not being permanent is just ridiculous. Yes, hardware can get old and fail. That is why you ALWAYS have backups of files in a different locations. Anyone who has priceless family photos and movie files stored ONLY on their computer hard drive is a fool. I'm sure the movie studios have multiple copies of the digital files of restored movies stored in separate locations. When one source fails, they simply make another backup on a new source, and yes with proper maintenance, it is PERMANENT!

Re: How can digital restorations fade?

But if the fly in the ointment is that the actual storage medium is corrupting, no matter HOW many copies you make if it at once, they're ALL susceptible to corrupting. Unfortunately, it's only when they have that anyone realizes and it's all too late.

Film, on the other hand, has reached a state of near perfection.

I might also add that currently, a digital restoration of a film is still about three times more expensive as a photochemical one.

Re: How can digital restorations fade?

I'm not so sure. For example, David Fincher considers the digital scan of the negative to be THE master format for Seven. It was performed at 2K on a Spirit Datacine, which at the time was state of the art. After all the grading and correcting, Fincher felt the digital master was a better representation of the film than the negative, which had already started to fade - probably because of the heavy use of bleach bypass.
But there are things that can be done digitally that can't be done easily, or at all, photo-chemically. Which I guess is why Warner went the digital route for The Searchers. Surely if they could do the same for less money photo-chemically, then that is what they would've done? It seems to me that if a negative is printable, then photo chemical means can cost effectively produce an element ready for a video transfer. But if the negative isn't printable, then digital technology offers lots of ways around dealing with later generation elements, Ultra Resolution from separations being one of them.

Re: How can digital restorations fade?

They're all potentially susceptible to corruption, but they don't all magically corrupt simultaneously. If one copy of the film is suffering problems, new digital copies can be made from one of the back up copies. I really agree with Mark P. on this one.

Re: How can digital restorations fade?

Hasn't the new 4K and 6K scanning techniques just resulted in an explosion in data storage and integrity companies? Companies like Deluxe have diversified to offer data storage and backup services, and of course there are dedicated companies like Iron Mountain who specialise in storing backups in secure locations.

Of course carefully storing digital copies of films is a huge issue, but is it really that different to proper storage of the camera negative?

Re: How can digital restorations fade?

Well the problem with digital storage is that you have to make backups because of potential corruption. And now you just doubled the storage size. Its not like you can fit The Searchers on a 10 GB SD card the size of your thumb--these things sit in big servers, rooms full of them, and of course they all need power and maintenance like film, and as this becomes more common there will probably be third-copy off-site digital storage. And you still need to be keeping the film vaults because you aren't just going to throw away the originals. And the vaults contain more than just the master negatives, they contain the raw O-negs from the camera, foreign prints, raw audio tracks and so forth.

One day though, I'm sure the vaults of studios will be digital. But not yet; and not anytime soon. There are hundreds of thousands of reels for thousands and hundreds of thousands of films in some studios collection. The act of scanning them all alone is a collosal undertaking. As well though, sometimes you simply need a hard copy for posteritty.

Plus there is also a bit of prestige in maintaining the original films. One could argue that a 8K scan of the Mona Lisa would yield a replicate indistinguishable to the eye, and will never fade so all this expensive and continuous restoration will be a thing of the past; but theres simply something satisfying and romantic about preserving the celluloid that actually captured the reflected photons from John Wayne's face as he sees his ranch burn in The Searchers, for instance.

Re: How can digital restorations fade?

Fast forward 30 or 40 years. The original production company is defunct. No one's been recopying the data to new media, because no one's been paying the bills. Some of the encryption keys are unknown, because employees have moved on...

Re: How can digital restorations fade?

I've been following this for days, and THAT, to me, is the biggest piece of truth yet thrown into the mix.

Re: How can digital restorations fade?

Encryption keys? It's not even that much. Try reading an unencrypted data file from a legacy computer program from 10-15 years ago. You need the original program, a computer system capable of running the program...and that's the kicker. There are programs written for Win9x that will not run on Windows Vista, and the companies are not around to fix the problem. Give it 30-40 years from now and the problem will be so much worse.

I have some floppy discs containing databases and spreadsheets that I put together in 1993. Neither the database program nor spreadsheet program exist anymore, as the companies were driven out of business a few years later by the proliferation of MS Office. I can copy the files from media to media all I want, but I have no way to access the data any longer.

Re: How can digital restorations fade?

Cost of storage isn't really an issue, computer storage gets cheaper every year.
Yeah, this is already happening, there are companies that specialise in data storage and integrity.
Of course! I'm not suggesting that digital will ever replace the original camera negative. But it offers many advantages for restoring films, and we must remember, companies don't make money out of old films until they get them into a digital video format. So having a digital copy of the film will actually promote more restoration and preservation of other films, because a digital copy can make revenue, whereas only having film prints can't.

Re: How can digital restorations fade?

But that's what the genius programmer guys are for. They can crack that stuff. Let me give you an example comparison to film stock. Back in the 1930's there were special 65MM widescreen processes like "Grandeur", "Realife" and "Magnifilm". Each was used to shoot only one or two movies and then became defunct. Fast forward 50 or 60 years, the film gauge doesn't match anything currently in use. How were these films restored? They custom built a frickin' projector that would work with these films. Of course now it would be no problem just to scan a film no matter what the guage, but they did find a solution to the problem and I'm sure old defunct software can be figured out by someone.
How about we all break for Christmas now???

Re: How can digital restorations fade?

When you have a few dozen million terabytes of data doubled to a hundred million terabytes of data, and then have that same number backed-up in reserves, suddenly its a significant figure though. Plus, you still have to maintain it, pay the electrical bill, transfer everything to new drives every few years(a huge undertaking), run data loss and error checks, make sure theres a guy checking and re-backing everything once in a while, etc. Of course its still a better option than dealing with film provided that its done right (in theory), but I'm just saying its not at all easy, and the costs of doing all of this are higher than simply building an air-condition room and shoving cans in it. Its an ideal that right now is very impractical, and will remain as such probably for most of our lifetimes.

A lot of transfers are also, unfortunately, HD, which is almost useless except for home video in terms of creating a digital negative. But I guess slowly, in time, some sort of system will develop and libraries built, at least for the major titles. By the time anything significant happens it will be ten or twenty years from now and by then storing hundreds of millions of terabytes can be done in a small room instead of an entire facility.

Re: How can digital restorations fade?

My point was simply that some studio executives aren't going to say "well, we COULD backup our data, but you know gee isn't computer storage EXPENSIVE, so let's not bother about it!" The cost of computer storage is the least of their concerns.

If The Searchers has a chance of making money on some format 10 years from now, then you can sure as hell know that Warner will do everything right to store the digital restoration properly (as far as I know they HAVEN'T scanned it back to film, because they wouldn't make much money releasing it on film).

All the logistics of how to store the data are delt with by companies like Iron Mountain who specialise in storing computer data, so that really isn't an issue. Warner or whoever just have to pay the monthly rental fee for xTb of storage.

A bigger issue is what format to store in, but it seems that scans are stored as raw uncompressed data which will be highly portable to future formats.

Encryption isn't an issue, why encrypt the data when it can only legally be released by the copyright holder?

Personally I don't think there is much chance of Time-Warner or News Corporation going broke, so I think the Warner Bros / M.G.M. / R.K.O and Fox archives are safe. The Paramount and Universal archives are safe, because they aren't doing much with them.

Warner, Fox and Columbia all seem to make their DVD transfers from HD masters. Though some of them are 1080i rather than 1080p. It doesn't take much computer storage to store files as 1080p, it's the 2K, 4K, 6K scans that count. But they are only used for major restorations anyway.

Re: How can digital restorations fade?

I've been involved with some of the periphery of these discussions, although at a much smaller level, and I know some people who really are involved at much higher levels.

Let's make some assumptions. First, an 8k scan should be pretty good. Second, 24-bit color depth is pretty much crap; 8-bits per channel yields 256 shades, and frankly, that ain't enough. Some say 14-bit isn't enough; so let's say 16-bit/channel.

A 2 hour feature film, picture alone, at 24-fps, yields 52 terabytes for the finished feature.

Now. What has the DVD revolution taught us?

It taught Hollywood a valuable lesson: all those extra frames come in handy later on; we might be able to resell them! Extra features! Screen tests! Costume tests!

Now, we're looking at a whole lot more. And as people are already complaining when the feature is anamorphic, and the suppliments are 4:3, likewise people will complain when the feature is fantastic, and the suppliments are only HD. Beside, you never know when Robert Wise is going to revisit a turkey and make something good out of it again, so we'd best keep it all.

To be conservative, the way a lot of people shoot films, between principle photography and second unit, we're now probably looking at an additional 500TB.

By today's standards, a Hitachi 1TB hard disk is available on-line for about $350. The drives to store that one feature and its suppliments cost $192,500.

Parked drives don't last forever. Assuming about $0.08/kwH, and a two hours a week, we're looking at an estimated $660 per year for power to spin the drives on occassion.

And in 3 years, we start a tech working full-time on copying them over to new disks, at $25,000/year salary. Benefits, taxes, et cetera, and this techie is actually costing somewhere closer to $60,000/year.

So to preserve the archives for one 'cycle,' we're looking at spending about $447,640 to preserve one film and its extras as digital files.

Now, some of you might say my "numbers are unrealistic." Maybe so. But the big problem is, how long is that drive REALLY going to last? 3 years? 4 years? 10 years? What percentage are going to fail prematurely? One major weakpoint in the above is that there is no room for a drive failure, and it also assumes that it's going to take a while to copy the data from disk to disk - I'm allowing about 2 years for the process, and I'm not charging the power bills for the copy process, even though I've heard it takes about 6 hours just to format a 1TB disk and check it for bad sectors.

The big difficulties are actually fairly simple:

1. How long does a given media format really last?
2. Do people realize just how much data is being talked about?! The magnitude for this sort of preservation work is insane.

If we project out and say that Hollywood produces, say, 260 films a year. If they want to save it all, we're looking at 130 PETABYTES per year, using the above numbers, at a 4-year cycle cost of about US$130,000,000.

Is Hitachi even making that many hard drives a year?

When you really start to think about it, the numbers soon get mind-boggling - unless you're used to working with Defense Department type numbers.

Leo

Re: How can digital restorations fade?

So 4k digital cinema isn't good enough? That's 300 gig, per film.

Re: How can digital restorations fade?

Leo, thats exactly what I was getting at. I'd say your figures are exagerated, but it still puts into perspective the enormous effort, cost and logistics of such an operation.

A more practical solution is that the raw footage, outtakes and raw materials aren't being scanned, and I would say that its foolish to do so at this point. In terms of digital preservation, the first priority is the actual negative of the films, the movies themselves, and an 8K scan is a waste of money for 35mm; many scans are 2K, but an ideal figure is a 4K scan. Having done testing myself, even though 35mm film will resolve closer to 6K, this is under idealised and rare conditions, and often becomes undetectable to the human eye from the 4K version. So, going by your numbers, a 4K digital negative of your typical 2-hour film is some 25 TB (I wish I knew anything about audio storage)

It also highlights what I raised before: the continuous maintenance needed. Hard-drives have a very short lifespan, and would need to be backed up to a new drive every 2-3 years (ideally HDD's should last 3-5 years--but we want to get them backed up before they start to go corrupt). Aside from the power bill to keept those drives spinning, the maintenance bill to have a guy keep an eye on the data, and the housing bill for the facilities where all these drives are kept, transfering 25 TB per film every two years is a big job. If we are talking about permanent digital storage of a studio vault, thats some hundred thousand titles--times 25TB each, plus audio. And this is simply the original negatives, this doesn't account alternate cuts, deleted material, and most importantly foreign versions (which is some 50% of any films market and just as important). Clearly, this will never be feasible in our lifetimes. But lets be selective: lets say we preserve the top 500 classics in a studio collection from before 1980, the ones that need digital preservation the most; then we select the next 500 most popular and/or important titles in the studio vault. 1000 films is an okay start but its only a fraction of the titles when you account for the amount thats available. But this is film preservation, and unfortunately we have to be selective about what we preserve because we have limited time, money and man hours. So with 1000 titles we have 25 TB each for the negative--lets round it out to 30 TB including the sound mixes (with all the different mixes--mono, stereo, six-track, DD5.1, remixes, etc--this is overlooking a deceptively complicated issue but lets try to just overlook that for now).

So if we have a select 1000 films thats 30,000 TB of storage space just for movies. Of course there is an on-site backup, and then another off-site backup, at the least. So now theres almost 100,000 TB of data that has to be maintained, and then totally backed up on to new drives every two years or so. By current standards, the cost of simply purchasing the physical drives alone (at roughly $350/TB) is about $10,500,000. Every two years. Housing this data requires its own facility, with its own employees and all the overhead that goes with an entire presevation facility. So you can probably double that number. At upwards of $20 million per year to digitally store the top 1000 titles in a studio vault that pretty significant. And meanwhile the film vault is continued to be used and maintained--this is all just a "bonus".

The reality is that studio executives don't care. They are only going to be with the company for a few years--all of this stuff is long-term, stuff that will only become useful decades later to the next generations of viewers and technicians. The truth is that if you are in charge of a studio you don't give a damn if some guy in fifty years will have an inferior version of Easy Rider to watch, because you'll be long dead. Why spend upwards of $20 million a year for that jerk fifty years from now when can simply spend the $2 or 3 million for the annual costs of remastering that years releases for home video?

Thats why, as I said, it won't be until ten to twenty years before this kind of undertaking is even considered. By then hopefully storage costs can be cut in half by the greater amount of data a drive can contain and the smaller sizes they can come in--but this is only a small portion of a large number of overhead issues. But what I do see happening is that by then studios would have been saving the scans that they already have done, so they might have a hundred films or so already saved in on-site storage by the time 2025 comes around and some guy at a studio actually proposes spearheading an iniative for a digital preservation mission of the studio's vaults.

Re: How can digital restorations fade?

25 Tb for a film sounds like a bargain to me. I concede that ten years ago that much storage would cost about $4.7 million, so you'd never bother even trying to do it (film scanners back then sucked anyway, so it wasn't even feasible).

But how much is 25 Tb of storage now? My estimate is $6800, based on 750 GB drives now costing $200. That sounds like a bargain to store a close to perfect copy of a film.

More to the point, by my calculations, the cost of storage has declined by about 690% in the last 10 years. If that happens again, by 2017, 25TB of storage will cost $10.

Re: How can digital restorations fade?

A story I heard at a trade show was that Space Jam was finished digitally, with the "original" files stored on hard drives. The files got corrupted, so the best version of the film was lost.

Re: How can digital restorations fade?

Maybe some of the studio execs don't care, perhaps at the very top levels, but there are quite a lot of people in the chains of command that DO care.

Their big problem probably isn't the money so much as How? If they could say, "we spend this $150,000 now, and it will last perfectly for 10 years," it probably wouldn't be a problem.

Disney stored a bunch of things on Exabyte tapes years ago. Some of those people were surprised to find that, "gee, in 8 years, those tapes are completely unreadable." Not just because of changing technology; they still had working drives, but because Exabyte was always notorious for not working. My father worked with it in research data aquisition. Their take was to write to the tape once during the test. Take it back to the lab, and read it off onto the servers there, and then junk the tape. Two passes through the machine; that was it. Any more than that had unrecoverable errors.

DAT tapes, DV tapes; they all have longevity issues. I've got a 4 year old DAT at work that has massive dropouts on it; this being the second time it's been moved through a transport.

Optical storage? What kind? We all know that -R and -RW (or the + form) are iffy at best. Stamped discs don't always hold up very well, either. Plus, they're slow. I've got some DVD-Rs that are about 8 years old; some of them are fine. Others, only about three years old, are having problems.

Holographic storage? Right... We have absolutely no track record on practical life-span of the recorded media.

It's kind of scary, but probably the best way to store a film - apart from on polyestar color separations, is on paper.

Yes, paper.

Imagine a nice big atlas-folio sized book, filled with acid-free "India" paper (the thin opaque stuff they print dictionaries on,) filled with 2-D barcodes. I can't remember the numbers I was throwing around when I was estimating that sort of project a few years ago, but I think a "reasonable" quality feature film would occupy about, what, 20 linear feet of shelf space, using ~ 28" x 18" pages? I suppose I could dig around to try and find those numbers again...

Pretty soon, the data storage people are going to just have to start hoping for a miracle; the quantities of data being generated that people want to keep around for decades is going up faster than the real-estate prices ever did.

Leo

Re: How can digital restorations fade?

I can believe that, the reason why some early silents exist is because dupe records of them were printed on a special paper film instead of good ol extra flammable nitrate.

Re: How can digital restorations fade?

Be sure to control for ink bleed. I was recently reading some reviews of A new kind of science, and some of them noted that production costs were much higher because of the high resolution illustrations of Cellular automata.

Re: How can digital restorations fade?

Robert Wise is dead.

Re: How can digital restorations fade?

So is Paul.