How can digital restorations fade? (1 Viewer)

[cafink]

They're all potentially susceptible to corruption, but they don't all magically corrupt simultaneously. If one copy of the film is suffering problems, new digital copies can be made from one of the back up copies. I really agree with Mark P. on this one.

 

[Simon Howson]

Hasn't the new 4K and 6K scanning techniques just resulted in an explosion in data storage and integrity companies? Companies like Deluxe have diversified to offer data storage and backup services, and of course there are dedicated companies like Iron Mountain who specialise in storing backups in secure locations.

Of course carefully storing digital copies of films is a huge issue, but is it really that different to proper storage of the camera negative?

 

[mike kaminski]

Well the problem with digital storage is that you have to make backups because of potential corruption. And now you just doubled the storage size. Its not like you can fit The Searchers on a 10 GB SD card the size of your thumb--these things sit in big servers, rooms full of them, and of course they all need power and maintenance like film, and as this becomes more common there will probably be third-copy off-site digital storage. And you still need to be keeping the film vaults because you aren't just going to throw away the originals. And the vaults contain more than just the master negatives, they contain the raw O-negs from the camera, foreign prints, raw audio tracks and so forth.

One day though, I'm sure the vaults of studios will be digital. But not yet; and not anytime soon. There are hundreds of thousands of reels for thousands and hundreds of thousands of films in some studios collection. The act of scanning them all alone is a collosal undertaking. As well though, sometimes you simply need a hard copy for posteritty.

Plus there is also a bit of prestige in maintaining the original films. One could argue that a 8K scan of the Mona Lisa would yield a replicate indistinguishable to the eye, and will never fade so all this expensive and continuous restoration will be a thing of the past; but theres simply something satisfying and romantic about preserving the celluloid that actually captured the reflected photons from John Wayne's face as he sees his ranch burn in The Searchers, for instance.

 

[JeremyErwin]

Fast forward 30 or 40 years. The original production company is defunct. No one's been recopying the data to new media, because no one's been paying the bills. Some of the encryption keys are unknown, because employees have moved on...

 

[Mark B]

I've been following this for days, and THAT, to me, is the biggest piece of truth yet thrown into the mix.

 

[nolesrule]

Encryption keys? It's not even that much. Try reading an unencrypted data file from a legacy computer program from 10-15 years ago. You need the original program, a computer system capable of running the program...and that's the kicker. There are programs written for Win9x that will not run on Windows Vista, and the companies are not around to fix the problem. Give it 30-40 years from now and the problem will be so much worse.

I have some floppy discs containing databases and spreadsheets that I put together in 1993. Neither the database program nor spreadsheet program exist anymore, as the companies were driven out of business a few years later by the proliferation of MS Office. I can copy the files from media to media all I want, but I have no way to access the data any longer.

 

[Simon Howson]

Of course! I'm not suggesting that digital will ever replace the original camera negative. But it offers many advantages for restoring films, and we must remember, companies don't make money out of old films until they get them into a digital video format. So having a digital copy of the film will actually promote more restoration and preservation of other films, because a digital copy can make revenue, whereas only having film prints can't.

 

[Mark-P]

But that's what the genius programmer guys are for. They can crack that stuff. Let me give you an example comparison to film stock. Back in the 1930's there were special 65MM widescreen processes like "Grandeur", "Realife" and "Magnifilm". Each was used to shoot only one or two movies and then became defunct. Fast forward 50 or 60 years, the film gauge doesn't match anything currently in use. How were these films restored? They custom built a frickin' projector that would work with these films. Of course now it would be no problem just to scan a film no matter what the guage, but they did find a solution to the problem and I'm sure old defunct software can be figured out by someone.
How about we all break for Christmas now???

 

[mike kaminski]

When you have a few dozen million terabytes of data doubled to a hundred million terabytes of data, and then have that same number backed-up in reserves, suddenly its a significant figure though. Plus, you still have to maintain it, pay the electrical bill, transfer everything to new drives every few years(a huge undertaking), run data loss and error checks, make sure theres a guy checking and re-backing everything once in a while, etc. Of course its still a better option than dealing with film provided that its done right (in theory), but I'm just saying its not at all easy, and the costs of doing all of this are higher than simply building an air-condition room and shoving cans in it. Its an ideal that right now is very impractical, and will remain as such probably for most of our lifetimes.

A lot of transfers are also, unfortunately, HD, which is almost useless except for home video in terms of creating a digital negative. But I guess slowly, in time, some sort of system will develop and libraries built, at least for the major titles. By the time anything significant happens it will be ten or twenty years from now and by then storing hundreds of millions of terabytes can be done in a small room instead of an entire facility.

 

[Simon Howson]

My point was simply that some studio executives aren't going to say "well, we COULD backup our data, but you know gee isn't computer storage EXPENSIVE, so let's not bother about it!" The cost of computer storage is the least of their concerns.

If The Searchers has a chance of making money on some format 10 years from now, then you can sure as hell know that Warner will do everything right to store the digital restoration properly (as far as I know they HAVEN'T scanned it back to film, because they wouldn't make much money releasing it on film).

All the logistics of how to store the data are delt with by companies like Iron Mountain who specialise in storing computer data, so that really isn't an issue. Warner or whoever just have to pay the monthly rental fee for xTb of storage.

A bigger issue is what format to store in, but it seems that scans are stored as raw uncompressed data which will be highly portable to future formats.

Encryption isn't an issue, why encrypt the data when it can only legally be released by the copyright holder?

Personally I don't think there is much chance of Time-Warner or News Corporation going broke, so I think the Warner Bros / M.G.M. / R.K.O and Fox archives are safe. The Paramount and Universal archives are safe, because they aren't doing much with them.

Warner, Fox and Columbia all seem to make their DVD transfers from HD masters. Though some of them are 1080i rather than 1080p. It doesn't take much computer storage to store files as 1080p, it's the 2K, 4K, 6K scans that count. But they are only used for major restorations anyway.

 

[Leo Kerr]

I've been involved with some of the periphery of these discussions, although at a much smaller level, and I know some people who really are involved at much higher levels.

Let's make some assumptions. First, an 8k scan should be pretty good. Second, 24-bit color depth is pretty much crap; 8-bits per channel yields 256 shades, and frankly, that ain't enough. Some say 14-bit isn't enough; so let's say 16-bit/channel.

A 2 hour feature film, picture alone, at 24-fps, yields 52 terabytes for the finished feature.

Now. What has the DVD revolution taught us?

It taught Hollywood a valuable lesson: all those extra frames come in handy later on; we might be able to resell them! Extra features! Screen tests! Costume tests!

Now, we're looking at a whole lot more. And as people are already complaining when the feature is anamorphic, and the suppliments are 4:3, likewise people will complain when the feature is fantastic, and the suppliments are only HD. Beside, you never know when Robert Wise is going to revisit a turkey and make something good out of it again, so we'd best keep it all.

To be conservative, the way a lot of people shoot films, between principle photography and second unit, we're now probably looking at an additional 500TB.

By today's standards, a Hitachi 1TB hard disk is available on-line for about $350. The drives to store that one feature and its suppliments cost $192,500.

Parked drives don't last forever. Assuming about $0.08/kwH, and a two hours a week, we're looking at an estimated $660 per year for power to spin the drives on occassion.

And in 3 years, we start a tech working full-time on copying them over to new disks, at $25,000/year salary. Benefits, taxes, et cetera, and this techie is actually costing somewhere closer to $60,000/year.

So to preserve the archives for one 'cycle,' we're looking at spending about $447,640 to preserve one film and its extras as digital files.

Now, some of you might say my "numbers are unrealistic." Maybe so. But the big problem is, how long is that drive REALLY going to last? 3 years? 4 years? 10 years? What percentage are going to fail prematurely? One major weakpoint in the above is that there is no room for a drive failure, and it also assumes that it's going to take a while to copy the data from disk to disk - I'm allowing about 2 years for the process, and I'm not charging the power bills for the copy process, even though I've heard it takes about 6 hours just to format a 1TB disk and check it for bad sectors.

The big difficulties are actually fairly simple:

1. How long does a given media format really last?
2. Do people realize just how much data is being talked about?! The magnitude for this sort of preservation work is insane.

If we project out and say that Hollywood produces, say, 260 films a year. If they want to save it all, we're looking at 130 PETABYTES per year, using the above numbers, at a 4-year cycle cost of about US$130,000,000.

Is Hitachi even making that many hard drives a year?

When you really start to think about it, the numbers soon get mind-boggling - unless you're used to working with Defense Department type numbers.

Leo

 

[JeremyErwin]

So 4k digital cinema isn't good enough? That's 300 gig, per film.

 

[mike kaminski]

Leo, thats exactly what I was getting at. I'd say your figures are exagerated, but it still puts into perspective the enormous effort, cost and logistics of such an operation.

A more practical solution is that the raw footage, outtakes and raw materials aren't being scanned, and I would say that its foolish to do so at this point. In terms of digital preservation, the first priority is the actual negative of the films, the movies themselves, and an 8K scan is a waste of money for 35mm; many scans are 2K, but an ideal figure is a 4K scan. Having done testing myself, even though 35mm film will resolve closer to 6K, this is under idealised and rare conditions, and often becomes undetectable to the human eye from the 4K version. So, going by your numbers, a 4K digital negative of your typical 2-hour film is some 25 TB (I wish I knew anything about audio storage)

It also highlights what I raised before: the continuous maintenance needed. Hard-drives have a very short lifespan, and would need to be backed up to a new drive every 2-3 years (ideally HDD's should last 3-5 years--but we want to get them backed up before they start to go corrupt). Aside from the power bill to keept those drives spinning, the maintenance bill to have a guy keep an eye on the data, and the housing bill for the facilities where all these drives are kept, transfering 25 TB per film every two years is a big job. If we are talking about permanent digital storage of a studio vault, thats some hundred thousand titles--times 25TB each, plus audio. And this is simply the original negatives, this doesn't account alternate cuts, deleted material, and most importantly foreign versions (which is some 50% of any films market and just as important). Clearly, this will never be feasible in our lifetimes. But lets be selective: lets say we preserve the top 500 classics in a studio collection from before 1980, the ones that need digital preservation the most; then we select the next 500 most popular and/or important titles in the studio vault. 1000 films is an okay start but its only a fraction of the titles when you account for the amount thats available. But this is film preservation, and unfortunately we have to be selective about what we preserve because we have limited time, money and man hours. So with 1000 titles we have 25 TB each for the negative--lets round it out to 30 TB including the sound mixes (with all the different mixes--mono, stereo, six-track, DD5.1, remixes, etc--this is overlooking a deceptively complicated issue but lets try to just overlook that for now).

So if we have a select 1000 films thats 30,000 TB of storage space just for movies. Of course there is an on-site backup, and then another off-site backup, at the least. So now theres almost 100,000 TB of data that has to be maintained, and then totally backed up on to new drives every two years or so. By current standards, the cost of simply purchasing the physical drives alone (at roughly $350/TB) is about $10,500,000. Every two years. Housing this data requires its own facility, with its own employees and all the overhead that goes with an entire presevation facility. So you can probably double that number. At upwards of $20 million per year to digitally store the top 1000 titles in a studio vault that pretty significant. And meanwhile the film vault is continued to be used and maintained--this is all just a "bonus".

The reality is that studio executives don't care. They are only going to be with the company for a few years--all of this stuff is long-term, stuff that will only become useful decades later to the next generations of viewers and technicians. The truth is that if you are in charge of a studio you don't give a damn if some guy in fifty years will have an inferior version of Easy Rider to watch, because you'll be long dead. Why spend upwards of $20 million a year for that jerk fifty years from now when can simply spend the $2 or 3 million for the annual costs of remastering that years releases for home video?

Thats why, as I said, it won't be until ten to twenty years before this kind of undertaking is even considered. By then hopefully storage costs can be cut in half by the greater amount of data a drive can contain and the smaller sizes they can come in--but this is only a small portion of a large number of overhead issues. But what I do see happening is that by then studios would have been saving the scans that they already have done, so they might have a hundred films or so already saved in on-site storage by the time 2025 comes around and some guy at a studio actually proposes spearheading an iniative for a digital preservation mission of the studio's vaults.

 

[Simon Howson]

25 Tb for a film sounds like a bargain to me. I concede that ten years ago that much storage would cost about $4.7 million, so you'd never bother even trying to do it (film scanners back then sucked anyway, so it wasn't even feasible).

But how much is 25 Tb of storage now? My estimate is $6800, based on 750 GB drives now costing $200. That sounds like a bargain to store a close to perfect copy of a film.

More to the point, by my calculations, the cost of storage has declined by about 690% in the last 10 years. If that happens again, by 2017, 25TB of storage will cost $10.

 

[Don Solosan]

A story I heard at a trade show was that Space Jam was finished digitally, with the "original" files stored on hard drives. The files got corrupted, so the best version of the film was lost.

 

[Leo Kerr]

Maybe some of the studio execs don't care, perhaps at the very top levels, but there are quite a lot of people in the chains of command that DO care.

Their big problem probably isn't the money so much as How? If they could say, "we spend this $150,000 now, and it will last perfectly for 10 years," it probably wouldn't be a problem.

Disney stored a bunch of things on Exabyte tapes years ago. Some of those people were surprised to find that, "gee, in 8 years, those tapes are completely unreadable." Not just because of changing technology; they still had working drives, but because Exabyte was always notorious for not working. My father worked with it in research data aquisition. Their take was to write to the tape once during the test. Take it back to the lab, and read it off onto the servers there, and then junk the tape. Two passes through the machine; that was it. Any more than that had unrecoverable errors.

DAT tapes, DV tapes; they all have longevity issues. I've got a 4 year old DAT at work that has massive dropouts on it; this being the second time it's been moved through a transport.

Optical storage? What kind? We all know that -R and -RW (or the + form) are iffy at best. Stamped discs don't always hold up very well, either. Plus, they're slow. I've got some DVD-Rs that are about 8 years old; some of them are fine. Others, only about three years old, are having problems.

Holographic storage? Right... We have absolutely no track record on practical life-span of the recorded media.

It's kind of scary, but probably the best way to store a film - apart from on polyestar color separations, is on paper.

Yes, paper.

Imagine a nice big atlas-folio sized book, filled with acid-free "India" paper (the thin opaque stuff they print dictionaries on,) filled with 2-D barcodes. I can't remember the numbers I was throwing around when I was estimating that sort of project a few years ago, but I think a "reasonable" quality feature film would occupy about, what, 20 linear feet of shelf space, using ~ 28" x 18" pages? I suppose I could dig around to try and find those numbers again...

Pretty soon, the data storage people are going to just have to start hoping for a miracle; the quantities of data being generated that people want to keep around for decades is going up faster than the real-estate prices ever did.

Leo

 

[Lord Dalek]

I can believe that, the reason why some early silents exist is because dupe records of them were printed on a special paper film instead of good ol extra flammable nitrate.

 

[JeremyErwin]

Be sure to control for ink bleed. I was recently reading some reviews of A new kind of science, and some of them noted that production costs were much higher because of the high resolution illustrations of Cellular automata.

 

[Zack Gibbs]

Robert Wise is dead.

 

[TravisR]

So is Paul.