Long term/archival storage solutions for digital original media?

N

Nick Ravich

New member
Hello all-

For some reason having a really hard time getting folks to talk to me about this issue so wanted to throw it out to the DVXuser community (not quite sure if this is right spot for it on the forum.)

I work for a small, non-profit arts organization that produces a pretty high profile nat'l documentary series for PBS. We’ve got a growing amount of digital original video material (multi-GB, broadcast-intended digital video files; mostly XDcam EX and P2 original) and we need to get serious about more long term/archival preservation – a system where I can reliably expect to access the media 5/10/20 years down the line. Currently all this media lives on multiple, but non-networked, non-RAIDED external drives; given the life expectancy for these kind of drives, I realize they’re really only a short term solution. Up to the last couple of years, almost all of our original footage was shot to tape; we’ve been creating protection masters, and storing masters and protections in separate climate controlled facilities. Obviously digital material requires a different solution.

One important thing to know about us - we have serious aspirations to preserve all of our originally-produced footage beyond the life of the organization, to eventually make publicly available for researchers, students, etc. So this is not a client-mandated need but instead something generated internally, motivated by our contemporary art and media centered mission. Being smart now about how we ensure the longevity/future usability of this material is crucial for us.

I know the terms "archival" and "long term" probably bring up more questions than answers but I'm wondering how folks in similar positions - smaller production companies producing a consistent (if not broadcaster level volume) of digital original material, who own their media and have a vested interest in preserving it - have dealt with this. Transferring to LTO5 tape? Some kind of cloud/network-based solution? In house? Out-sourced?

Honestly, very surprised there isn't more discussion out there about this. Really hoping I can spark something here.

Sincerely, Nick
 
IMHO,
1) The only viable archival medium is digital mag tape. 99.9998% of this planet's most valuable (and mundane) data has been archived this way for many many decades. It is the archival medium of choice by people with unlimited resources.
a) Hard drives are NOT archival. They are guaranteed to fail at some (unpredictable) point. But they are OK for short-term convenience/backup.
b) Flash memory is NOT archival (where "archival" means > 20 years). Oxides are not perfect (especially in bargain chips) and charges leak eventually.
c) Field-writable optical disks may be "archival" if you limit your definition to a few decades (at best). I don't trust them > 3 years. I have had them fail in 6 months.
2) Computer-format tapes formats (LTO, etc.) are more long-lived and broadly supported than any particular video tape format du jour.
a) To be sure, there will likely be people who keep various video tape/cassette machines running long after their normal life-cycle. But they will charge PREMIUM prices to copy an old tape to a current format.
b) There will always be more services to chose from when recovering an old computer-format tape than any video-format tape. Just because there are 1000x (100000x?) more computers than video cassette or camcorder machines out there.
c) Computer file formats are far more stable and long-lived than any digital video format. (Although not necessarily the required codecs.)

That said, I can't afford LTO. I archive my stuff on external hard drives, and copy it over to the next latest and greatest drive every several (~5) years. With hard drive capacity continuing to increase (at virtually no higher price), this is a viable method for the mid-term.

Note that it is prudent to keep TWO copies on two different brand drives, and prefereably in geographically diverse places.
Hard drives have been known to suffer from batch failures. A whole raid array (or even multiple independent copies) assembled from drives of the same manufacturer/model/capacity/batch is an avoidable risk. Send an annual archive hard drive to Aunt Bertha in Albuquerque each Christmas along with the fruitcake. She can keep it in the back of her sock drawer for you. (The hard drive, not the fruitcake.)
 
Last edited:
Richard-
Thanks so much for breaking it down like that for me.
So overall takeaway is - go for LTO or like computer-format tape if I've got the $.

Curious what you said here - "c) Computer file formats are far more stable and long-lived than any digital video format. (Although not necessarily the required codecs.)" Presumably there's not a computer file format I could transcode my XDcam EX/DVCPro HD/H264 files and still maintain as video? IE I'm stuck with them as unstable video file formats. But presumably there is some "loss-less" codec I could transcode the originals to that would extend the usability/planned obsolence of the original files. Someone just recently pitched me JPEG2000 as an option.

Interesting to hear about your still HDD-based method. Yeah, been too reliant on one brand of drive lately - Gtech. You've motivated me to diversify. Geographic diversity, check.

BTW - also posted on another part of the forum and got this reply:
http://www.dvxuser.com/V6/showthrea...-digital-original-media&p=2436239#post2436239
I'd pose the same question to you - if I'm looking for an out of the box LTO solution, any ideas on where I could go? Or is out of the box not really feasible, that it's necessarily going to involve out of house IT?

Thanks again, Nick.
 
It would be great if there was a better way to preserve media, without constantly having to migrate to new formats. I suppose todays technology makes this easier then it was many years ago in the days of analog video. If you think about it, we have made some pretty tremendous strides in video and television technology. It wasn't long ago, maybe 40 years ago, every home in the country had a black and white television, and today most people have a led tv or two in there home. It makes me wonder what will come next in the world of media and storage technologies.
 
Last edited:
Nick Ravich said:
Richard-
Thanks so much for breaking it down like that for me.
So overall takeaway is - go for LTO or like computer-format tape if I've got the $.

Curious what you said here - "c) Computer file formats are far more stable and long-lived than any digital video format. (Although not necessarily the required codecs.)" Presumably there's not a computer file format I could transcode my XDcam EX/DVCPro HD/H264 files and still maintain as video? IE I'm stuck with them as unstable video file formats. But presumably there is some "loss-less" codec I could transcode the originals to that would extend the usability/planned obsolence of the original files. Someone just recently pitched me JPEG2000 as an option.
Click to expand...

Codecs come and go. But since they are software, it seems more reasonable to assume there will be purveyors of antique/obsolete codecs so we can recover our old video files. Certainly it is much easier to port a codec to a new operating system than it is to maintain video heads (that wear down with use) and rubber parts (belts, pucks, pinch rollers, etc.) that age and harden and crack simply with age. It takes dedicated fans of old video tape equipment to keep them running. I presume the same thing will happen with video codecs at the end of their active life-cycle.

Interesting to hear about your still HDD-based method. Yeah, been too reliant on one brand of drive lately - Gtech. You've motivated me to diversify. Geographic diversity, check.
Click to expand...

Gtec is not a brand of hard drive. It is a brand of the box around the hard drive. When you buy a fancy "name brand" hard drive product, you never really know what's inside unless you take it apart and look for yourself. I have not found those "name brand" hard drive boxes to offer anything that is worth their premium price (vs. plan old raw hard drives plugged into a dock).

BTW - also posted on another part of the forum and got this reply:
http://www.dvxuser.com/V6/showthrea...-digital-original-media&p=2436239#post2436239
I'd pose the same question to you - if I'm looking for an out of the box LTO solution, any ideas on where I could go? Or is out of the box not really feasible, that it's necessarily going to involve out of house IT?
Click to expand...

I haven't done a lot of research on LTO (or any of the similar computer tape backup solutions.) They are all too rich for my blood, so I haven't put much time into them. They seem to be available in both external (standalone) or internal (hard-drive form-factor) varieties. I didn't think they was anything particularly magical or esoteric about them. Just another computer peripheral (although one that costs big bucks).
 
If your company has the funds, The Cache-A system of LTO5a along with Cat-DV handling database end of things is a good solution.
 
Taking a broader view of this -- do you want to be an IT person, or a video content producer? There's all kinds of nifty hardware and software out there for backup and archiving. But all of it costs money to buy, takes up space, consumes electrical power and, most important, requires skilled and dedicated people to operate and organize effectively. It will also all need to be upgraded and replaced every few years -- and don't forget the time and costs involved in retraining your staff to work with the new technology as it comes along. One of the key business tenets I've picked up over the years is to define what business you're in, do by yourself the parts that make you unique, and outsource as much of the rest as possible. You probably wouldn't want to employ carpenters and electricians to maintain your own building, or a mechanic to fix your production vehicles when they break. Similarly, for mass storage of data -- and that's all it really is, yours isn't more or less special or valuable than any other business' key intellectual products -- it makes sense to me to find a reliable outside service that does this work on a mass scale for clients with data storage needs similar in scale to your own, and contract with them rather than trying to build your own infrastructure. If they're serious and dedicated about their business, they will stay up to date with technological changes and, because they achieve economies of scale by consolidating many customers' data storage into one massive (and hopefully highly redundant) system, can do it with fewer people and less dedicated equipment than you could in-house, therefore making it a cheaper and more reliable solution over the long haul. And you can employ more of your resources in making new, great programs rather than worrying too much about archiving the past ones.

Take this as one of life's lessons learned from someone who was several times hired to produce content, wound up running way too much of his own IT infrastructure, then got phased out when the entire operation was outsourced. Result: little content produced, and no IT job (despite much knowledge and experience gained) to fall back on.

- Greg
 
You must log in or register to reply here.
Back
Top