The Storage Crisis

HowdyDoo

HowdyDoo

Well-known member
I can't help but feel that with the advancement in camera technology (i.e. tapeless/filmless video and still cameras) the world of archiving and storage is still 5 years behind. I currently shoot with a Sony EX1 and an HVX200. I just sold my HVX and am about to purchase a Canon 5D which shoots 21 MP stills. As it stands , I have over 20 hard drives holding archived projects. Hard drives are NOT ideal for archiving. And the more media I capture, the more hard drives I need, and the more likely I will loose something in the process.

I was hoping that Blu-Ray media would be much cheaper by now, but it's still a bit pricey.

Does anyone know of an alternative yet stable method of archiving media??? Anything that will last years?
 
I don't know either. Hard drives all fail eventually so for valuable footage you really need to double your number of hard drives to have back ups to your hard drives. I've got a large collection of ac power warts to keep up with too. It makes me wonder about the future. 50 years from now, will we have the the same legacy we have with film?
 
We have a VXA based tape back up system. I dont have the model number in front of me but we got the rack mount ($2400). Then each tape is $100 and can store up to 600GB. I know there are really cheap desktop versions of it as well.
 
Hard drives are perfect for archiving. Firstly the permativity of a metal platter is waaaaaaay higher than tape. Secondly the electronics, motor, solenoid, and bearings don't wear-out in storage. Thirdly, IDE, EIDE, SATA, SATA-II, SCSI will be easily interface-able for decades to come. Good luck trying to find the right tape drive (and drivers and software) 20 years down the road. I have tapes from the mid nineties that were unplayable in only three years because the manufacturer went out of business.

I have 5 Meg Winchester hard-drives from the early 80's (one from 79) that still work (on a Onyx minicomputer I restored a decade ago).

"Data doesn't exist unless it's in three places." Copy your data to three drives and store them. I have drives going back to the 90's with my video archives on it. I now go through about a 500 gig drive a month (cheap, and the client pays for it anyway).
 
I have over two hundred drives here with archived media and projects on them going back 8 years. I booted up several of the older ones to "exercise" them a few months ago. Out of twelve that I tried, three of them bit the dust and would not boot.

Hard drives are horrible for reliability and long term storage. Tape is okay, either HD tape or DLT tape but even those are not proven past about twenty years so far. If you really want to get fifty or one hundred years, going to film is the only proven, reliable visual storage medium.

Dan
 
Currently, I don't think there's anything decent available, which is rather unbelievable considering how many digital acquisition tools there are now. As stated before, I have 20 hard drives and 2 of those have already failed. Each were 500 GB drives. The larger they are, the more you have to loose.

Despite all the advantages of tapeless media, it sure was easier keeping a drawer full of tape than a room full of hard drives.

Still waiting for BD-R Blu-Ray, I suppose. It sure is taking it's sweet time to drop in price.
 
pmark23 said:
"Data doesn't exist unless it's in three places."
Click to expand...

pmark is correct about storage in multiple places. Our workflow is:

-Ingest into RAID 10 drives for storage and editing/compositing/FX work.
-Back up upon completion to GSAFE cheap raid 1 drives
-Back up to deep archive VXA tapes.

Before we decided on the most dependable storage medium we went to very high profile financial institutions. The IT people told us simply backing up to HDD and putting them on a shelf is like playing Russian roulette.

They highly recommended LTO first and VXA second. We went with VXA for price but will probably change over to LTO in 2 years or less.
 
puredrifting said:
If you really want to get fifty or one hundred years, going to film is the only proven, reliable visual storage medium.
Click to expand...



But the things that I want to back up are not only images. What about my project files? It's alot more difficult to come back and work on my project if it's on film.
 
pmark23 said:
Hard drives are perfect for archiving. Firstly the permativity of a metal platter is waaaaaaay higher than tape.
Click to expand...

That's simply not true. I can take a magnet, put it on your hard drive and poof.
Also read this article:
http://www.larryjordan.biz/articles/lj_hard_disk_warning.html

pmark23 said:
Good luck trying to find the right tape drive (and drivers and software) 20 years down the road. I have tapes from the mid nineties that were unplayable in only three years because the manufacturer went out of business.

I have 5 Meg Winchester hard-drives from the early 80's (one from 79) that still work (on a Onyx minicomputer I restored a decade ago).
Click to expand...

I have a VCR and Cassette tape player in my closet, at my work we have DLT, DST and exabyte drives to restore things from our archives. Also, the backup industry has realized that changing the form factor of their media is counterproductive and seems to have standardized on the LTO tapes. We're now on LTO3 which fits 400GB per tape.

I guess the point of all this is though, no medium is going to last forever and if something is important, you're going to need to transfer it to a new medium at some point.
 
USing ULtrium LTO4 Data tape for archiving P2 footage - a great working solution!

USing ULtrium LTO4 Data tape for archiving P2 footage - a great working solution!

HowdyDoo said:
I can't help but feel that with the advancement in camera technology (i.e. tapeless/filmless video and still cameras) the world of archiving and storage is still 5 years behind. I currently shoot with a Sony EX1 and an HVX200. I just sold my HVX and am about to purchase a Canon 5D which shoots 21 MP stills. As it stands , I have over 20 hard drives holding archived projects. Hard drives are NOT ideal for archiving. And the more media I capture, the more hard drives I need, and the more likely I will loose something in the process.

I was hoping that Blu-Ray media would be much cheaper by now, but it's still a bit pricey.

Does anyone know of an alternative yet stable method of archiving media??? Anything that will last years?
Click to expand...

Well I think I have an excellent working solution for this problem after looking at alternatives for a long time prior to my 18 month old HVX-200... (my Radius VVS/TC!)

This is really straight forward if you have that many disk drives and dont trust the spinning disk. Here's how.

• use LTO4 Utrium tape drives (Data tapes). Data tape will last for 30 years when preserved in moderate conditions. being realistic and say 5 years as you move to a higher capacity an cheaper technology over time.. sorry this is how it is for any CMF storage media.
• data tape are very stable however like any magnetic storage media you need to keep them away from that Bose or Marshall or MEsa-Boogie speaker stack and away from fridge magnets.
• data tape costs for Ultrium LTO4 area as low as $HK400 / €40 / $USD50ish. Even cheaper in packs of 10! Sony is good, Imation are great .. work out what works for you. Simply this means approximately €0.05/GB | $USD0.06/GB or $USD0.06/DVCPROHD 1080P/minute-ish.... versus the disk cost and the power and aggravation that aimless spinning disks give you for material you hardly if at all access.
• capacity = 800GB or 750+ minutes of 1080P DVCPROHD (always uncompressed)
• one x LTO4 Ultrium SAS tape drive - use HP1840 .. dont use QUANTUM their support is deadful an the SAS HH drive is incompatible with several SAS HBA's - take great care when selecting else just ask those that have working systems
• one x SAS HBA (depending on your setup as a MACOSX, Linux or redmond software) - use ATTOTECH EXPRESS SAS or if you must use LSI SAS3433E HBA (no mac drivers)
• Software: well try one that ARCHIVES/RECALLS (not backup). The workflow is different. I use Tolisgroups BRU-PE (in BETA) . THis is a STANDALONE .app for OSX .. non linux nor any microsoft os needed. ON the HIGH end SGL's FLASHNET™, XENDATA, DIVArchive and 5 more.. .. these are for ARCHIVING.. also mixed platforms such as DAX and quite a few more.. me I just want to have this on my MACPRO on a single LTO4 ultium tape drive... so Tolisgroup BRU-PE is for me.

This setup absolutely works great!

Data rates for Ultrium LTO4 data tape drives when the source or destination on disk supports AT LEAST 110MB/sec! (UNCOMRESSED) else the tape drive will slow down and even wait!. This atchiving or recalling to a system wher ea single disk drive module (ddm) is the sourec and destination and where is supports far less of this data rate will cause the data tape to ls down. Lots f tweaking may be done including consolidation of LARGE objects.

I have lost many Le CIE 500GB and 1TB BIG disks and detest them for use to hold irreplacebale footage.

Long term strategy:
• make TWO instances of the material on TWO different Ultrium LTO4 tape volumes.
• us e the ARCHIVE software to manage placement of the material you wan archived
• archive all the metadata and goodies on the data tape id you must - but mostly ESSENCE
•*bank on updating to newer technologies within 10 years for greater capacities, faster sounds and cheaper storage. Yes the old data tape media media will find its way to the rubbish bin over time as several of my 1TB Le Cies BIGDISKS have found their way to the (Hong Kong Landfill).
• lower electricity/power costs instead of having disks spin aimlessly around all day in the hope you might access them for THAT object you wanna edit. Unless you have enterprise disk arrays with at least high end SAS DDms , the SATAx disks we all like will eventually fail.. or fail because a manufacturers firmware needs an upgrade an only a S/A MSDOS 6.x platform will upgrade the disk drive (oh please Seagate!) .. It's how it is.

Lots of chat about this Ultrium LTO4 archive workflow and field acquisition workflow on the red forums at http://www.reduser.net forums

non=event alternatives:
  • DVD5- lousy capacity for DVCPROHD, take ages to write out and one scratch on the surface and its all over - so use TWO instances and buy a large canister to put them in.. oh and get a sleeping bag and train your kid to change out the DVD5 writer when it fills up making sure they dont put their grubby fingers on the shiny side. DVD5's are cheaper than beer coasters at the local bar and probably as reliable these days. (sic)
  • DVD-9 DL's - still very pricy, not too much extra capacity at a usable 8GB but where I live are a relatively huge factor more expensive than ye old DVD5s, VERY slow to write - same as above for automated robotics (a la your kids)
  • BD's - yeah maybe, but very slow to write (at 40Mbits/sec on a good day - the non Taipei hybrid I mean). - expensive too with only 24GB .. well i can wait a longer and use 8 x DVD5's... same rules apply for robotic changers
  • el cheapo DISK storage: hmm .well as long as you trust SATA and cheap disk controllers. If your faith of your entire collection and business rests on a pallette full of FIREWIRE disks then you will alos be waiting for some time.. maybe to mamage the dat alost and file sturcture rebuilds.. same for USB.. else you might wanna invest in.....
  • high capacity disk arrays: sure..but the power will get you after a while if a shot spindle or two does not. THe reconstruction time should be ok as long as you have a GOOD HBA that can do it, some SPARES DDMs in the disk array rank and some fault tolerant controller and that the price of power comes down over time all for the benefit so you can have INSTANT access to that footage you have not looked at for 3 years (what about a proxy workflow??)..
  • CLOUD comtent storage; yes this is the way to go in the future - watch this take off in 4 years.. just need could comms
  • solid state?: maybe if the prices come down to the price of a box of 1000 toothpicks or a bowl of steamed rice.
  • DROBO™ like devices for VIDEO work?: some of all he above rolled in one coupled with an EFFECTIVE data rate that would make a snail impatient.

Ther ETHERNET connected data tape drives such as the SDLT600A and the LTO3A [with their MXF encapsulation] - well not for me they are way way too slow for any most large object transfer. However everything to taste you could look at these but they are way to slow and not in mainstream use. For one thing the tape cartridges for LTO3A cannot be read by a NON 'A' LTOx type tape drive.. so watch out there.

Future? Not sure yet but until something is out there that is fast, relaible, tested and economical to maintain a COLLECTION of goodies (business or other) that you want to move to new technologies over time DATA TAPE is a very cost efficietnt way at present, especially where LARGE OBJECTS such as the material we work with is concerned.

Roadmaps very and as many of us may not trust the brochureware, LTO5 ULtrium is 1.6TB with many more traks on an 610m-800m long cartridge with 220-250MB/sec native on a SAS or FC channel I'd expect. LTO6 Ultrium?? well I guess we'll wait and see. Still a way to go for the 5TB on a data cartridge promised by some storage vendors in 2003.. but its getting there. There's many other data tape formats including DDs, AIT all worth a look but these are relativey slow compared to LTO4 Ultrium. IBM's 3592 and SUN/STK's tape drives retail list at arund €40,000 each ($HK400,000)and they are used by the big banks, aerospace, geophysical and insurance.. u get the idea.. I could not get into (buy) one of those.:(

This is the tip of a huge problem for most people now as their whole life story is buried on a scratched CDR or a DVD in photos.. their clients material and their production work and one scratch and its all over.... sad.. but true.

I have this all working with Squarebox's CATDV 7.0 as a MAM with FCP .. all on a MACPRO OSX 10.5.6.

My config for the above is:
• HP LTO4 Ultrium Full Height SAS i/f tape drive (most awesome and quiet) with an
• ATTOTECH SAS EXPRESS R380 SAS HBA (awesome HBA)
• Tolis GRoups BRu-PE (beta) software
• also a QUANTUM HH LTO4 Ultrium SAS tape drive desktop (not recommended but works ok with ....
• also ... a LSI SAS3224E-R SAS HBA ( i use this wi the QUANTUM LTO4 tape because it is the only thing tat works on OSX with this silly tape drive).
• OSX 10.5.6 (just installed)
•*PROAVIO E8MS JBOD enclosure with 8 x 1TB SATAII DDMs (450MB/sec with AJA tool) and ATTO R380 HBA SAS
• lots of HP and SONY LTO4 Ultrium 4 data cartridges that dont use electricity and cooling when idle!

Costs vary but I can pass on where I purchased these components if anyone is interested.

Aside from daily work, I am presently migrating all legacy formats to a CMF in FCP media manager under CATDV™ control maintained on data tape by BRU-PE with two instances!

Yes this archive/collection will last for many years much longer and safer than spinning disk .. as several posters have acknowledged in this thread.

PM me you have any q's else post on the forums.


HTH

w
HK
 
That's simply not true. I can take a magnet, put it on your hard drive and poof.
Click to expand...

I had a job erasing tapes and destroying data on disks. Tapes can be destroyed with a very high power AC magnet (a regular magnet won't work), and drives can only be destroyed by crushing them or doing a low-level format in firmware (we had a machine to crush them -- a lot easier!)

Also read this article:
http://www.larryjordan.biz/articles/lj_hard_disk_warning.html
Click to expand...

Interesting article, and may have an effect on future drives, but as I said early I have drives going back to the late seventies (almost 30 years now) which still work. Something to keep an eye on though.

I have a VCR and Cassette tape player in my closet, at my work we have DLT, DST and exabyte drives to restore things from our archives. Also, the backup industry has realized that changing the form factor of their media is counterproductive and seems to have standardized on the LTO tapes. We're now on LTO3 which fits 400GB per tape.
Click to expand...

I have an entire storage room filled with computer equipment that is obsolete and can never be used. I have a big box of tapes from the early nineties when I started archiving to tape that can't be read.

As a former Unix tech I had to archive machines and load the backups when something went missing. There was always a problem when dealing with tapes (and that was why we always made three copies). My experience is that tapes are cumbersome, expensive, and unreliable. Maybe things have changed?

I guess the point of all this is though, no medium is going to last forever and if something is important, you're going to need to transfer it to a new medium at some point.
Click to expand...

No argument there.
 
Archive Appliances

Archive Appliances

Our company, Cache-A, manufactures archive appliances that make it easy to protect and access content on LTO4 tape.
Please visit www.cache-a.com for more information.
 
You must log in or register to reply here.
Back
Top