Advice on a long term storage solution. Go for a RAID?

O

oxband

Well-known member
So, I'm accumulating a bit too much footage from various things to keep using G-tech drives. And I'm about to start editing a project where I have about 4 GB just in AVCHD files just from filming, files that haven't yet expanded from being transferred.

I need to start thinking long term about storage.

Does that mean getting a RAID setup and just buying a bunch of external hard drives for it? I guess I could use externals in cases, like G-Teach drives, buying one drive for editing and a cheaper external just for copying it as a backup instead of editing. Are there other options? I'm not sure where to go. I just know I need to start planning.
 
ive been laying off things to naked drives in docks for years now with no incident.

if its just storage this is probably the easiest and cheapest route. need RAID redundancy, just get two drives and make a second copy. it will be cheaper and less of a headache than a raid enclosure.
 
For proper, reliable archiving, I'd be looking at something like LTO tape, or Sony's newer archival disc technology, both of which are rated for 50 years redundancy I believe.

The costs of these systems are a lot more reasonable these days than they used to be.
 
Every few years I look into LTO, and I always find them to expensive for what they are.

Let's say I need to backup my 15 TB RAID.

LTO 6 Drive is around 2200 bucks, you need a SCSI or SAS card $300ish

Than I would need at least 6 LTO 2,5 cardridges = $300

That's 2800 bucks - that's like 4 times the price of my 15 TB RAID
It would be cheaper to buy two more 15 TB RAIDS and have two backups.
Or even cheaper to buy 10 3TB HDDs and also have 2 Backups if I want that.
 
If 2 LTO tapes is too expensive, then 2 copies on powered-down hard drives is reasonably safe, if stored properly. Two things I recommend: don't store both copies on the same brand of drive. This is to reduce the risk of a design fault taking out both copies simultaneously. Secondly, power them up and do a scan once a year.

I see many recommendations for optical media, quoting how 10+ year old CDs are still working fine, but I'm very suspicious of the longevity of writable Blu-ray. I have CDs from 1996 that still test fine each year, but they cost over $7 for each 600Mb in quantity.
 
Yup! I'm running outta room too! It's amazing to me that there are no leaders in the archival dept who have reliable, permanent and reasonably priced storage options. Especially given the explosion of data these days just for casual users! That said, I've just tried to find "highly rated", reasonably priced 1-2TB internal hard drives and store them safely in different locations.

I concur with a poster NOT to get 2 or more drives in the same batch, or manufacturer if possible. This is just in case they have a bad production run of a certain brand. I see that all the time in reviews. A guy buys 6 drives from a source and 4 of them crap-out. I don't trust the bigger drives (3-4TB) just yet. I'll wait for them to shake-out. Also, the bigger static drives still seem to be having their problems in the field. I always feel that internal HD's are the best storage options in this age. At least it is "assumed" the drives will be used constantly, so they are made a little more durable. YMMV.....Good Luck!
 
All hard drives will fail. Every. single. one.
The only question is when? Today, or in a few years? A decade? Twenty? True long term storage is looking way beyond a couple decades and so they don't rate hard drives as acceptable for that kind of time span. That's why LTO has value. Regardless, if the data is valuable make multiple copies and keep them at separate locations to guard against theft or disasters like a water pipe bursting or a hurricane.
 
LTO if you have the money, they even make a USB3 LTO6 drive, it's expensive though.

RDX is you don't have a lot, but want to spend more than using blu-ray discs.

Blu-ray discs is the cheapest option.
 
BluRay discs will take a long time to write and I don't think they're really going to be that great as a long term solution. A lot of the DVDs burned ten years ago won't read today due to fading dyes and other unexpected issues with recordable discs.
 
I really think the best "realistic" solution for non-enterprise budgets is raw drive archiving with docks.

I keep a folder for archiving (completed projects) and dump them onto raw drives in a voyager dock. The drives go in those plastic static-protector boxes. And, ummm... well, they're stacked in a closet.

At some point I'll buy a wholesale case of drives and duplicate each archive drive and store it off-site somewhere - at least I'll have 2 copies.

I do constantly have to go back to older archives and find some footage or revise an edit (I've had my biggest client for something like 15 years). So cycling those drives and docking/undocking? Someday one will fail. And I don't know if tape is any less fragile than a hard drive, realistically speaking. At least drives give you extremely fast access to your files. I used tape in the SCSI days and un-archiving took ages.

There's cloud backup, but with the work members of this forum do, that might be an option for duplicating hard drive archives - I mean, how long does it take to upload a terrabyte to the cloud? Seems like a third-copy last resort backup solution - dump a terraybte drive worth of archive to the cloud when you'll be away from your desk for a day. Or three.
 
Multiple LTO copies is a proven solution. Of course for some of use low-level operators, it's cost prohibitive... albeit not as cost-prohibitive as losing vital footage.

Anyway... here's the general system I have, which has only really become feasible in the last few years:

I have two RAID 1 arrays at home... one USB and one on a NAS. These store home media (music, video) and business media (documents, invoices, licenses, etc).

iMac hard drive holds system and app files.

Working media goes onto an eSATA RAID 5 stack of 4x 2TB hard drives (OWC, I think).

Here's the semi-innovative, possibly risky and self-destructive part:
Once a project is finished, it goes onto multiple standalone EXTERNAL USB hard drives or bare 2.5" HDD's (I keep a few 2.5" USB enclosure kits around as well). For a while I was using bare dockable 3.5" hard drives, but switched to this recently. Here's why:

- disk prices are always falling as storage sizes always rise (my entire working drive from 5 years ago fit on a 160GB hard drive)
- I have a huge binder full of old small projects on 4-8GB DVDs. At the moment I don't even have an optical drive on my main editing machine. I wound up copying all of these to HDD and pitching the DVD copies... took up much less space and easier to access.
- I have racks of DV tapes (long said to be a great archival medium). Going back to one project, the tapes started to suffer dropouts. I went through and recaptured everything to HDD (far easier to access that way). Never had a dropout on that old footage since.
- For many shoots, I am asked to ship hard drives of raw footage. If I ship a bare HDD, they'll get justifiably PO'd, but are thrilled if they get a decent external hard drive.
- Roughly every year or two, I can effectively swap out my entire set of backup drives. Two old drives' contents will fit on one new drive. The old drive is big enough to ship footage after a secure erase, and I'm reimbursed for it on the job invoice. In another year or two, I repeat the whole process - larger, newer archival disks, send off the old and get a few bucks in exchange. It keeps the disks new enough to minimize the chance of failure and always keeps my archival medium current. How silly is it to have '50 year archival Blu-Ray discs' if, in 50 years, we won't be able to find a Blu-Ray drive outside of an antique store?

So far it's working. I'll let you know how it's going in 50 years... or not, because I'll probably be a little bit dead.
 
Andrius Simutis said:
All hard drives will fail. Every. single. one.
The only question is when? Today, or in a few years? A decade? Twenty? True long term storage is looking way beyond a couple decades and so they don't rate hard drives as acceptable for that kind of time span. That's why LTO has value. Regardless, if the data is valuable make multiple copies and keep them at separate locations to guard against theft or disasters like a water pipe bursting or a hurricane.
Click to expand...

Mechanically perhaps.... but if HD's are used for storage only I don't see them failing, ever! I agree that magnetic tape has its value but its not the end all solution. The need exists for another reliable, long term solution. Not to mention cost$... I have enough stored data to filled thousands of dollars worth of tape

Hard drives have worked for me for the last 15 plus years, with only one or 2 mechanical failures (the data was retrieved). The only data lost so far, that I remember was my own stupidity! My biggest problem is the draw full of old 40gb IDE's I need to upload to my newer drives.....ugh!
 
ronedee said:
Mechanically perhaps.... but if HD's are used for storage only I don't see them failing, ever!
Click to expand...

Just because you can't imagine them failing doesn't mean they don't.

I have been working with computers for 36 years now, and the fact of the matter is…. nothing is foolproof. At some point no matter what you backup to, a loss will happen.

I have had hard drives, tapes, CDs, DVDs, Blu-Ray discs, all fail over the years.
 
I didn't say they "don't" fail.... I listed 3 things combined to make them worthy of my use: 1. In "theory" they are made to be used. So are durable. 2. They are used as storage. So they are NOT being used constantly. 3. They are reasonably priced.

I've had better luck with HDD than all else that you've mentioned.... So I'm confused why you are calling me out.

So to make a point, and ask you a question: How does a HDD fail sitting in a box? And even "if" they do at some point won't the data still be there?
 
ronedee said:
So to make a point, and ask you a question: How does a HDD fail sitting in a box? And even "if" they do at some point won't the data still be there?
Click to expand...

I think drive failure's unlikely in storage. it's swapping them that could damage connectors or introduce a static spark, I suppose.

I understand a drive can get "stiction" when sitting unpowered... the drive head finally succumbs to the magnetic pull of the disk and attaches to the drive surface. Not an expert on that and could be some urban legend in there, and newer technology may address that.

Storage is subject to the environment - if your roof leaks onto your drawer, you might have trouble, or if your house burns down. Good data preservation comes down to multiple off-site copies... and that doesn't address years or decades of media that's pretty much considered "disposable". I found a raw drive the other day, but it's (I dunno, IDE? Can't recall) and I don't have a single thing that will run it, an older technology. Hell, I still have some 20MB Syquest drives with old work archived on 'em. Hope I don't need anything from the 90's!

My garage is a falling down wreck, slated for tear down and a nice new one built. Considering sticking a small fridge out there and running Ethernet into it and using it as my backup of my nightly backup, and extra drive storage. You can set them up to be a very dry environment, and cool. They say the last thing to go in a fire is the fridge (so keep your passport in there - plus nobody tends to check your fridge when they're after jewelry, guns, and oxy!) I'll need to research this big idea of course...

I really need to instant access of HDDs for archival stuff (unless tape is nowadays instant access). Too many times I need one clip from a project and have to open a lot of files to find it. When I backed up to DAT, it was a nightmare recovering files.

I'm coming around to the belief that affordable archiving is this:

Archive to raw HDDs and store them properly;
Duplicate to a 2nd HDD and store off-site;
Pay for the fastest internet connection you can afford, and store a third copy in "the cloud" for emergencies.
 
You must log in or register to reply here.
Back
Top