Cost effective AND future proof hardware strategy for archiving

E

eco_bach

Well-known member
Not currently doing post work full-time yet so don't want to break the bank, but I need to start seriously considering a long term reliable archiving strategy.


I'd like to rule out tape because ease of access is a high priority.


I am currently running a 2 yr old iMac(that new Mac Pro looks sweet...maybe Xmas??) with 3 external lacie drives for backup-archiving. 2 1 TB and 1 6TB striped Raid. (3TB capacity)


But as I am increasingly shooting 4K I find I have a serious issue when it comes time to archive.




How is everyone backing up & archiving?


Archiving is a higher priority for me right now.
 
I am actually running Always Synch to backup my 2TB "live" video drive in real time to another 2TB backup drive.

I regularly swap my 2TB backup with another one. I keep one or the other in a safe at my bank.

This is my live work / backup loop.

Once a project is over, I get it out of my live loop onto another hard drive that is only used for archiving of projects I am no longer working on.

I also try to keep only projects that are my own or that I know I might come back to. If I am filming a concert for a client and then give him a copy, I erase said concert after a while, but I'll always keep my Short Film projects intact.

There is two shortcoming to my approach:

1 - A hard drive has moving parts. And thus, shall one of its head fails, the HDD drive is pretty much lost (OR I could get lucky with a super pricey data recovery service).
2 - I should most probably consolidate projects that I know I won't get back to for recutting. This would save a considerable amount of space since I'd get rid of all the rushes I'll never use.

And I'm only working in HD, when I get to 4K I'll need to be even more strict with what is worth keeping, while still needing to get bigger drives.

Unfortunately, for quick access and low cost per Gigabyte, there is not much alternative to hard drives.

Blu-Ray disks have small capacity, are slow to burn and read.
Solid State Drives are fast but considerably more expensive for way less capacity.

But in any case, maybe you should consider your Live Loop (Live Drive + Rotating Backups) and Archiving Storage as two different things.

I think that for a live loop, used for current projects, hard drives are perfectly fine, even with a disk failure, you still have rotating backups.

As for your Archiving Storage, this is a totally different matter. I work in the IT business and I can tell you that big companies, even banks, still use tape backups and optical drive for archiving.

It is slow and clunky, but to this day, we haven't found a better, or more reliable long term solution for archiving data (to my knowledge).

Archiving is not just a mere workflow dilemma, it is a real technological challenge for which we haven't found any foolproof solution yet.

One section of the movie Side by Side ( http://www.imdb.com/title/tt2014338/combined ) about the evolution of filmmaking from an analog to a digital realm talks very eloquently of the archiving problem.

Even after only 10-20 years media can become unreadable, either because the support rots or because players are no longer available.
 
Last edited:
Should you ever really delete ANYTHING though? Never known when someone will come back in 5 years to repurpose some footage for something else. I would hate to be the guy that had to tell em "that footage doesnt exist anymore." Guess you could always give client the raw so at least they have it, but id be wary of completely destroying anything.
 
Yes, it is a really tough decision, something that kinda makes you miss good ol' DV tapes you can stack in your attic along with your film negatives.

But then, if you truly want to keep everything, and keep it "forever", the better solution as of now still seem to be Tape Backups. But it's not exactly cheap (This is just an example, but regardless of the hardware maker, the price seems to be around the same):

http://www.bhphotovideo.com/c/search?Ntt=cache-a&N=0&InitialSearch=yes&sts=ma&Top+Nav-Search=

The question is then, once you gave everything to the client, maybe even the rushes, is archiving still your responsibility?

Is being in the business as a filmmaker / videomaker necessarily implies that you also are providing an archiving service which can quickly cost you a lot? Should you then charge for it?

I don't have an answer, but it sure is an important question to ask I feel.
 
Well, for our biz, cloud backups are pretty much a non-starter...

I keep my active work drive space to 2TB and backup nightly to a 2TB drive. When 4K is more prevalent that will grow I'm sure.

When jobs are done, I use a Voyager dock and archive to hard drives. I keep a searchable spreadsheet of what's on 'em. I also rotate drives, every 6 mos. or so my main drives are replaced and the used ones move to archive duty.

Yes, those archive drives are subject to failure - they're kept in cases and rarely powered up, but still... it's a weak link. I really should make 2 archive copies and store one off-site.

My garage, at this time, doesn't have power and is more of a shack. In a year or two we plan to have a solid garage built, and I think I'll run ethernet to it and keep a 2nd backup drive in something like a small refrigerator on low with humidity dialed off... dry and cool (this is texas!). Insurance people note that the contents of your fridge are the last to go in a fire - really takes a hell of a fire to get in the fridge (many people say keep your passport in the fridge).
 
There is no income from archiving, the idea that they may need it sometime in the future isn't the same as them paying you to keep it. So I'd try really hard to avoid archiving anything long term. If they can keep reusing old stuff they have no reason to create new things, and creating new things is an income generating business. As a result I'd say hard drives for short term archives is fine, should last at least 3-5 years. After that it's gone.

If the client wants an "archive" sell them all the footage and give it to them on a hard drive or blu-ray.

For my own projects I just keep rotating hard drives. I usually have two full copies live and a few older spare hard drive copies as backup. Every few years buy new even larger hard drives, swap those into your main and backup drives. Retire those main and backup drives as "spares". It does mean I have a large plastic tub of old hard drives (half are IDE even) but my main "world" is still here, still backed up. No protection from an external disaster though. Maybe these new LTO6 tapes might be a good solution. I do have some stuff on cloud storage and blu-ray discs but there is no practical way to put everything on the net.
 
On the Offsite Backup subject, I'd like to get back with the safe deposit box solution. Check with your bank: for a good size box that can hold say 6 naked hard drive (and nothing else), it's really not that expensive.

Around a hundred bucks a year last time I checked with mine.

The whole point of an offsite backup is precisely to protect you in case something happens to and around your immediate workplace, and thus your backups, if they're too close.

Unless you live in the same building as your bank, which is about as unlikely as your deposit box getting stolen or destroyed, you're covered.
 
Those with bigger budgets will offer better solutions (such as tape).
I use multiple external hard drives for each workstation with redundant copies of footage on 2-4 drives each. Plus a dedicated drive on each workstation where I keep footage that I foresee needing in the near future. I also have another external drive located off site.
Not the best solution but it's what I've been doing for 1.5 decades since I stopped using optical discs for backup. Fortunately I haven't had any data loss yet.
I have wording on each contract that archival is responsibility of client
 
For low cost universally readable robust digital archiving I'm using M-discs. 25GB archival Blu-ray media that is pretty much impervious to anything other than physical damage. Not sensitive to light, moisture, or fairly severe temperature swings. It is not common phase change or dye based media that suffers bit rot as the organic dyes degrade and it is non magnetic. It is an inorganic mineral based medium, literally written in stone.
Only problem is it takes a lot of them, 40 discs/TB, about $200. So only worth the effort for stuff you really need or want to keep long term . Writable and readable with standard Blue-ray data drives, and video players if written to BD format standards.
http://www.mdisc.com/
 
Well, your question is difficult and the answer is even more complicated.

If you want a REAL backup and archiving solution, I think the best is LTO tapes. Highest capacity is 6TB. Don't know the exact price but it's very expensive.

Unless you have the actual need and money, better leave it to big companies.

For the rest of 'us" and by that I mean maybe 99% of people here on dvxuser, that need a (uni)personal solution, better invest in a NAS with enough drives that fits on your budget.

Do a RAID 5 or better 6, depending on the tradeoffs you are willing to have between disk space and data redundancy. Maybe invest directly in an 8 or 12 drives bay. You can start with buying 4 drives and then fill it in time. 4TB drives are not that expensive, you can start with this capacity and change for bigger drives when technology becomes available and prices goes down. You're kinda future proof that way.

You can easily access your data by having you NAS on your local network but don't think to work on it as you'll be limited to around 100MB/s.

At home I have my first NAS with 4x1TB drives, RAID 5 so around 3TB capacity and recently got a second one with 4x4TB for about 12TB of space in RAID 5 also. It suits my needs for the moment but next one will probably be an 8 or 12 disks enclosure. I hope 6TB drives will get cheaper in the mean time.
 
Last edited:
msorrels said:
There is no income from archiving, the idea that they may need it sometime in the future isn't the same as them paying you to keep it. So I'd try really hard to avoid archiving anything long term. If they can keep reusing old stuff they have no reason to create new things, and creating new things is an income generating business. As a result I'd say hard drives for short term archives is fine, should last at least 3-5 years. After that it's gone.
Click to expand...

I've gotta say, with as much respect as I can muster... this is just insane advice.

I've got clients going on 15 years. If I were an employee at my biggest client, I'd have the most seniority short of the sales dept. And my archives are a definite money maker. I can pull up all kinds of things to flesh out new projects. They're constantly doing short social media edits to get landing page clicks, and I've got a decade of customer interviews on every subject and benefit and footage of every aspect of their business. It's probably one of the top reasons they stick with me (and they're often 70% of my income). And we do plenty of nice new work, too.

I'm not in the business of forcing clients to "create new things" or making them archive stuff that I often need in minutes. They think it's awesome that during creative discussions I can bring up footage from years back that helps flesh out a new project, and I can have it in an edit the next day.

What I AM in the business of is one thing and one thing only - helping my clients be more profitable. I happen to do that primarily with cameras and audio and animation. But I also know their business and products inside-out and end up discussing strategy, marketing direction, etc. If that means I keep a closet full of raw hard drives and a spreadsheet of what's on them, that's a miniscule, microscopic, nothing of a hassle compared to the perception they have that I am taking really good care of them. (And yeah, when I quote for new clients, I make sure that "Archiving" is listed as part of the bottom line).

MSorrels has a great attitude if you have a constant flow of new clients. I've found the long-term approach works for me. I rarely even quote on gigs for my best clients. I rarely show a reel or a portfolio, even for new business. I'd much rather have a roster of decade-long relationships than spend all my time chasing down new work.
 
You must log in or register to reply here.
Back
Top