Massively increasing storage needs of HD workflow

Blackout

Blackout

Veteran
Just wondering what some of you are doing for the massively increased storage needs of HD workflow?

Currently, I bought a BUNCH of 2 terabyte samsung drives, for $75 a pop you can't beat them on newegg, and I combine them to make 2 4 terbyte raids for actual working, and then I have about 3 of them in a tower as single drive for storage / backup of original masters and master timeline backups. Performance is amazing and blazing fast even though they are 5200 rpm they out perform my seagate caviar blacks.

I then for triple safety burn to dual layer 50gb blu-ray discs all my 32gb memory cards before I format them. Things are starting to get crazy and soon I won't have any more drive bays left in my antec 1200 tower.

- Blackout
 
What you're doing is basically sound. But, I hear you about running out of drive space. Obviously, you can add on more drives in external enclosures, or you can outfit your computer with these:

http://www.amazon.com/gp/product/B0028Y4DAO/ref=ord_cart_shr?ie=UTF8&m=ATVPDKIKX0DER

This enables you to hot swap hard drives. I have them in all my machines and can't imagine living without them.
You don't have to buy this particular one. There's lots of similar products around. Newegg has a bunch.
In fact, here's an intersting one that has built-in fans:

http://www.cooldrives.com/index.php/mrsatrmorawi.html

Hope that helps.
 
Blackout said:
Just wondering what some of you are doing for the massively increased storage needs of HD workflow?

Currently, I bought a BUNCH of 2 terabyte samsung drives, for $75 a pop you can't beat them on newegg, and I combine them to make 2 4 terbyte raids for actual working, and then I have about 3 of them in a tower as single drive for storage / backup of original masters and master timeline backups. Performance is amazing and blazing fast even though they are 5200 rpm they out perform my seagate caviar blacks.
- Blackout
Click to expand...

Back in the olden days, when I attempted to get on the entrepenural escalator of success, the product my partner and I were working on was a device to support Network Attached Stoarage... these days such things are ubiquitous... but alas... we were too early for the wave...

Anyway, I think if you do have this sort of storage need, you should create your own archive system. I was told like 4 days ago, that the cross over point has been reached that Gigabit Ethernet is now outselling 10/100 based systems world wide. The main point there being you can buy Gigabit switches for 'pennies'... I got rid of my fibre Gigablt switch years ago... big space heater... and retail price appropriate for such...

So you can probably create an archive fabric to handle multiple T's worth of storage, perhaps not fast enough for 'realtime editing', but at least 'reasonably fast' access for getting the media to a disk that is faster...

I've seen bays with removable disks as well, so, you could create a combination of online and offline storage for archiving... all you need is the robotic arm to access the offline disk packs... (you can also savor the thought that this was part of the Multics Operating System, but back then it was tape that was being transported from a warehouse to the tape reader, via robotics... to implement that 'if ever it was' function...)

But basically it sounds like you need a storage farm...
 
i simply have a raid scratch disk for projects in post and archive to external hdds when i'm done. i've never needed to restore an old project, but have them saved just in case (delete transcoded footage, keep avchd). i could probably get away with saving just the final mastered videos.

so i guess i don't really see what the big deal is. when was the last time you had to go back into a project you edited a year ago?
 
I bought a BUNCH of 2 terabyte samsung drives, for $75 a pop you can't beat them on newegg
Click to expand...
i dont scrimp on drive mechanisms. seagate baracuda, wd caviar black or nothing else
 
wgzn said:
i dont scrimp on drive mechanisms. seagate baracuda, wd caviar black or nothing else
Click to expand...

Got any data that brings the Baracudas and Caviars above the rest of the crowd? ;)

Some Hitachi hard drives are apparently phenomenal regarding reliability, according to a Russian data recovery centre. More expensive isn't necessarily better.
 
Got any data that brings the Baracudas and Caviars above the rest of the crowd? ;)
Click to expand...
other than nearly 20 years of service with no data loss. and all drives (that im still in possession of) working as new?
no.
 
averan said:
i simply have a raid scratch disk for projects in post and archive to external hdds when i'm done. i've never needed to restore an old project, but have them saved just in case (delete transcoded footage, keep avchd). i could probably get away with saving just the final mastered videos.

so i guess i don't really see what the big deal is. when was the last time you had to go back into a project you edited a year ago?
Click to expand...

It's rare... but it's happened... especially with stand up comedians and actors... an old client calls 6 months later and wants a shorter tighter re-edit. Easier to load up the original project than to start again from scratch raw files.
 
wgzn said:
i dont scrimp on drive mechanisms. seagate baracuda, wd caviar black or nothing else
Click to expand...

I mistyped my drives... it's wd caviar black and baracudas.... and I am experimenting with the samsungs. Seagate has really taken a down turn.. go on newegg and go on forums and MANY people are seeing seagates fail lately. I had TWO new seagate drives fail within 6 months. I'm done. No more seagate for me.
 
Blackout said:
It's rare... but it's happened... especially with stand up comedians and actors... an old client calls 6 months later and wants a shorter tighter re-edit. Easier to load up the original project than to start again from scratch raw files.
Click to expand...

The Wife did wedding photography for years, and pre-2001, all of it was 'film'. We have had a couple of customers call and ask for wedding photos, even their official Album... what we have done is quoted them a price for digitizing their film (the Wife saves everything...), then if an 'album' was to be delivered, but they never got around to specing it out... we qoute them a fee for the changes in how things are processed. We have had two types of responses... 'ok, I pay the additional fees because it has been a long time'... or 'no thinks'... In the case of the 'no thinks' I will bet dollars to donuts they scan their 'proof' albums and get those reprinted... as the Wife never had the proofs water marked... the trusting soul that she is...

In the post-2001 era, we pretty much quote a 'reupload fee', and put the proofs back up on line, and have the customer go from there...

The Wife hasn't been shooting weddings of late, so, this is now pretty much for weddings that are pre-2007 or so...
 
Blackout said:
I mistyped my drives... it's wd caviar black and baracudas.... and I am experimenting with the samsungs. Seagate has really taken a down turn.. go on newegg and go on forums and MANY people are seeing seagates fail lately. I had TWO new seagate drives fail within 6 months. I'm done. No more seagate for me.
Click to expand...

i've seen the same trend and have been running samsungs for a few years now with not a single problem. so good so far.

re: the re-edits
transcoding my raw clips to edit only takes an hour or two at most. its not much hassle for me to do again, reading avchd files straight from the archived external sata and writing mjpg clips to the scratch disk.
 
averan said:
i've seen the same trend and have been running samsungs for a few years now with not a single problem. so good so far.

re: the re-edits
transcoding my raw clips to edit only takes an hour or two at most. its not much hassle for me to do again, reading avchd files straight from the archived external sata and writing mjpg clips to the scratch disk.
Click to expand...

I'm assuming then you're on mac? I don't transcode I edit natively... I just can't see the point of transcoding when Edius 6 does 10 bit like butter and Sony vegas 10 does 32 bit full color not quite as smooth but fairly well and with image quality that is simply insane.

Just popped in another Samsung 2 tera and getting 130 megabyte /sec transfer rate from one samsung to another in copying very big blu-rays. That's my last sata port on my evga x58 mobo though.... even though I have 2 or 3 more bays... can one "daisy chain" sata drives?
 
Buy 2 of the NewerTech drive docks from OWC and drop in drives pairs as RAID 1, via sata connection, when the drives are full, store them and drop in a new pair. No need to worry about big boxes of drive bays.

These guys created the idea from digital teching huge photo shoots for years: http://imagemechanics.com/products/

Shoot, fill, store, get more bare drives. Boxes are the worst thing to spend money on now with Thunderbolt as the now, new and future Mac interface. You think you have speed now, choose faster RAID configs for transfer speeds that will be insane even from a laptop.
 
I just got a laptop that has a raid in it (unheard of) and usb 3 and I was amazed to see a 36 gig blu-ray transfer in 3 minutes over usb 3. The thing that scares me about raid though is twice the chance of failure (one drive fails and the raids dead)
 
Had 2 Seagates failing with write errors last week, both from the same batch. I have some Samsungs which have had no problems, except the thinner Samsung 3.5" drives get awfully hot and one failure. If you install a SMART utility, it checks the health of the drives and ( theoretically ) warns you to back up the drive before failure.

http://www.passmark.com/products/diskcheckup.htm (WINDOWS)

http://sourceforge.net/projects/smartmontools/ (WINDOWS)

http://www.volitans-software.com/smart_utility.php (MAC)

http://www.corecode.at/smartreporter/ (MAC)
-----------------
 
Last edited:
Blackout said:
I'm assuming then you're on mac? I don't transcode I edit natively... I just can't see the point of transcoding when Edius 6 does 10 bit like butter and Sony vegas 10 does 32 bit full color not quite as smooth but fairly well and with image quality that is simply insane.

Just popped in another Samsung 2 tera and getting 130 megabyte /sec transfer rate from one samsung to another in copying very big blu-rays. That's my last sata port on my evga x58 mobo though.... even though I have 2 or 3 more bays... can one "daisy chain" sata drives?
Click to expand...

When I browse/surf the adobe hardware forum Harm Millaard recommends a dedicated Raid 3 Card with its own memory and processor for this purpose.
Now there is no need to transfer files to anywhere.
Just have one large Raid 3 array that you edit natively from and your set. Raid 3 operates different than Raid 5 and is more suited to large file transfers and large data I/O streams (like video editing).

The Antec 1200 holds 12 drive bays, 3 of which are normally external. Lets say we use one drive bay for the OS and put the 8 remaining bays with 2TB disks.

In Raid 3 one disk is used for parity and its advised you put one drive as a hot spare. So with 8 Disks in the array (1 for parity and 1 for hot spare) you get 6 useable drives for a 12TB raid with full security against hardrive failure plus the speed of *near* raid 0. Definately fast enough to edit multiple HD streams with it. The other good news is Raid 3 doesnt suffer any significant performance hit when a hardrive fails. You can still edit with not discernible difference in performance. Raid 5 slows down like syrup and you have to wait x hours for the array to be rebuilt.
So go raid 3.
The downside? Raid 3 cards are only made by Areca and the ones that handle 8 drives (like this) start at $450.

Thats all my understanding of it anyways.

Edit: Just saw this post by Harm that kind of explains all the above in a nicer way.
Also I must add that there are cheaper Raid 3 Cards from Areca like this but they only handle 4 drives in an array.
 
Last edited:
You must log in or register to reply here.
Back
Top