State of the Data, 2018: how are you managing the terabytes?

M

mcbob

Veteran
OK... so I'm choking on data. Looking for a review of current "best practices" as I redo what I store, where, and how.

I have terrible DSL connectivity, so all cloud storage methods for anything but the smallest, most vital stuff are off the table.

Current cacophonous conglomeration of crap are two 4-disk arrays (RAID5) of 15 and 6TB, a single raid-0 for family stuff, and then multiple bare hard drives fed into a BlackX Duet drive toaster as needed. Everything is nearing full, one RAID has occasional power problems and might be dying, the other wierdly occasionally just freezes the computer as it reads data from certain folders (and needs rebuild or replace). I've tried to gradually refresh drives every ~4 years so that I don't have any drives more than 4 years old, when possible. I know I do have a few older than that, relegated to storing non-critical old client stuff (just in case).

So seeing as I'm in for a complete overhaul... what are you doing that makes sense?
 
$100 gets a 6tb external HD, I jsut buy them in pairs, wrtie my rushes to both, keep one at home and the other in the office. quick, dirty.

I also have a google spreadsheet with Disk ID, job title, date, description. Eash shoot kept in a folder/directory called 'job_title' in the form 2018_07_12_MyClient_ProjectTitle

As a side note I also 'export project' into the folder from my davinci edits, so if my main pooter goes pop I dont lose my edit choices.
 
Funny, about two months ago I tried to access something archived to optical media maybe 4 years ago... unreadable disc. I abhor optical media.
 
If a drive fails, then you have a spare - whats the chance of two failing - the same two - ie 12a and 12b, sure 12a might fail or even 12a and 17b but neither situation loses data.
 
I had an external raid for editing my projects for several years, but it crapped out on me finally, and since hard drives have gotten so cheap what I decided to do was just buy pairs of off the shelf 8tb external USB 3 drives. Instead of doing software RAID, I set up a program called Super Duper to nightly copy one drive to the other. Super Duper has a smart update feature, so it will only copy what has changed after the initial copying. It also will mount and eject the hard drive being copied to, so it's not spinning up and using power when not in use. I check the log periodically to make sure all is running smoothly.
 
mcbob said:
Funny, about two months ago I tried to access something archived to optical media maybe 4 years ago... unreadable disc. I abhor optical media.
Click to expand...

Interesting, was it DVD? I've found DVD's to be pretty bad at getting scratches and not being able to read them later.
Especially the ones you burn to by laser (DVD-R or DVD+R.) I used to archive some still photos on DVD and had some
issues with a couple (although I was lucky and using one of those 'resurface' type gizmos worked for me.
However, the brand of Blu Ray disks I get from Amazon have been much better for me. They are very resistant to
scratches for some reason, and are much more durable than DVDs. I have 164 disks and every single one
of them still works. I started archiving on them in 2012 so not the longest time period to compare. But in that same time,
I have had 8 external drives, 3 internal drives, and 1 RAID system fail on me. So yeah, I'm staying with optical media.
 
morgan_moore said:
If a drive fails, then you have a spare - whats the chance of two failing - the same two - ie 12a and 12b, sure 12a might fail or even 12a and 17b but neither situation loses data.
Click to expand...

I actually had both my RAID AND my external backup drive fail on me the same day. Not only did I lose a ton of client work,
but even worse, I lost personal footage of my daughter winning a state championship. #Idon'ttrustharddrivesanymore.
 
alaskacameradude said:
Interesting, was it DVD? I've found DVD's to be pretty bad at getting scratches and not being able to read them later.
Especially the ones you burn to by laser (DVD-R or DVD+R.) I used to archive some still photos on DVD and had some
issues with a couple (although I was lucky and using one of those 'resurface' type gizmos worked for me.
However, the brand of Blu Ray disks I get from Amazon have been much better for me. They are very resistant to
scratches for some reason, and are much more durable than DVDs. I have 164 disks and every single one
of them still works. I started archiving on them in 2012 so not the longest time period to compare. But in that same time,
I have had 8 external drives, 3 internal drives, and 1 RAID system fail on me. So yeah, I'm staying with optical media.
Click to expand...

Last year I had to dig into a project on DVD I burnt over 12 years ago for a client. Loaded up just fine. I have way too much media to archive to spend time burning to disks though.
 
Batutta said:
Last year I had to dig into a project on DVD I burnt over 12 years ago for a client. Loaded up just fine. I have way too much media to archive to spend time burning to disks though.
Click to expand...
That's a very valid point. When I archive to optical disk, I can only burn 128GB at a time, because that's what
the quad layer Blu Ray disks hold. And when you have TB's and TB's of data.....it isn't ideal. I think this is
actually one of the biggest issues for today's smaller video production companies or one man band filmmakers.
There is really nothing out there that is affordable, reliable, and fast.
 
FreeNAS Mini with as much storage as you can afford. 8 drives are optimal because you can get decent size with super redundancy (Zraid3). Or build your own with one of the many good youtube tutorials like the ones from lawrence systems.

LTO backup is always nice, but more expensive than a second FreeNAS that you use to hold snapshots.
 
alaskacameradude said:
That's a very valid point. When I archive to optical disk, I can only burn 128GB at a time, because that's what
the quad layer Blu Ray disks hold. And when you have TB's and TB's of data.....it isn't ideal. I think this is
actually one of the biggest issues for today's smaller video production companies or one man band filmmakers.
There is really nothing out there that is affordable, reliable, and fast.
Click to expand...

Maybe SSD's will be cheap enough soon to be viable. I've read conflicting info about how long they will last. There was a study that got blown out of proportion saying you could lose your data if an SSD is unplugged for 7 days, but that was for a drive at the end of its lifespan in a harsh environment. It seems like with no moving parts, it should be more reliable over the long haul as an archive format.
 
It'll be interesting to see how reliable SSD's turn out to be in the longterm. I've heard that when an SSD fails, it's gone, you lose everything. Versus when a spinning HDD dies, it's usually just the disk that won't spin-up or the heads won't engage, so if you want to spend the money, there are several companies that specializes in reclaiming/rescuing the data off of those disks.

Too bad we can't just chisel it all in stone and know it'll still be around in 10,000 years.



I did just pick up a SanDisk 1TB Extreme SSD a few weeks ago, "on-sale" ~$279. I wanted a HD I could carry around in my camera bag and not have to worry about taking up space, adding weight or getting knocked around. This thing is ridiculously small. Maybe half the size of my iPhone X.
 
I would hope that the FreeNAS mini devices are big enough that coffee is not an issue, but the others are always a concern. I will say that this summer I suffered electrical gremlins with both of my FreeNAS servers, both were corrupt boot devices. Both recovered well enough that no data has been lost, one of them needs more work but the other is running fine again. It's a fairly durable configuration which normally can be recovered by moving data drives to a new system.

For files smaller than video, snapshots (even encrypted) can be automatically moved to some popular cloud services, not sure it makes sense for most of us to buy a fast enough connection to make this possible for video. Around here you are lucky if you can get an honest 10mbps upload. If my money depended on it, and I had the opportunity, I'd locate a second server in an out building connected on fiber, that way fire would be less of an issue. I would probably just send automatic snapshots to it each night. Long term storage would probably go to LTO tapes stored off site, not sure Bluray backups are the answer yet, I've had too many coasters on high end disks to think it will be OK.

As far as SSD, one needs to be careful what type of technology is used to make the drive, TLC is very slow on writes once the buffer runs out. SLC and MLC are better for big writes like we sometimes use.
 
Hardly scientific, but I've had way better luck with external drives as opposed to internal. I've had 3-4 internal drives fail on me over the past 3 years. Haven't had an external drive fail yet (knock on wood).
 
Many ways to approach this. I'm assuming the OP is thinking about working media and storage media.

Storage media should be spinning disk drives, as opposed to SSDs. This is because in case of a failure, it's still possible to recover. If you can afford it, RAID is great because of redundancy. But you'd still want a replica of that RAID in a separate location in case of a fire, flood or other type of severe event. Happened to me once (pipe burst). Personally, I can't afford multiple duplicate RAIDS as back-ups in separate locations, so for storage, I just buy pairs of cheap 8TB spinning drives and duplicate them, storing them in separate locations. Don't need them to be RAID, because I think duplicating data into two separate drives is redundancy enough since if both of them fail, they are spinning disks so I'd still be likely to retrieve the data. My compromise is price-based.

For working media, I have a 4TB SSD RAID ) drive that I use because of it's incredible speed. Most of my projects don't exceed 4TBs, except for the few features I do. When the project exceeds 4TB, for example, when working on a feature, I work in 4TB chunks. Still, my SSD RAID ) is not super safe because it has no redundancy in itself, and if it fails, the SSDs will not be recoverable. But that's ok for me because I already have the data backed up on my two spinning disk drives. Again, this is my compromise for having blazing speed, at a reasonable cost.

In short, my approach is fast SSD RAID 0 for working media, with multiple (2 or 3) cheap spinning disk drives for back up in different locations. This is working for me so far. But this is only based on my scale of work load and income. If cost isn't an issue, I'd have multiple 100 TB RAID 5 drives duplicated and at different locations. Alas, that is not necessary for me...haha.
 
abreu-canedo said:
Many ways to approach this. I'm assuming the OP is thinking about working media and storage media.

Storage media should be spinning disk drives, as opposed to SSDs. This is because in case of a failure, it's still possible to recover. If you can afford it, RAID is great because of redundancy. But you'd still want a replica of that RAID in a separate location in case of a fire, flood or other type of severe event. Happened to me once (pipe burst). Personally, I can't afford multiple duplicate RAIDS as back-ups in separate locations, so for storage, I just buy pairs of cheap 8TB spinning drives and duplicate them, storing them in separate locations. Don't need them to be RAID, because I think duplicating data into two separate drives is redundancy enough since if both of them fail, they are spinning disks so I'd still be likely to retrieve the data. My compromise is price-based.

For working media, I have a 4TB SSD RAID ) drive that I use because of it's incredible speed. Most of my projects don't exceed 4TBs, except for the few features I do. When the project exceeds 4TB, for example, when working on a feature, I work in 4TB chunks. Still, my SSD RAID ) is not super safe because it has no redundancy in itself, and if it fails, the SSDs will not be recoverable. But that's ok for me because I already have the data backed up on my two spinning disk drives. Again, this is my compromise for having blazing speed, at a reasonable cost.

In short, my approach is fast SSD RAID 0 for working media, with multiple (2 or 3) cheap spinning disk drives for back up in different locations. This is working for me so far. But this is only based on my scale of work load and income. If cost isn't an issue, I'd have multiple 100 TB RAID 5 drives duplicated and at different locations. Alas, that is not necessary for me...haha.
Click to expand...

What are you using for your ssd enclosure, or system?
 
You must log in or register to reply here.
Back
Top