Tapeless Workflow Archiving Solution

AwakenedFilms

AwakenedFilms

Veteran
I am currently working with an investor on a potential archive solution for tapeless workflows. As a part of my research, i have a few questions for P2, SxS, RedCode and RAMPak shooters, editors & producers:

1) What is your average typical file size for a shoot?

2) What is your average total file size per project (including media, sound and graphics)?

3) How much archive storage space would you as a typical professional need per year?

4) How much would you be willing invest per project or per year for a service that offers an offsite tapeless workflow archive solution (including verification, insurance and encryption of all files)?

5) What is the speed of your internet connection?


Thank you in advance for your response,

Jason Schuler
 
Last edited:
For a service? 0. If I can't do it myself right then and there and take my solution with me it loses it's value. Not to mention once I pay for my solution it's done. So if I get something like P2, which was a never a fan of simply based on cost, but let's use it for an example. Once I have some P2 cards all I have to do is use whatever I have that I upload the footage to. I can take this wherever I go and be in total control of it. This is just a personal opinion. People with bigger budgets may be thrilled to get something like this.

-Nate
 
Nathyn, I think the question is what do you do with the footage after you are done editing. Hard Drives are not a good solution, since they will fail and need to be booted and mounted periodically. LTO's are a better option, but are still out of the price range of many indies. Discs are not very efficient for large files. I keep most of my client-based files for at least a year, and my own stuff forever (if possible)

My guess is that we will be seeing a solid state option in the near future.

As for the questions, my file size per shoot varies widely. Sometimes I shoot in SD for a client, other times as much as 4K on the RED. The type of shoot also effects how much space I use. A PSA is going to require a lot less storage than a feature.

But on a yearly basis, I would say that in 2008 we had at least a Terabyte of footage that we shot and needed to archive, and that number will keep growing as we move more in to HD and tapeless workflows. This coming spring we are shooting a feature on the RED, and I expect that project alone to quadruple that number.
 
Thanks for the posts guys. Jim, you are right. I am talking about a very safe, potentially long term solution for the GB and TB of media a professional acquires, not a HDD on a shelf. Tapeless workflow users know that this is a serious problem, but especially when we get into high data rate formats like DNxHD, DVCPRO HD, AVC INTRA-100, RED...etc. What we gained in speed of capture and file transfer on the front end, we loose in archive options on the back end. Codecs get better, resolution gets better, NLEs get better, output options get better; but where are the technological leaps in archiving?

I can't give away too many details right now, but suffice to say what we are looking at could make a professional tapeless workflow user sleep much better at night.

Please keep the responses coming.

Jason
 
My company videotapes depositions for attorneys. We just got the HPX-170 and have been trying to figure out the tapeless archiving solution. We record large amounts of video (2-8 hours per deposition, sometimes 3 or 4 depositions a day) making archiving without tape a challenge as our data rate can be as high as 300 to 700 GB per day. Mini DV is only about 4 or 5 bucks a tape right now, so paying much more than that doesn't seem to make sense.
 
ZeDood said:
My company videotapes depositions for attorneys. We just got the HPX-170 and have been trying to figure out the tapeless archiving solution. We record large amounts of video (2-8 hours per deposition, sometimes 3 or 4 depositions a day) making archiving without tape a challenge as our data rate can be as high as 300 to 700 GB per day. Mini DV is only about 4 or 5 bucks a tape right now, so paying much more than that doesn't seem to make sense.
Click to expand...

That is a massive amount of data to store, about 2.5TB each week. USB hard drives would work, but you should have 2 or better yet 3 copies of each file, just in case.

Are you allowed to convert the files to a more compressed format?

IF you are allowed to have a SD back-up copy, why not use a video DVD. A 4.7GB DVD can record 2 hours of video with reasonable SD quality and can be pushed to 6 hours storage, assuming you allow for lower resolution and more compression.

No matter what system you chose, you should have a second or better yet a third copy, just in case something goes wrong. The second/third copy should be stored at another location.


Bob Diaz
 
Thanks for the feedback ZeDood. 300-700GB per day is a big number! Bob, you are right. One copy is not enough, especially if that one copy is not really a long term viable solution. DVDs and Mini-DV tapes may work for some professionals and their clients, but I believe there are many who want peace of mind when it comes to archiving full resolution data files.


J
 
Jim Brennan said:
Nathyn, I think the question is what do you do with the footage after you are done editing. Hard Drives are not a good solution, since they will fail and need to be booted and mounted periodically. LTO's are a better option, but are still out of the price range of many indies. Discs are not very efficient for large files. I keep most of my client-based files for at least a year, and my own stuff forever (if possible)
Click to expand...

Ah, okay, that makes sense.

AwakenedFilms said:
DVDs and Mini-DV tapes may work for some professionals and their clients, but I believe there are many who want peace of mind when it comes to archiving full resolution data files.
Click to expand...

I'm still using tape so I guess I never thought about this. I'm sure someone would go for this.

-Nate
 
This summer, I asked the very same question of the RED guys. They move around files a LOT larger than we are using with our HVX/HPX/EX1 type cameras. The answer was LTO. Yes, it's a tape solution. But it's also excellent, and hold a LOT of data.

Storing the data in a playable format for them is essentially impossible. No 4k tape backup tape solutions exist.

Frankly, I've moved to the DNxHD 36 mbps solution for backups. This requires about 15GB per hour and allows me to write my files onto BluRay. I can store nearly 3 hours of 1080p material on a dual layer. Or using single layers essentially I can place 1.5 hours of HD material on an $8 disk. This is essentially the same cost per hour I was seeing with high quality master DV tape. So our transition from SD to HD in terms of archive footage costs has not increased much if at all.

I suspect in 2 years, BluRay costs will mirror DV tape costs now, so it'll be a wash. But if I had a LOT of data to backup daily, I'd be looking at LTO or somethijng similar. I have a few dozen terabytes to be responsible for at work, and we do it all on hard drive with multiple layers of redundancy, but I don't know that I'd like to do that for video.
 
PerroneFord,
I find it interesting you 'archive' to DNxHD. What are format are you shooting: AVCHD, XDCAMHD, DVCPROHD, AVC-INTRA or REDCODE? How do you feel about the compression / transcoding process?

One of my concerns with data BluRays or LTO is that a single copy of the data would be on a shelf in my studio. It offers no redundancy, no protection for fire or theft. There is also the cost of time involved in burning a 25/50GB BluRay or writing to LTO / DLT. I think there is a better solution and that is what I am working on.


J
 
Yes, it is a lot of data to store. We record at least 2 independent copies on the spot. One to tape, and one to DVD. The DVD is very handy for our archive but for legal work we need another form of backup. That's where tape comes in.

We need to archive for at least 10 years so tape seems to be a reliable option for that (although 10 years from now who knows how many options will be available for playback.) The solution that I have come to for the HPX files is to compress them to an MPEG2 format which is more than adequate quality wise, and archive those. The question is, other than HDD, what is the best way to do so.

A tapeless backup is a very interesting option depending on the costs associated. The problem with LTO tape is that the initiall investment is quite high. What kind of "tapeless" system are you developing? It sounds like that for our purposes, the archive solution you are developing would probably not fit our needs. Bummer. Some day.
 
AwakenedFilms said:
PerroneFord,
I find it interesting you 'archive' to DNxHD. What are format are you shooting: AVCHD, XDCAMHD, DVCPROHD, AVC-INTRA or REDCODE? How do you feel about the compression / transcoding process?

One of my concerns with data BluRays or LTO is that a single copy of the data would be on a shelf in my studio. It offers no redundancy, no protection for fire or theft. There is also the cost of time involved in burning a 25/50GB BluRay or writing to LTO / DLT. I think there is a better solution and that is what I am working on.


J
Click to expand...

I am shooting in XDCam EX. So it's 35 Mbps maximum. The lowest form of DNxHD is a 36 Mbps codec. I've done difference tests on dozens of files. And honestly, the difference between XDCam EX and DNxHD 36 is essentially the same as I was seeing going out to lossless quicktime. Essentially nothing in luma, and just a shade of chroma. The file size is a couple of percent larger. And for that I get a standards based, multi-platform option that I can hand off to anyone (with the codecs to go with it).

As for my archival to BluRay, it's no different to me than my mini-DV, DV, or DvCam tapes sitting on my shelf right now. My work is not mission critical. I keep copies because I want to, not because I have to. So for me, a single copy is sufficient. My building is secure. DVD/BluRay is impervious to flooding and it's impervious to halon or other fire suppressents. Tape is not. It has higher resistance to heat than tape, and it's not susceptible to magnetic flux like tape. There are dozens of armed guards here, as well as 3 security doors and my locked door before you could worry about theft, so that's moot.

The cost of time writing to BluRay for me is immaterial. I run it overnight when everything else is idle.

For *ME* there is not a better solution available. This one is cost effective and efficient. For someone else this may not be true, and whatever solution you may be working on may be terrific. The data from my REAL job is protected via highly redundant and geographically dispersed HDD. There are 3-6 copies of everything we own spread over 800 miles and physically secured. I'm comfortable with that, but my video data doesn't need that.
 
Thanks for the posts. I am not interested in proving a point. I simply want to gather a cross-section of how users are protecting their data. Every business model is different and data protection needs vary from company to company.

PerroneFord, it sounds like for you the solution is just right and I am happy that you are happy. I am not very familiar with Avid or the proprietary DNxHD codec, but can see how it might be a decent fit for 35mbps XDCAM HD footage. Why do you transcode instead of archiving the XDCAMHD (is there something I am missing)? The bigger point for me is the nightly burning to BluRay and onsite storage of the discs. Do you ever shoot more than 50GB of data a day? Is there anyone else who burns to BluRay on a regular basis as their only form of backup?

ZeDood, currently your workflow either shoots in SD or down-converts to SD, correct? Do you ever shoot in HD on P2 or direct to disc capture? Depositions don't strike me as the kind of shoot that would benefit too much from HD (unless your company is trying to differentiate itself).

Unfortunately, I can't release too many details about the project at this point but would not rule out your needs from benefitting. There are a number of facets that we are exploring, not just high data rate archiving.


Jason
 
No, HD is useless for depositions at this point. Our company has recently started to do other types of legal video which can benefit from HD, but the amount of data coming from those projects would not be nearly as massive as from the depositions. I shoot my depositions directly to P2 cards as I said (we just got an HPX-170) which is what got me thinking about how I should archive the footage from this camera.
 
AwakenedFilms said:
PerroneFord, it sounds like for you the solution is just right and I am happy that you are happy. I am not very familiar with Avid or the proprietary DNxHD codec, but can see how it might be a decent fit for 35mbps XDCAM HD footage.
Click to expand...

http://en.wikipedia.org/wiki/DNxHD_codec

That gives the basics, but it's wrong in that there are 3 selectable bitrates. Actually, there are about a dozen with varying suitability for the material being encoded. I use the 36 Mbps for EX1 footabe, but there is 40, 60, 75, 115, 145, 175, and even 220, most in either 8 or 10bit. The codec is not proprietary. The source code is freely downloadble, and it is SMPTE VC3 standard. The codec is also free for Mac and PC users.


AwakenedFilms said:
Why do you transcode instead of archiving the XDCAMHD (is there something I am missing)?
Click to expand...

I transcode because the mpeg2 codec of the XDCamEX does not hold up in post. The DNxHD encoded files also are much easier to play on my timeline as the frames do not have to be constructed on the fly.


AwakenedFilms said:
The bigger point for me is the nightly burning to BluRay and onsite storage of the discs. Do you ever shoot more than 50GB of data a day?
Click to expand...


My projects are sporadic, and I don't burn nightly. If I was, I'd be looking at different solutions. My work is project based, and I do 10-20 projects per year. Some are :30 PSA type deals, some are multi-day conference where I will come home with 10-20 hours of footage. Just depends. This week I am transcoding a conference shoot I did in August. I shot it with the EX1 in HDV mode, and I suspect that it will be asked for this coming summer. In the meantime, I am putting it in an archive format so I can work with it later.
 
Looks like Jason (the original poster) has a new way of providing some facility or solution for CMF archiving. I'd be very interested to hear it (Jason please keep me posted) . Cloud based perhaps? :beer::beer:

Although Jason did not ask for peoples solution this thread has several useful & quite worthwhile opinions so I thought I would add mine. This forum, REDUSER, COW and several other provide many ideas on this as it is now a growing problem as physical video tape is playing a less of a role in the creative and distribution workflow.

I have had this archive dilema for some time and have now implemented a very good working archive system for MAC OSX using:
  • LTO4 Ultrium TAPE drives
  • MAC OSX hosted archiving applications
  • [optional] MAM application (osx based)

I have posted details of this at these sites for this that are interested.

For those interested, it is a very cost efective solution for your business and certainly worth a look and is quite fast - data rates exceed 110MB/sec uncompressed ofcourse for SAS I/F Ultrium LTO4 tape drive. NAturally always maintain two (2) instances of all essence and metadata! :)

HTH
w
 
Thanks for your input, ZeDood and PerroneFord.

Warwickt, I will keep you posted and appreciate your insight into cost-effective archiving solutions. I will be looking more closely at your posts for the finer details of your workflow.


Jason
 
I have read Blu-Ray is supposed to be storing larger amounts in the very near future, but it has always been my understanding that CDs/DVDs/Blu-Ray are not reliable for archive because they are 'burned' discs. Is this true if the discs are never used unless an emergency?
 
You must log in or register to reply here.
Back
Top