C500: CRL Data Management

D

digitalinnovations

Well-known member
Now with all of the various cameras shooting CRL at HUUUUUUGE file sizes I am curious how everyone is approaching their data management? I’m assuming some of you just hand off the footage and it’s not your problem after that but if some of you are like me (shoot, edit, deliver, etc.) how are you choosing to store that footage? Are you converting it over to ProRes (or something else) - if so what flavor? Or are you just biting the “storage bullet” and keeping all the original files?
 
I keep all of the original files anywhere between 6-12 months after a project is complete.

Then when it's time for them to go, I'll have a back-up of the following:

- All of the footage on a timeline as is; untrimmed & uncorrected & ungraded and exported to ProRes 422 HQ.
- All of the footage on a timeline that was trimmed, corrected and graded (if applicable).
- Final product timeline exported only with dialogue, no music.
- Final product timeline exported only with music, no dialogue.
- Final product timeline exported only with sound effects (if applicable).

Graded and ungraded versions of those three above as well.

That should cover me for the random "Hey, can you fix/add this?" months later.

And if the project is very short or very, very important, I will always keep a consolidated FCP X library for it which has all of the original footage. (Still wish you could consolidate trimmed footage.)
 
I like that setup; especially the various exports because you are exactly right about covering most bases between those renders. And yes, I wish FCPX would do that as well - it would make the process much easier and save a bunch of hard drive space.

I wish everyone thought like you because I’ve been handed off projects before that had the final audio mix and then I’m expected to work miracles editing that with burnt in music.

Up to this point, I have kept all the original files, timelines, final edits with graded footage, etc. pretty much indefinitely...just moving them off my media raid onto external drives, as you said, after about a year or so / depending on the likelihood that I been doing additional edits / projects with that particular client. But now at these resolutions and bandwidths I don’t think that is going to be practical...at least not for the majority of the projects.
 
Thank you.

I was actually always one of the rare birds who never used RAIDs to edit. All of my projects in my life have been under 4TB and at most I was using two 2TB SSD drives from a very, very early time in the world (around 2010/2011) and paying astronomical amounts for them (but I still use them today so I guess that's good).

Speed is blazing fast even with 8K ProRes from my recent R5.

I edit more than I shoot, and I will probably edit like that forever. And then I'll put the masters I mentioned above on cheap 8TB hard drives that can be had for $100-$150 on Amazon.

I'll keep one SSD on the side with the safety PR HQ (or even LT) back-ups of just the trimmed/graded footage because this is probably the most important because (1) 99% of the final product will usually come from this file if I ever had to recreate it (2) the untrimmed version of all of the RAW footage could be 2-4x the size and what are the chances I'll ever need something from that one that's not in the cleaned-up footage.

One thing I forgot to mention is I will also export all of the audio from the raw footage just as one audio file.
 
P.S. A common practice in the industry by many is to shoot interviews at a higher compression ratio (if you can...and usually you can because no one cares about having the absolute best talking heads quality for the limited air time), and shooting b-roll in the best, but that's unfortunately one of the weaknesses of CRL and being stuck at one data rate versus ProRes, REDCODE or BRAW where you could have 5-10+ different options.
 
Yes! I wish there were some compression options for sure. I did essentially what you are describing with ProRes and loved the flexibility; the more control you have with setups / lighting / DR you can get away with more compression / lower data rates. Now I would break that “rule” sometimes with interviews but the point is...it’s nice to have options. It’s nice also to “de-tune” the camera a bit either because of budget constraints or longer form presentations and such. Granted the 422 10bit gives an option and then recording only proxies covers the lowest end of the spectrum plus whatever the new Long GOP is (haven’t dug into that as I just received my first C500 today). I think what’s missing is something in that upper range but below the current CRL data rate or like a 444 XFAVC but I think there was a thread on here where it was determined the data rate didn’t save much at all from CRL and then you lose the RAW flexibility in post....always compromises.

If I could afford it, I’d just love to shoot everything in RAW for the best quality and flexibility but now I’m just being a brat.
 
NorBro said:
I keep all of the original files anywhere between 6-12 months after a project is complete.

Then when it's time for them to go, I'll have a back-up of the following:

- All of the footage on a timeline as is; untrimmed & uncorrected & ungraded and exported to ProRes 422 HQ.
- All of the footage on a timeline that was trimmed, corrected and graded (if applicable).
- Final product timeline exported only with dialogue, no music.
- Final product timeline exported only with music, no dialogue.
- Final product timeline exported only with sound effects (if applicable).

Graded and ungraded versions of those three above as well.

That should cover me for the random "Hey, can you fix/add this?" months later.

And if the project is very short or very, very important, I will always keep a consolidated FCP X library for it which has all of the original footage. (Still wish you could consolidate trimmed footage.)
Click to expand...

I like those combinations, you're covered for everything.

Since trying to shoot CRL for almost everything now, I've become much more aggressive in having the data management conversation before the project begins.

I think I'm in the minority for data storage - in general I have slightly lower post rates than fulltime editors, so I feel better about providing fewer services in that area.

I know there's money to be made for proper file storage, but I've avoided going down this path and make sure the people I'm working with have (backed up) the physical hard drives rather than holding onto everything myself.
 
Last edited:
NorBro said:
P.S. A common practice in the industry by many is to shoot interviews at a higher compression ratio (if you can...and usually you can because no one cares about having the absolute best talking heads quality for the limited air time), and shooting b-roll in the best, but that's unfortunately one of the weaknesses of CRL and being stuck at one data rate versus ProRes, REDCODE or BRAW where you could have 5-10+ different options.
Click to expand...

This is a good point. You could easily shoot your interviews in AVC and your b-roll in raw. The AVC will be more than good enough for a talking head.
 
For sure...for the C300/500 III/II, that's great.

C200's go big or go home (even though some would use the lower-quality 8-bit option and be okay).
 
NorBro said:
For sure...for the C300/500 III/II, that's great.

C200's go big or go home (even though some would use the lower-quality 8-bit option and be okay).
Click to expand...

I did that all the time. In fact, 99% of the time that I shot CRM on my C200s the past 2.5 years was for my personal enjoyment/satisfaction/ego. None of my clients ever needed it and only a few of them were curious enough to want to try it.

If I was shooting CRM it was because I thought that I would want to showcase the project or use some of it in my reel.

After shooting CRM on the C500MKII, I don’t think I would ever bother with shooting it unless I needed 6K for a certain shot or for slowmo shots.
 
NorBro said:
I keep all of the original files anywhere between 6-12 months after a project is complete.

Then when it's time for them to go, I'll have a back-up of the following:

- All of the footage on a timeline as is; untrimmed & uncorrected & ungraded and exported to ProRes 422 HQ.
- All of the footage on a timeline that was trimmed, corrected and graded (if applicable).
- Final product timeline exported only with dialogue, no music.
- Final product timeline exported only with music, no dialogue.
- Final product timeline exported only with sound effects (if applicable).

Graded and ungraded versions of those three above as well.

That should cover me for the random "Hey, can you fix/add this?" months later.

And if the project is very short or very, very important, I will always keep a consolidated FCP X library for it which has all of the original footage. (Still wish you could consolidate trimmed footage.)
Click to expand...

This seems a lot of work which probably isn't paid extra?
How often did it happen that you needed these files?

I keep a
- FCPX consolidated library with all footage
- XML of the final edit
- ProRes Master in full resolution no matter what the hand over was.
- audio track of dialogue (if applicable)

I delete all generated files, optimized, proxies and chaches

Having the dialogue audio track separate is the most important for me.
Mostly I have that anyway as I apply a "Mike Russell VO effect rack".
If a client want's a different edit you have to re-edit the music and SFX anyway.
I don't export ungraded versions. If the client needs a different grade I will be happy
to do it from the FCPX project or XML file and write an invoice for that.
 
Last edited:
Clermond said:
This seems a lot of work which probably isn't paid extra?
How often did it happen that you needed these files?

I keep a
- FCPX consolidated library with all footage
- XML of the final edit
- ProRes Master in full resolution no matter what the hand over was.
- audio track of dialogue (if applicable)

I delete all generated files, optimized, proxies and chaches
Click to expand...

Clermond, your workflow sounds pretty much in line with what I have done for years and years too and it’s served me and my clients pretty well. I’m just wondering if there is a better / different / more efficient workflow once we start talking about the higher data-rates / storage needs of RAW.

That said, the methods / exports that NorBro mentioned seem to make a lot of sense and really wouldn’t take much time in FCPX; just duplicate/snapshot the timelines, mute/delete the various audio tracks and kick them out when you render your master. Ever since FCPX added the ability to batch export, the whole process is just a few mouse clicks.

Since a lot of the stuff I shoot involves some form of interview I could see the version with just dialogue being pretty useful as that is what clients will probably revisit most often - to remove personnel that have left, etc. But having those exports would CYA in case your project file goes pear-shaped OR you move on to a different editing software or whatever...I could see that saving more time down the line than it takes to just render it out along with the master initially.

Clermond said:
...I will be happy to do it from the FCPX project or XML file and write an invoice for that.
Click to expand...

I think all of us would be more than happy to do that regardless off our workflow; whether that’s doing it now or later.

These types of topics are not sexy (like cameras, lenses, etc.) but I am loving hearing how everyone approaches these types of things differently because managing storage expenses as well as expectations from the client is just part of the business.
 
Isn't ProRes 422 basically the same file size as the CRM? I just keep the CRM backed up, I never shoot interviews in anything but XFAVC. If you keep an eye on white balance an exposure, it looks great.
 
npm said:
Isn't ProRes 422 basically the same file size as the CRM? I just keep the CRM backed up, I never shoot interviews in anything but XFAVC. If you keep an eye on white balance an exposure, it looks great.
Click to expand...

No, Prores4444 is pretty close though. At matching resolutions Prores422 HQ is about 20% smaller, and Prores422 is close to half the size.
 
Clermond said:
This seems a lot of work which probably isn't paid extra?
How often did it happen that you needed these files?

I keep a
- FCPX consolidated library with all footage
- XML of the final edit
- ProRes Master in full resolution no matter what the hand over was.
- audio track of dialogue (if applicable)

I delete all generated files, optimized, proxies and chaches

Having the dialogue audio track separate is the most important for me.
Mostly I have that anyway as I apply a "Mike Russell VO effect rack".
If a client want's a different edit you have to re-edit the music and SFX anyway.
I don't export ungraded versions. If the client needs a different grade I will be happy
to do it from the FCPX project or XML file and write an invoice for that.
Click to expand...

The only extra work is editing the project, which is of course paid. I edit 99% of what I shoot.

After that, all of those files above can be prepared in 5-10 minutes by simply disabling the aforementioned on the timeline (you click on and off).

If I didn't edit, there would be no dialogue, no music, no sound effects - and I would just dump everything (all of the raw footage) into a timeline (#1) and export. This one would take less than 5 minutes.

And I'd rarely ever have to use the files (once or twice a year), but it's effortless to do just in case.
 
One other tip...

So when you initially import all of your footage into FCP X the library can obviously be massive. Eventually you may only use 30-40% of the footage in your library for the final edit, but the library size of course doesn't change, so with some projects when the final timeline is complete I copy that timeline into a new library, which only brings with it what's used.

So a 1TB library may then be consolidated down to a 250GB library. And then I'll still have all of the ProRes back-ups.

And eventually when it's been so long, I may also just export everything to H.264 straight from FCPX. If the footage is extremely high-quality no one would ever notice a difference. And the H.264 saves tremendous amounts of space (1000s of GBs over time).
 
NorBro said:
One other tip...

So when you initially import all of your footage into FCP X the library can obviously be massive. Eventually you may only use 30-40% of the footage in your library for the final edit, but the library size of course doesn't change, so with some projects when the final timeline is complete I copy that timeline into a new library, which only brings with it what's used.

So a 1TB library may then be consolidated down to a 250GB library. And then I'll still have all of the ProRes back-ups.

And eventually when it's been so long, I may also just export everything to H.264 straight from FCPX. If the footage is extremely high-quality no one would ever notice a difference. And the H.264 saves tremendous amounts of space (1000s of GBs over time).
Click to expand...

Actually I use b-roll footage from old projects for other edits. That's why I keep also the unused material.
 
I meant in general for anyone.

And that's a particular case; most couldn't do that unless you're working for the same company on handfuls of projects.
 
You must log in or register to reply here.
Back
Top