Managing raw, file sizes and workflow

Essami

Essami

Well-known member
Hi,

I've been shooting with the FS700 for over a year now and have been quite happy with it. Multiple projects, art, music videos and currently we're shooting an indie feature film. So far I've been shooting internal AVCHD codec and the file sizes are so tiny compared to raw. I've shot with the Red one and that was a dream, even my old duo core laptop could handle the 4K raw files with no sweat and it equaled 1min / 1GB. Very manageable.

I'm considering getting the Odyssey 7Q to record our feature from April onwards. I won't be capable of handling the enormous 4K raw file sizes. If 1TB is 60 minutes of material this would eat our budget dry getting SSD's to record on and hard drives to back up up to. It's just not an option for us.

My intuition is that 4K prores would be the way to go for this project. One option would be to shoot 4K raw and then transcode this to 4K prores and - gulp - destroy the original 4K raw files... But that just don't feel good... :)

1) Is there a way to compress 4K raw after it has been recorded to something similar to Red's codecs?

2) How much space per 60 minutes is 4K prores 444 / 422?

Is anyone in the same sort of dilemma and how have you dealt with it?

Sami
 
If you look at it this way, really good 7200 rpm HDs are about $180.00 (there are cheaper ones). So if you have a stack of them and a SATA dock then you can just use those to store RAW footage pretty inexpensively really.

You may need to dedicate an intern and one or 2 laptops for file transfers (and don't forget backup). If you use 2 Thunderbolt docks it will be pretty fast, one for the CD SSD and one for the HD.

Or you could bring a RAID5 on location those can be up to 40TB or more and are fast and redundant. This is my plan anyway.
 
olof said:
If you look at it this way, really good 7200 rpm HDs are about $180.00 (there are cheaper ones). So if you have a stack of them and a SATA dock then you can just use those to store RAW footage pretty inexpensively really.

You may need to dedicate an intern and one or 2 laptops for file transfers (and don't forget backup). If you use 2 Thunderbolt docks it will be pretty fast, one for the CD SSD and one for the HD.

Or you could bring a RAID5 on location those can be up to 40TB or more and are fast and redundant. This is my plan anyway.
Click to expand...

Lets say we shoot 100 hours of footage. 1 hour is 1TB. 100TB and back it up another 100TB. 200TB put on 4TB USB3 drives at even the cheapest solutions (in Finland around 130€ per 4TB drive) would cost around 6,500 euros (9000 USD). That's way over our budget. In addition you need to calculate costs for Odyssey SSD's and hard drives for proxies as obviously you can't edit from 50 separate hard drives.

4K raw in it's current form is just not a viable solution for us budget wise.

Sami
 
Hey Essami,

Here's what I've been doing. I have a 4 TB G-Raid. I just copy over the RAW clips then I transcode them using Resolve into Pro-Res. I work with the Pro-res clips for a few days to make sure they are fine, then I just delete the RAW clips. Kind of sucks to delete the RAW clips, but I have to because otherwise I would be spending a lot on hard drives. It's worked well so far.
 
gene_can_sing said:
Hey Essami,

Here's what I've been doing. I have a 4 TB G-Raid. I just copy over the RAW clips then I transcode them using Resolve into Pro-Res. I work with the Pro-res clips for a few days to make sure they are fine, then I just delete the RAW clips. Kind of sucks to delete the RAW clips, but I have to because otherwise I would be spending a lot on hard drives. It's worked well so far.
Click to expand...

Do you encode as 422 HQ? That would amount to 1Gb per minute in my experience. 400GB versus 400TB is a massive difference.

I really do think raw to prores might be the way to go for us as well. If there are any outrageous under/overexposure mistakes then you could just save those raw files but if everything is ok and you know what you're looking for image wise... I don't see a problem for us.
 
Essami said:
Lets say we shoot 100 hours of footage. 1 hour is 1TB. 100TB and back it up another 100TB. 200TB put on 4TB USB3 drives at even the cheapest solutions (in Finland around 130€ per 4TB drive) would cost around 6,500 euros (9000 USD). That's way over our budget. In addition you need to calculate costs for Odyssey SSD's and hard drives for proxies as obviously you can't edit from 50 separate hard drives.

4K raw in it's current form is just not a viable solution for us budget wise.

Sami
Click to expand...

So how much would it cost to shoot in 35mm film...

OK every project has a different budget. $10K to $20K media cost for a serious production is not much really considering the quality you have. If you do jobs in 4KRAW you have to charge for it. It can't and should not be priced the same as HD or even a 4K compressed workflow. Good clients will understand this and if you are shooting indie film, just max out a couple more cards, or get creative with your financing...

Not every project needs 4KRAW or even 4K.

I was just pointing out that it is not impossible expensive, especially if you compare to traditional ways to get this quality.

ProResHQ is very good and would be a fine way to store large amounts of footage.

All this will be moot in a few weeks because we will have 4K compressed by NAB in the Odyssey.

There may still be times when you want 4KRAW and then the HD storage is the only real way to store those files, you probably want all the files on RAID 5 or on multiple drives in multiple locations to be real safe.

100 hrs on 4TB drives is 25 drives, even at $200 per drive that is only $5K, so back up that is $10K not bad really. And if you buy 50 drives you will get a much better price.

I still remember my first 20MB Lacie HD back in1990 it cost almost $5,000.00 in 1990 $$'s. Things are cheap now.
 
Last edited:
Yes, 10-bit 4:2:2 HQ. I also tried 4:4:4 12-bit Pro-Res. I did a crazy, extreme grade to it and between the two, I barely, barely noticed any difference even on a very extreme grade, so probably not worth the extra space.
 
we do a fair amount of MoW's & indie features, the workflow is the same be it shot on Epic, Alexa, F55... or now a FS700

camera orignal cards to post
cards to raided server
server to editorial for transcodeing / offline
server to LTO for archive
delete camera orignals from the server once confirmed on LTO
High rez conform from LTO once picture locked

a 400gig LTO tape costs about $30... lasts for forever
 
Last edited:
There's more to consider besides just file size, vis-a-vis hard drive space. System thruput, in order to get RT playback, or just plain file copy time are considerations. Data thruput to play back a 4K file is about 275MBps. No conventional hard drive is gonna do this. Looks like RAID is the only solution. And consider file copy time from the SSD to an onboard drive....If a terabyte of HD is full, at 200 MBps, we're talking an hour to an hour and a half...and that's for a modern SSD. Older 6Gbps disk platters transfer data around 100-110MBps......

so, I may have mixed my bits and Bytes.....but....just copy times are not trivial.

I love that I can record 4K, don't get me wrong. Just sayin'....it takes a pretty nice budget to be able to process in 4K.
 
Last edited:
cuervo said:
There's more to consider besides just file size, vis-a-vis hard drive space. System thruput, in order to get RT playback, or just plain file copy time are considerations. Data thruput to play back a 4K file is about 275MBps. No conventional hard drive is gonna do this. Looks like RAID is the only solution. And consider file copy time from the SSD to an onboard drive....If a terabyte of HD is full, at 200 MBps, we're talking an hour to an hour and a half...and that's for a modern SSD. Older 6Gbps disk platters transfer data around 100-110MBps......

so, I may have mixed my bits and Bytes.....but....just copy times are not trivial.

I love that I can record 4K, don't get me wrong. Just sayin'....it takes a pretty nice budget to be able to process in 4K.
Click to expand...

That is why I stated you may need to dedicate an intern and a couple laptops to file management in the field, when shooting production 4KRAW. And either a bunch of drives or a couple RAID 5 boxes.

I am sure I will shoot some 4KRaw in the next year but I will charge top dollar for it.

Editing is a whole different ballgame. Personally I am getting a new Trashcan for this as well as a new TB2 studio RAID. I already have 5 RAID5's (TB, SAS and SCSI) in the studio, but no TB2 yet. I also have over a 120 HDs for storage and hundreds of linear footage of tape shelves, actually the HDs and tapes are in file cabinets, in climate controlled rooms. A lot of my clients also store drives on their premises. I never guarantee safe storage an I have production insurance, and it has come in handy, unfortunately s**t happens...
 
gene_can_sing said:
Yes, 10-bit 4:2:2 HQ. I also tried 4:4:4 12-bit Pro-Res. I did a crazy, extreme grade to it and between the two, I barely, barely noticed any difference even on a very extreme grade, so probably not worth the extra space.
Click to expand...

Sounds perfect!
 
Well one consideration is that compressing on set may be an option, particularly since media read speed is not so much of an issue but bringing high speed mass storage is. It might actually be faster in some cases to encode directly to compressed cineform RAW or some other format on set and throw out the massive 4K DNGs. What I'm suggesting would be recording RAW normally and instead of copying the footage, transcode directly to cineform on set. I think I have heard of people doing this for Blackmagic Cameras and others, though I'm not sure what the CPU load is and whether it may be a bottleneck. If the CPU is not the bottleneck, it could be a good option since the CD SSDs should allow read speeds of up to 1.5x realtime, so you could "download" your footage to cineform raw at that speed without any fast or huge RAIDs on set.

I think the uses for recording an storing uncompressed 4K raw on set are probably very limited in that most productions don't need uncompressed raw especially at 4K and most productions won't be able to handle it. Compressed raw from RED at 30-100MBps is hard enough to deal with on set as it is, so the burden of 4K uncompressed raw at 318MBps is pretty unmanageable and unprecedented except maybe for a handful of projects shot on a 4K Dalsa camera or something.
 
I think the bottom line here is 4KRAW is great but of pretty limited use the 4K compressed will be what a lot of us will find very useful. But I must admit that 90% of my work next year will probably be HD mixed O7Q recorded and FS700, C100 and C300 internally recorded, as all as other cams like the EX3 and various action cams.

It is great that we now or shortly will have these great tools to expand our offerings. But it is all client driven and most of my clients will not need 4Kraw at this point, everyone is different, but this is my projection for the rest of this year.
 
Yeah I still have a hard time thinking of any situation I'd want to record 4k raw instead of 4k prores. Even 4K prores sounds a bit daunting at ~100MBps.
 
When we used the Sony FS700 AXS-5R for a short film, we dealt with a nightmare of headaches for post. When we shot it, the R5 was barely out so the support for it was very minimal. We made dailies / proxies through Da Vinci Resolve and took the Sony RAW Viewer and made massive DPX files. We shot about 1 TB for a 10 minute short film (45-60 minutes of footage atleast) and then made about 2.5 TB of DPX files. After the edit was finished, we graded the DPX files at snails pace (I was the colorist and I wanted to die) and reconnected the DPX files. Export our Master and various versions for output. I can't imagine tackling a full feature with the same workflow. I hope the 4K DNG files are better implemented in Adobe's workflow this year. I need to also just buy Resolve outright so I can unlock the full possibilities of the software, then no worries about 4K RAW.

We will be releasing the short film online in a secret link. I will post it on here when I can, because most of that was shot on 4K and I think it is damn impressive for what our cameras can do. We shot it on the Schneider Xenar III's with Sony R5 recorder. Gorgeous footage.
 
Chris Langer said:
When we used the Sony FS700 AXS-5R for a short film, we dealt with a nightmare of headaches for post. When we shot it, the R5 was barely out so the support for it was very minimal. We made dailies / proxies through Da Vinci Resolve and took the Sony RAW Viewer and made massive DPX files. We shot about 1 TB for a 10 minute short film (45-60 minutes of footage atleast) and then made about 2.5 TB of DPX files. After the edit was finished, we graded the DPX files at snails pace (I was the colorist and I wanted to die) and reconnected the DPX files. Export our Master and various versions for output. I can't imagine tackling a full feature with the same workflow. I hope the 4K DNG files are better implemented in Adobe's workflow this year. I need to also just buy Resolve outright so I can unlock the full possibilities of the software, then no worries about 4K RAW.

We will be releasing the short film online in a secret link. I will post it on here when I can, because most of that was shot on 4K and I think it is damn impressive for what our cameras can do. We shot it on the Schneider Xenar III's with Sony R5 recorder. Gorgeous footage.
Click to expand...

Don't forget R5 records 16b with 3.5:1 compression so the 7Q's FS700 12b 4K uncompressed raw footage will be about 2.5x the size...
 
You must log in or register to reply here.
Back
Top