SlimRAW - compress DNG Files

Timo Flink said:
I am wondering how this compares to converting uncompressed RAW to ProRes 444 XQ? Have you tested how much of a difference is there between compressed RAW vs. ProRes 444 XQ as to the file size?
Click to expand...

This is actually easy to calculate and compare since ProRes is a constant bitrate codec.

Let's take forrestbro's files for the illustration. 312.5 GB compressed down to 120.8GB means CinemaDNG lossless compression down to 38.7% of the original size in this particular case. (Remember that lossless CinemaDNG ratios aren't fixed and depend on frame content).

Now suppose we have 24fps 2K raw footage from the FS700 taken on the 7Q. Each 2K raw frame is around 3.37MB. So in 1 second of uncomrpessed raw we have 3.37*24=80.88 MB, or a rate of 80.88 MB/s
24fps 2K ProRes 444 XQ has a bitrate ot 453 mbps. Let's convert to MB/s: 453/8 = 56.625 MB/s
24fps 2K ProRes 444 has a bitrate of 302 mbps. That is, 302/8 = 37.75 MB/s
24fps 2K Prores 422 HQ has a bitrate of 201 mbps. Or 201/8 = 25.125 MB/s

For the losslessly compressed raw (using forrestbro's compression level for illustration) we have an average rate 80.88 * 0.387 = 31.3 MB/s

So the losslessly compressed 2K CinemaDNG footage is not only much smaller than the corresponding ProRes 444 XQ file (a lossy codec), but also smaller than the ProRes 444 file (also a lossy codec). So with the CinemaDNG losslessly cmpressed raw you get the perfect original quality at a size smaller than an intermediate lossy codec.

If you wonder how is this possible, the reason is simple. Once the raw image is debayered the data size increases 3x times (because each pixel gets two additional channels to RGB). ProRes is then working on this debayered RGB image which is 3x the size of the raw data. CinemaDNG lossless compression, on the other hand, obviously doesn't need to handle the debayered image and compresses the smaller original raw data.
 
Last edited:
I am wondering if O7Q will have an update at some point, which enables this kind of compression in the device itself, without any need for SlimRAW (which I now need to get).
 
Timo Flink said:
I am wondering if O7Q will have an update at some point, which enables this kind of compression in the device itself, without any need for SlimRAW (which I now need to get).
Click to expand...

The thing which makes slim RAW mature over the 7Q is, that it takes atvantage of the processor
in your computer.
 
Finally I bought it too. It slimmed down a folder of 313GB / 90.525 files 4K and 2K both to 148GB. I forgot save the log but probably it wasn't mor than two hours on a maxed iMac Retina. No difference in Resolve. Great job, cpc!
 
Clermond said:
Finally I bought it too. It slimmed down a folder of 313GB / 90.525 files 4K and 2K both to 148GB. I forgot save the log but probably it wasn't mor than two hours on a maxed iMac Retina. No difference in Resolve. Great job, cpc!
Click to expand...

Congrats! How can you exactly tell that theres no difference in Resolve? It seems quite unlogical when reducing the
files over 50%...but maybe i don't understand the math^^
 
Clermond said:
Finally I bought it too. It slimmed down a folder of 313GB / 90.525 files 4K and 2K both to 148GB. I forgot save the log but probably it wasn't mor than two hours on a maxed iMac Retina. No difference in Resolve. Great job, cpc!
Click to expand...
Thanks, Clermond. :)

Fader8 said:
Congrats! How can you exactly tell that theres no difference in Resolve? It seems quite unlogical when reducing the
files over 50%...but maybe i don't understand the math^^
Click to expand...

Add one of the frames as the clip, then the other as an external matte. Add a layer mixer node. Then connect both the clip frame and the matte frame as inputs of the layer mixer. Choose Difference for the Composite Mode. If both files are the same image, the result is pure black.
Resolve-diff.png



Here is a simple example of extreme levels of lossless compression:
Suppose we have a 1000x1000 image, all black, RGB 8-bit. That's 3 000 000 bytes, all zeroes.
Now suppose we decide to compress it in the following way: each time we see successive bytes of the same value in the image, we will write out the count of these bytes, and their value (remember, they have the same value), instead of the actual bytes. In the case of our all black picture, this compresses down to just a couple of values: a count of 3000000 (stored in 4 bytes) and the value 0 (stored in 1 byte). Guess what? We just losslessly compressed 3MB of data with 5 bytes. No kidding. A compression ratio of 600 000:1.
The decoder will just read the count 3000000 and the value 0 and will insert 3 million zero bytes when decompressing, perfectly reconstructing the original 3MB 1000x1000 black image from just 5 bytes.

Now this is obviously a contrived example, but it shows how data can shrink down 600000 times without any loss of information. This is called Run-length encoding btw, and is used for lossless compression in some picture formats (especially for images with transparency). (And no, it has nothing to do with the CinemaDNG compression scheme, I just used it as an example because of its simplicity. :) )
 
Fader8 said:
Congrats! How can you exactly tell that theres no difference in Resolve?
Click to expand...

well.. I applied an extreme grade with various secondary color masks and I did not see a difference on my hardware calibrated Eizo preview monitor. If I don't see a difference that's what matters for me ;-)
 
Yep, I can confirm as well. It works beautifully! On my particular system I'm getting a slight chug when trying to play back 4K raw in realtime which I wasn't getting before, but 900 gigs of data was compressed to 360. Love it.
 
atarijedi said:
What happens if there is an audio stream in the MXF file?
Click to expand...

Nothing, because there is no support for MXF containers, only for sequences.
What camera outputting MXF CinemaDNG do you have in mind? I'm not aware of any.
 
Hi there,

Just a few words to voice out my take on SlimRAW.

Well, like everyone said, it's kind of magical ! I compressed a few TB of FS RAW footage, both FS700 and FS7 and SlimRAW does the job absolutely perfectly.

Around 30/40% of compression but I saw it going over 60%.

Opens flawlessly in Resolve and in After Effects' Adobe Camera RAW.

Fast processing and simple (but not simplest) GUI.

And to end this I MUST say that support, answering one of my dumb question (about batch processing) was lightning fast and informative, many thanks to Mihail !

Cheers,
Steph.
 
New version 1.2 is out.
It runs significantly faster and compresses even better than the previous version, so a highly recommended update for everyone using the previous version. Plus, there is a Maximum compression option for people that want the smallest file possible (still with perfect original quality). The default compression profiles are pretty good (ok, that's an understatement, they are excellent ;) ) but this option will sometimes save a little bit more space. And there is a Fastest option, for when processing speed is critical.
Also, slimRAW can now recompress CinemaDNG and DNG files that are already losslessly compressed (by other means) for further space gains.
Full release notes here: http://www.slimraw.com/relnotes.html
 
great new features! A little bit off topic but SlimRaw compresses now Pentax 645z DNG files about 20%. Down from 65MB to 53MB - almost the same size as a 5DM3 DNG. Many thanks, Mihail!
 
And version 1.3 now released.
slimRAW can now output compressed raw footage to two locations simultaneously, so CinemaDNG footage can be offloaded to main storage and backup storage at the same time, with on the fly lossless compression and checksum generation/verification.
As always, release notes here: http://www.slimraw.com/relnotes.html Recommended update for all users.
 
Version 1.4 now released. Adds 3:1 and 4:1 lossy CinemaDNG compression. Lossy CinemaDNG is currently compatible with Davinci Resolve.
It is a free update, so do upgrade and feedback is welcome.
 
cpc said:
Version 1.4 now released. Adds 3:1 and 4:1 lossy CinemaDNG compression. Lossy CinemaDNG is currently compatible with Davinci Resolve.
It is a free update, so do upgrade and feedback is welcome.
Click to expand...
Hi! Many thanks for your software! Could it be possible to add in a future upgrades any sound when the process is finished? I use SlimRAW with my Odyssey7Q and I use it on field during shooting. I have 3 ssd cards so when I'm shooting I have to go everytime to my laptop and check is the process is finished. The sound would be very helpful - it will signal that I can go and put another ssd. Anyway thank you for this beautiful software!
 
Oleksiy Oliks Kardash said:
Hi! Many thanks for your software! Could it be possible to add in a future upgrades any sound when the process is finished? I use SlimRAW with my Odyssey7Q and I use it on field during shooting. I have 3 ssd cards so when I'm shooting I have to go everytime to my laptop and check is the process is finished. The sound would be very helpful - it will signal that I can go and put another ssd. Anyway thank you for this beautiful software!
Click to expand...

Thanks Oleksiy. This is an interesting suggestion, I will see what can be done.
 
Hi Cpc, I just bought Slimraw and it does an incredible job! Could you give a broad elaboration on the new 4:1 compression? What sort of degredation are we talking here? For instance, is 4:1 raw still better than prores hq broadly speaking? My biggest barrier to even using raw in my general workflow is hd space of course. But 4:1 makes it a very attractive prospect!
 
You must log in or register to reply here.
Back
Top