SlimRAW - compress DNG Files

Hey Kyron, thanks for supporting slimRAW.

The amount of information culled by the lossy compression modes depends a lot on the amount of information in the original image. Note that information is not the same as data: the purpose of compression is to more effectively represent information into less data. For example, for the same resolution, the image of the Blackmagic cameras will have higher entropy than the image of the FS cameras. So more information is discarded when doing lossy compression on BM raw images compared to FS raw images. On the other hand, BM cameras exhibit significantly more noise and aliasing which helps mask image changes caused by lossy compression.

One way to think about this is the following: the better a raw image compresses when using lossless compression, the less information needs to be discarded when doing lossy compression on the same image. To illustrate the case above, BM cameras produce images that usually compress losslessly to around 1.5:1-1.6:1. Sony FS cameras produce raw images which usually compress losslessly to better than 2:1 (sometimes up to 2.8:1). Hence, less information is discarded when doing lossy compression on Sony FS images.

Since the lossy compression in slimRAW is fairly light and perceptually based (what's commonly -- and somewhat arbitrarily -- touted by marketing as "visually lossless"), ideally degradation will be negligible. I can say I am pleased with how it turned out. You should run tests though.

Compared to ProRes422HQ, 4:1 will produce slightly smaller files (10-12% smaller). The main advantage of using raw compression vs ProRes is that you can still do a full raw workflow (control debayer quality, adjust WB, etc). Also, ProRes422HQ is 10-bit and CinemaDNG compression is 12-bit. And ProRes operates on full RGB samples (vs Bayer samples in raw compression), which puts it in a bit of a disadvantage sizewise. On the other hand, lossy CinemaDNG is only supported by Davinci Resolve at the moment.
 
Last edited:
Thanks so much for the detailed reply cpc, I really appreciate it. I'm really impressed with slimraw and will definitely do some damaging tests to the 4:1 footage compared to my usual Prores workflow and see how things go :)
I don't see the 7Q being able to do this anytime soon so thank goodness we have an extremely well priced and well supported piece of software here. Hats off to you cpc, awesome work my friend!
 
Slim RAW compress my DNGs from 182GB to 65 GB is that even possible withough losing quality? (Setting:lossless!)
 
Slim RAW compress my DNGs from 182GB to 65 GB is that even possible withough losing quality? (Setting:lossless!)

Yeah, it is perfectly possible with the FS700. And with the FS7 you can get even more compression.
 
I have done some tests with the 4:1 compression and it is quite remarkable. To get 4k raw that takes up a little less space than 4k Prores HQ is pretty amazing.

The next challenge is making these 4x256gb drives I have work on set at 12mins of footage per drive. On a middling computer doing a direct 4:1 compression directly off the ssd it was taking 22mins per drive to compress and dump a full ssd. So roughly double the length of the footage.
With an on the ball DIT, the 4x drive rotation will just be enough to keep shooting indefinitely although that leaves no time to back up the original footage at all which is somewhat scary.

In any case, this has opened up options to me that were never possible before Slimraw (besides spending a small fortune on more SSD's!)
 
Kyron, what computer and connection are you using to dump your ssds? Those speeds sound really impressive to dump the footage and compress it at the same time. So are you not keeping the uncompressed cinema DNGs at all?
 
Hey EAumen,

My test involved plugging the SDD into a USB3 dock and slimrawing it directly from the dock. Copying the files locally first and then slimrawing them took a fair bit longer which is why I was lamenting about not having the time to back up the original DNG's on set. (Which is why I haven't tried this on a real project yet)

If i bought a couple more 256gb SSD's I'm sure I could though. As long as someone was a dedicated DIT that I could trust with all my precious data!

I run a windows machine with an i7 4930k 3.4ghz and 32gb ram.
 
I run a windows machine with an i7 4930k 3.4ghz and 32gb ram.

This CPU should be enough for significantly faster offload than 22 min/256GB. The storage system is the likely bottleneck. How long does a plain copy offload take?
 
That works out to about 198MB/sec. That is a pretty good transfer rate for USB3. If you are transferring to a 7200 Rpm hard drive, that is about the max you can expect without a RAID 0 array or a fast SSD for the destination.
 
Last edited:
Yeah, 200 MB/s is pretty good. Some USB3 controllers go to a bit higher than 400MB/s (with ~500 MB/s being the theoretical maximum for USB3). And even at 400 MB/s with 4:1 a clean defragmented 7200rpm HDD should be able to handle the output data rate (~100MB/s).
The hexcore IvyBridge CPU itself should have no problem sustaining a throughput higher than 500MB/s in slimRAW.
 
Could it be possible to add in a future upgrades any sound when the process is finished? I use SlimRAW with my Odyssey7Q and I use it on field during shooting. I have 3 ssd cards so when I'm shooting I have to go everytime to my laptop and check is the process is finished. The sound would be very helpful - it will signal that I can go and put another ssd. Anyway thank you for this beautiful software!

slimRAW v1.5 is now out, and it has optional sound notification on finished processing. :)
Full list of new stuff here: http://www.slimraw.com/relnotes.html
 
Version 1.6 now released with 4 new compression modes: 5:1 lossy, two variable bit rate lossy modes, and a 10-bit log encoded lossless mode. 10-bit log lossless is widely compatible with video production software (including anything Adobe) and will produce a bit smaller files than straight Lossless. The log transform itself is transparent and will be reversed by the raw processing software on import.
More about the various types of compression here: http://www.slimraw.com/article-cdngmodes.html
 
Hey Mihail!

Thanks for the update, one little suggestion...

why don't you start a collaboration with Kinefinity to implement KRW2.0 files to CinemaDNG conversion inside SlimRAW?

It would be beneficial for both of you guys! ;)

Or maybe you could outsource to them your conversion engine or maybe developing a Kinefinity-branded app for footage ingestion and conversion?

What do you think??
 
Hey Mihail!

Thanks for the update, one little suggestion...

why don't you start a collaboration with Kinefinity to implement KRW2.0 files to CinemaDNG conversion inside SlimRAW?

It would be beneficial for both of you guys! ;)

Or maybe you could outsource to them your conversion engine or maybe developing a Kinefinity-branded app for footage ingestion and conversion?

What do you think??

Anything is possible and this is a thought that has certainly crossed my mind before. The main problem is that (my) time is a limited resource. DNG compression is a tiny niche, and krw-to-dng recompression would be a tiny fraction of that, so even tinier. There is a high opportunity cost involved in such a decision.
But you never know. :)
 
Version 1.7 is now out. New feature is Bayer-to-Bayer downscale to half resolution in both dimensions. That is, 2K from 4K, 1080p from 2160p, etc. This is mainly inteded for raw proxies from high res CinemaDNG (coupled with 5:1 compression, it will result in 20:1 size reduction), greatly reducing storage throughput, CPU and GPU resources needed for editing and coloring, while preserving the raw workflow. And, of course, if storage space is critical, the 2K downscaled footage can be used for delivery. The image is interpolated from the full res mosaic so there are less artifacts than on-sensor binning and lineskipping.

Here is a quick write-up on using DNG proxies in Resolve and Premiere: http://www.slimraw.com/article-proxies.html
Resolve is particularly well suited for the task.
 
Has anyone tested the 4k to 2k yet? Is it just usable for Proxies or can you use the downscaled material high end for the full post production. This would be amazing, since your PC needs less power and the 2k would need less space. But you could record 4k to get the benefit of a full sensor rea out.
 
slimRAW 1.8 now released with 7:1 CinemaDNG compression. This mainly targets 4K+ cameras and oversampled deliveries, and it is also handy for CinemaDNG proxies.
Also, a small performance boost for all lossy compression, so update is recommended.
 
what do you guys think is the sweet spot for 4k RAW Cinema DNG regarding compression rate?
I found 4:1 to be a pretty good deal.
 
In general, 4:1 is a safe default choice for 4K raw from the FS-series, and you can usually just bump to 5:1 if you know you are going to deliver 2K/HD.
 
Back
Top