SlimRAW - compress DNG Files

5MB BMCC 2.5K raw frame compressed down to 3.2MB

I wonder how this will work with BM's new 3:1RAW compression...If the two are the same method, then the usefulness is moot. (beyond converting existing footage that it)
 
Is this for storage purposes only? Does NLE's recognize the compressed DNG sequences like normal ones?
 
Essami said:
Very interesting... I wonder what the file size reductions will be in real life...
Click to expand...

Here is a record of some of my test footage, sorted from highest to lowest compression:

FS700 mix of 4K and 2K on the 7Q: compression ratio 2.67:1, or size down to 37.4%
Indiecam GS2k/POV: 2.63:1, down to 38%
Kinemini4k: 2.4:1, down to 41.6%
Bolex: 2.27:1, down to 44.1%
Canon DSLR ML raw (assorted cameras, but mostly 5dm3): 1.96:1, down to 51%
Kinemax6k: 1.88:1, down to 53.3%
Ikonoskop: 1.87:1, down to 53.5%
BMCC: 1.53:1, down to 65.6%

Take these with a grain of salt. While these numbers are from deliberately varied input (selected single frames from clips, so that numbers aren't skewed by longer clips), they are by no means entirely representative.
You'll note that BMCC lags behind. That's because it lacks an OLPF, resulting in generally sharper and/or aliased footage, which compresses less compared to the other cameras.
In any case, it is best to just run some frames from your footage to see what compression levels you can expect for your own video. There is a trial which does that.


seanmc said:
I wonder how this will work with BM's new 3:1RAW compression...If the two are the same method, then the usefulness is moot. (beyond converting existing footage that it)
Click to expand...
According to BM, they'll be keeping their lossy compression proprietary (for now, at least), so that's that. We probably won't see it used on other cameras or software soon.

Of course, converting existing footage is one of the goals. But, imho, the software gets really cool when it is used to offload footage from camera/recorder media to main storage while losslessly compressing it on the fly (yes, it is fast). Obviously, on-the-fly compression only concerns cameras and recorders which record uncompressed CinemaDNG like Bolex, fs700/fs7 + 7q, Ikonoskop, etc.


Essami said:
Is this for storage purposes only? Does NLE's recognize the compressed DNG sequences like normal ones?
Click to expand...

From the site:
software supporting losslessly compressed CinemaDNG includes Blackmagic Design DaVinci Resolve, Assimilate Scratch, The Foundry NUKE, Adobe Premiere CC, Adobe SpeedGrade CC, Adobe After Effects, Adobe Photoshop and Adobe Lightroom.
Click to expand...


edit:
forgot disclaimer: I made it, fwiw
 
Awesome, great details CPC, thanks!

So, are there any downsides at all to this? 'Losslessly Compressed" seems to sound like there is no downside at all. (Maybe just more work for CPU/GPU to decode and uncompress each frame while live playback/editing? Kinda like h264 has to do, just on a RAW scale)

***just read over the site, and am downloading to try the trial right now. This could be an amazing program for anyone shooting 7Q raw... Those files are just so darn big. (One reason R3D files rock, is the variable compression ratios)
 
Last edited:
Dbatch said:
Awesome, great details CPC, thanks!

So, are there any downsides at all to this? 'Losslessly Compressed" seems to sound like there is no downside at all. (Maybe just more work for CPU/GPU to decode and uncompress each frame while live playback/editing? Kinda like h264 has to do, just on a RAW scale)

***just read over the site, and am downloading to try the trial right now. This could be an amazing program for anyone shooting 7Q raw... Those files are just so darn big. (One reason R3D files rock, is the variable compression ratios)
Click to expand...

The GPU doesn't come into play with CinemaDNG lossless compression/decompression beause lossless compression is not GPU friendly and I don't think any video application uses the GPU to decompress. So it comes down to CPU power. In general, losslessly compressed raw plays very nice with balanced GPU+CPU systems because while the CPU decompresses, the GPU debayers in parallel. Decompression and debayering just mesh together well. If you are processing uncompressed raw, the CPU is mostly idling.

The couple of potential downsides I can think of are: 1) if you are using a weak CPU, you can get slower playback compared to uncompressed; 2) if in your workflow you are using some video application which can't read losslessly compressed CinemaDNG, obviously it won't read the files.
I think 1) is nicely offset by the fact that with lossless compression storage bandwidth requirements are substantially lowered. You don't just save storage, you also lower the need for extreme storage speeds for playback since the files are smaller.
I'd also expect that some applications are less optimized for decompression compared to others. I am mainly a Resolve user and Resolve seems to love losslessly compressed CinemaDNG.

Would love to hear your impressions and what compression you get with your files - my fs700 footage is a bit limited in variety so my fs700 numbers are shaky. :)
 
The thing thats counts is that the quality can be maintained. I'd love to hear what others think, if its really lossless
 
interesting indeed but as there ale only 3 files in the demo it's quite impossible to judge if the compression works properly and really lossless.
 
Clermond said:
interesting indeed but as there ale only 3 files in the demo it's quite impossible to judge if the compression works properly and really lossless.
Click to expand...

Actually, the limit is files per sequence. You can do as many sequences as you want in a single batch and end up with tons of files to pixel peep.
 
@ Mitch: is there a possibility that CD engineers will take a look at this? Or is this concurrence?;)
 
@Fader8 we had this discussion before with Adobe DNG Converter which can also convert DNG files to a lossless compressed DNG. As far as I remember the Sony RAW files are compressed and have to get uncompressed. He said it would cause issues if you re-compress these files. Nevertheless I had no issues with the quality in Resolve - but I did not find a way to import the metadata as TC etc from the original files to the compressed DNGs. Copying the xml file into the new folder did not work.
btw this is sth I cannot test with the 3 files limitation..
 
Clermond said:
@Fader8 we had this discussion before with Adobe DNG Converter which can also convert DNG files to a lossless compressed DNG. As far as I remember the Sony RAW files are compressed and have to get uncompressed. He said it would cause issues if you re-compress these files. Nevertheless I had no issues with the quality in Resolve - but I did not find a way to import the metadata as TC etc from the original files to the compressed DNGs. Copying the xml file into the new folder did not work.
btw this is sth I cannot test with the 3 files limitation..
Click to expand...

thanks for th info. And where is the difference between slim raw and adobe dng converter?
 
Fader8 said:
thanks for th info. And where is the difference between slim raw and adobe dng converter?
Click to expand...

the files have not the exact same file size, so there might some difference. More important: slimRAW copies the XLM file with metadata and it does a checksum.
 
Clermond said:
the files have not the exact same file size, so there might some difference. More important: slimRAW copies the XLM file with metadata and it does a checksum.
Click to expand...

Here is a more complete list:
slimRAW is significantly faster: on my quad-core Intel i7-4770 it is 8 times faster than DNG Converter. Actually, this is the main reason slimRAW exists.
slimRAW compresses better, how much better depends on the frame content.
slimRAW doesn't butcher frame embedded cinemaDNG metadata, unlike DNG Converter which simply removes all of it.
slimRAW doesn't renormalize color matrices, unlike DNG Converter. Wtih slimRAW all color metadata is exactly the same as in the original uncompressed file.
slimRAW doesn't add thumbnails or bloated tags to frames, unlike DNG Converter.
slimRAW output works in Premiere CC/Speedgrade CC, unlike DNG Converter.

In short: slimRAW is about video, DNG Converter is about stills. :)

edit:
This article here (written by me) might give some more perspective on this: http://www.shutterangle.com/2015/slimraw-cinemadng-raw-video-lossless-compressor/
 
I bought SlimRAW yesterday had have been testing it out a bit with DNG files from FS700 and the Odyssey 7Q.

Folder of 91,349 DNG files.
Original: 312.5 GB
Slimmed: 120.8 GB

The files were on a older USB drive and took a little over 4 hours to compress, plus about an hour to verify using a maxed out 5k iMac.

I was wondering if using the Slim DNG would be slower than using the normal DNG files because they needed to be uncompressed, so I did an After Effects test to convert both the Slim DNG and normal DNG files to a Prores Quicktimes. The compressed DNG files were actually about 10-15% faster to convert to Quicktime. I imagine that's because the smaller, compressed DNG files were quicker to read from my Hard-drive.

I also did a test using the Difference overlay mode in After Effects to look for differences in the DNG files before and after compression...both files were exactly the same.
 
How do the new files hold up in grading or when you start working on them? Any noticeable difference?
 
forrestbro said:
I bought SlimRAW yesterday had have been testing it out a bit with DNG files from FS700 and the Odyssey 7Q.

Folder of 91,349 DNG files.
Original: 312.5 GB
Slimmed: 120.8 GB

The files were on a older USB drive and took a little over 4 hours to compress, plus about an hour to verify using a maxed out 5k iMac.

I was wondering if using the Slim DNG would be slower than using the normal DNG files because they needed to be uncompressed, so I did an After Effects test to convert both the Slim DNG and normal DNG files to a Prores Quicktimes. The compressed DNG files were actually about 10-15% faster to convert to Quicktime. I imagine that's because the smaller, compressed DNG files were quicker to read from my Hard-drive.

I also did a test using the Difference overlay mode in After Effects to look for differences in the DNG files before and after compression...both files were exactly the same.
Click to expand...

I am wondering how this compares to converting uncompressed RAW to ProRes 444 XQ? Have you tested how much of a difference is there between compressed RAW vs. ProRes 444 XQ as to the file size?
 
You must log in or register to reply here.
Back
Top