SlimRAW - compress DNG Files
- Thread starter Fader8
- Start date
cpc
Well-known member
There will always be a little less color information available in 5:1 compared to 4:1. But because you'll be downscaling for the final result, you get the benefit of multiple pixels contributing to one output pixel, so there is still a significant amount of color info affecting the output. I suggest doing a quick side by side experiment: choose a challenging shot, do it in 4:1 and 5:1, apply the same grade to both, export to 1080p, see if you notice any difference.
Hey Mihail!
Thanks for the update, one little suggestion...
why don't you start a collaboration with Kinefinity to implement KRW2.0 files to CinemaDNG conversion inside SlimRAW?
It would be beneficial for both of you guys!
Or maybe you could outsource to them your conversion engine or maybe developing a Kinefinity-branded app for footage ingestion and conversion?
What do you think??
This would be awesome! Then every Kinefinity user would be lining up to buy this.
cpc
Well-known member
There is no publicly available specification for Kinefinity's raw format and communication dies somewhere in China.
Feature Request for SlimRaw:
SlimRaw is such a awesome application. I basically makes it possible to use 4K RAW with the non compressed files coming from the Sony Cameras. Though, it would be nice to see some more features, which I think shouldn't be to hard to implement:
- being able to create a queue with multiple jobs. It would be so convenient to set up multiple jobs with possibly different settings and compression rates, so that when one job ends the next starts immediately.
- when you have different resolutions on your SSD and you want to compress 2k at a different rate than 4k it is very unconvenient, because you have to check each folder coming from e.g. the Odyssey, sort it and have multiple jobs (see request above!). It would be amazing if SlimRaw would be able to read the resolution automatically and compressed different resolutions at different compression rates.
- SlimRaw should tell the OS not to fall into sleep mode
- would be nice to see the writing speed of the files
- create a Plugin / Program so that compressed DNG files can be opened without the need for Resolve
I hope SlimRaw keep evolving!
SlimRaw is such a awesome application. I basically makes it possible to use 4K RAW with the non compressed files coming from the Sony Cameras. Though, it would be nice to see some more features, which I think shouldn't be to hard to implement:
- being able to create a queue with multiple jobs. It would be so convenient to set up multiple jobs with possibly different settings and compression rates, so that when one job ends the next starts immediately.
- when you have different resolutions on your SSD and you want to compress 2k at a different rate than 4k it is very unconvenient, because you have to check each folder coming from e.g. the Odyssey, sort it and have multiple jobs (see request above!). It would be amazing if SlimRaw would be able to read the resolution automatically and compressed different resolutions at different compression rates.
- SlimRaw should tell the OS not to fall into sleep mode
- would be nice to see the writing speed of the files
- create a Plugin / Program so that compressed DNG files can be opened without the need for Resolve
I hope SlimRaw keep evolving!
Last edited:
combatentropy
Veteran
How does the lossy work?
On October 5, 2012, Adobe added lossy compression with version 1.4 of its DNG spec. DNG already had lossless, using a formula called Huffman. This new lossy way uses good old JPEG. In the spec it's limited to 8 bit, because at the time JPEG only did 8 bit. (Version 1.4 also added a second lossless formula, DEFLATE.)
On January 20, 2014, the Independent JPEG Group released version 1.9 of JPEG, which added bit depths up to 12 bits per channel.
On November 20, 2014, Blackmagic introduced lossy RAW, starting with a 3:1 ratio. Elsewhere I read that they base it on the aforementioned lossy DNG spec. But instead of limiting it to the spec's official 8 bits, they went with 12.
My question is this. Before compression, how does lossy RAW split up the Bayer image? If you were to compress the whole image with JPEG, it would be inefficient. JPEG could do it, but it relies on swaths of color for savings. Meanwhile a raw Bayer image is guaranteed to have every pixel different from the one one right next to it.
To me it makes the most sense to first extract all the green pixels, collapse the spaces in between, and JPEG-compress that image. Then do the same for red and blue. So a RAW compressed file would really be, in a sense, three images, each compressed separately. I have read that this is how Redcode does it. It just uses JPEG2000 instead of JPEG. In fact, the DNG spec itself seems to suggest this technique (see p. 20, Horizontal Difference X2).
I just wasn't sure, because another article, talking about lossless compression, seemed to say that it just coalesced every other row, in effect splitting it into two different images, instead of three. It said that every other row has the same pattern, so the pixels would line up.
But I think that it would still be much less efficient than cutting a Bayer image into three separate channels.
---
Unrelated, I would like to see 10-bit log lossy compression, not just 12-bit.
On October 5, 2012, Adobe added lossy compression with version 1.4 of its DNG spec. DNG already had lossless, using a formula called Huffman. This new lossy way uses good old JPEG. In the spec it's limited to 8 bit, because at the time JPEG only did 8 bit. (Version 1.4 also added a second lossless formula, DEFLATE.)
On January 20, 2014, the Independent JPEG Group released version 1.9 of JPEG, which added bit depths up to 12 bits per channel.
On November 20, 2014, Blackmagic introduced lossy RAW, starting with a 3:1 ratio. Elsewhere I read that they base it on the aforementioned lossy DNG spec. But instead of limiting it to the spec's official 8 bits, they went with 12.
My question is this. Before compression, how does lossy RAW split up the Bayer image? If you were to compress the whole image with JPEG, it would be inefficient. JPEG could do it, but it relies on swaths of color for savings. Meanwhile a raw Bayer image is guaranteed to have every pixel different from the one one right next to it.
To me it makes the most sense to first extract all the green pixels, collapse the spaces in between, and JPEG-compress that image. Then do the same for red and blue. So a RAW compressed file would really be, in a sense, three images, each compressed separately. I have read that this is how Redcode does it. It just uses JPEG2000 instead of JPEG. In fact, the DNG spec itself seems to suggest this technique (see p. 20, Horizontal Difference X2).
I just wasn't sure, because another article, talking about lossless compression, seemed to say that it just coalesced every other row, in effect splitting it into two different images, instead of three. It said that every other row has the same pattern, so the pixels would line up.
But I think that it would still be much less efficient than cutting a Bayer image into three separate channels.
---
Unrelated, I would like to see 10-bit log lossy compression, not just 12-bit.
Last edited:
cpc
Well-known member
@combatentropy:
Blackmagic cameras compress all raw data as a single field. They do this in both lossless and lossy modes. Works ok for them in lossless (and yes, can be optimized, the nature of their log curve sort of hinders higher ratios), and in lossy they use the single field as a way to do bitrate control. slimRAW, on the other hand, splits channels in both lossless and lossy modes. The specification gives you leeway.
I can't see any theoretical or practical need for nominal 10 bit log lossy. You can have any actual bitdepth below 12 by using 12-bit lossy and adjusting quantization accordingly. In fact, BM cameras (and Red cameras, and pretty much all cameras recording lossy raw) vary the true bitdepth all the time to achieve the desired constant data rate.
Blackmagic cameras compress all raw data as a single field. They do this in both lossless and lossy modes. Works ok for them in lossless (and yes, can be optimized, the nature of their log curve sort of hinders higher ratios), and in lossy they use the single field as a way to do bitrate control. slimRAW, on the other hand, splits channels in both lossless and lossy modes. The specification gives you leeway.
I can't see any theoretical or practical need for nominal 10 bit log lossy. You can have any actual bitdepth below 12 by using 12-bit lossy and adjusting quantization accordingly. In fact, BM cameras (and Red cameras, and pretty much all cameras recording lossy raw) vary the true bitdepth all the time to achieve the desired constant data rate.
cpc
Well-known member
This is symptomatic for (very) slow storage. SlimRAW will use all the cpu power if your storage can keep up. What's your storage setup?
cpc
Well-known member
OK, this is a 4x improvement. You are probably not sustaining 300 MB/s throughput though. SlimRAW should be able to handle 700-800 MB/s on your CPU.
Аn indicator of the current bottleneck during processing (input storage, output storage or CPU) is on the ToDo list.
(And don't forget to update, version 1.8.2 was released yesterday: http://www.slimraw.com/download.html )
Аn indicator of the current bottleneck during processing (input storage, output storage or CPU) is on the ToDo list.
(And don't forget to update, version 1.8.2 was released yesterday: http://www.slimraw.com/download.html )
@CPC:
Everybody is talking about ProRes RAW. In my opinion it won't be too different except that it's containered. That's one of the downsides for me with cinema DNG; loads of small files are slower to copy, archive, work with on HDDs / HDD RAIDs. I get about half performance with my 4 x 3 TB RAID 0 when working with cDNG.
If you want to make a smart move an gain some new customers; offer a container converion over Slim Raw. E.g. uncompressed DNG to 3:1 / 4:1 containerd DNG. I would pay for the upgrade
Cary Knoop
Veteran
JPEG compression uses DCT compression which does not rely on swaths of color. That would not be very efficient as human color perception is much worse than luminance perception.
DCT compression compresses the luminance separate from the color information (usually in the form YCbCr). The luminance channel and the two color "channels" are compressed by calculating the correspondence to a range from low to high frequency patterns with a bias towards the low frequency patterns.
For raw data there is only luminance data (on a fixed color pattern), so one DCT channel should be sufficient for compression. However a Bayer pattern in not balanced color wise (green is prevalent) and simply mapping DCT technology onto raw data would seem to introduce certain problems.
Last edited:
combatentropy
Veteran
On swaths of tone in each channel, caused by swaths of color in the real world. You are right that JPEG compresses the colors separately. In fact I was the one who posted this set of videos about it, http://www.dvxuser.com/V6/showthread.php?360018-How-Image-Compression-Really-Works, which failed to create the viral sensation I had hoped
Still it would be a very wasteful not to split out the red, green, and blue pixels from a RAW sensor before compressing them. Otherwise the image is needlessly complicated:
cpc
Well-known member
@CPC:
Everybody is talking about ProRes RAW. In my opinion it won't be too different except that it's containered. That's one of the downsides for me with cinema DNG; loads of small files are slower to copy, archive, work with on HDDs / HDD RAIDs. I get about half performance with my 4 x 3 TB RAID 0 when working with cDNG.
If you want to make a smart move an gain some new customers; offer a container converion over Slim Raw. E.g. uncompressed DNG to 3:1 / 4:1 containerd DNG. I would pay for the upgrade
The problem comes from the lack of support for single container CinemaDNG in post production software. Even with MXF CinemaDNG support in slimRAW (there is MXF mapping defined in the CinemaDNG spec), the files would be unusable -- they can't be imported anywhere. A push in this direction should probably come from Blackmagic, cause they produce both CinemaDNG cameras and one of the major video production apps.