Page 8 of 8 FirstFirst ... 45678
Results 71 to 80 of 80
  1. Collapse Details
    #71
    Default
    How does the lossy work?

    On October 5, 2012, Adobe added lossy compression with version 1.4 of its DNG spec. DNG already had lossless, using a formula called Huffman. This new lossy way uses good old JPEG. In the spec it's limited to 8 bit, because at the time JPEG only did 8 bit. (Version 1.4 also added a second lossless formula, DEFLATE.)

    On January 20, 2014, the Independent JPEG Group released version 1.9 of JPEG, which added bit depths up to 12 bits per channel.

    On November 20, 2014, Blackmagic introduced lossy RAW, starting with a 3:1 ratio. Elsewhere I read that they base it on the aforementioned lossy DNG spec. But instead of limiting it to the spec's official 8 bits, they went with 12.

    My question is this. Before compression, how does lossy RAW split up the Bayer image? If you were to compress the whole image with JPEG, it would be inefficient. JPEG could do it, but it relies on swaths of color for savings. Meanwhile a raw Bayer image is guaranteed to have every pixel different from the one one right next to it.

    bayer.gif

    To me it makes the most sense to first extract all the green pixels, collapse the spaces in between, and JPEG-compress that image. Then do the same for red and blue. So a RAW compressed file would really be, in a sense, three images, each compressed separately. I have read that this is how Redcode does it. It just uses JPEG2000 instead of JPEG. In fact, the DNG spec itself seems to suggest this technique (see p. 20, Horizontal Difference X2).

    I just wasn't sure, because another article, talking about lossless compression, seemed to say that it just coalesced every other row, in effect splitting it into two different images, instead of three. It said that every other row has the same pattern, so the pixels would line up.

    Bayer_Pattern_Actual.gif

    But I think that it would still be much less efficient than cutting a Bayer image into three separate channels.


    ---

    Unrelated, I would like to see 10-bit log lossy compression, not just 12-bit.
    Last edited by combatentropy; 08-25-2017 at 08:58 PM.


    Reply With Quote
     

  2. Collapse Details
    #72
    Default
    @combatentropy:
    Blackmagic cameras compress all raw data as a single field. They do this in both lossless and lossy modes. Works ok for them in lossless (and yes, can be optimized, the nature of their log curve sort of hinders higher ratios), and in lossy they use the single field as a way to do bitrate control. slimRAW, on the other hand, splits channels in both lossless and lossy modes. The specification gives you leeway.
    I can't see any theoretical or practical need for nominal 10 bit log lossy. You can have any actual bitdepth below 12 by using 12-bit lossy and adjusting quantization accordingly. In fact, BM cameras (and Red cameras, and pretty much all cameras recording lossy raw) vary the true bitdepth all the time to achieve the desired constant data rate.


    Reply With Quote
     

  3. Collapse Details
    #73
    Senior Member
    Join Date
    Nov 2012
    Posts
    231
    Default
    @cpc:

    on my hexacore i4930k processor SLIMRAW only uses around 3% of CPU power - therefore, compression is rather slow. Is there a way to make Slimraw use more CPU-Power?


    Reply With Quote
     

  4. Collapse Details
    #74
    Default
    Quote Originally Posted by Fader8 View Post
    @cpc:

    on my hexacore i4930k processor SLIMRAW only uses around 3% of CPU power - therefore, compression is rather slow. Is there a way to make Slimraw use more CPU-Power?
    This is symptomatic for (very) slow storage. SlimRAW will use all the cpu power if your storage can keep up. What's your storage setup?


    Reply With Quote
     

  5. Collapse Details
    #75
    Senior Member
    Join Date
    Nov 2012
    Posts
    231
    Default
    Quote Originally Posted by cpc View Post
    This is symptomatic for (very) slow storage. SlimRAW will use all the cpu power if your storage can keep up. What's your storage setup?
    I now tried compressing from my RAID 0 (approx. 300-350MB) to a SATA SSD...now I get 13% CPU-Usage.


    Reply With Quote
     

  6. Collapse Details
    #76
    Default
    OK, this is a 4x improvement. You are probably not sustaining 300 MB/s throughput though. SlimRAW should be able to handle 700-800 MB/s on your CPU.
    Аn indicator of the current bottleneck during processing (input storage, output storage or CPU) is on the ToDo list.

    (And don't forget to update, version 1.8.2 was released yesterday: http://www.slimraw.com/download.html )


    Reply With Quote
     

  7. Collapse Details
    #77
    Senior Member
    Join Date
    Nov 2012
    Posts
    231
    Default
    Quote Originally Posted by cpc View Post
    OK, this is a 4x improvement. You are probably not sustaining 300 MB/s throughput though. SlimRAW should be able to handle 700-800 MB/s on your CPU.
    Аn indicator of the current bottleneck during processing (input storage, output storage or CPU) is on the ToDo list.

    (And don't forget to update, version 1.8.2 was released yesterday: http://www.slimraw.com/download.html )
    @CPC:


    Everybody is talking about ProRes RAW. In my opinion it won't be too different except that it's containered. That's one of the downsides for me with cinema DNG; loads of small files are slower to copy, archive, work with on HDDs / HDD RAIDs. I get about half performance with my 4 x 3 TB RAID 0 when working with cDNG.

    If you want to make a smart move an gain some new customers; offer a container converion over Slim Raw. E.g. uncompressed DNG to 3:1 / 4:1 containerd DNG. I would pay for the upgrade :-)


    Reply With Quote
     

  8. Collapse Details
    #78
    Senior Member Cary Knoop's Avatar
    Join Date
    Jan 2017
    Location
    Newark CA, USA
    Posts
    1,058
    Default
    Quote Originally Posted by combatentropy View Post
    H
    My question is this. Before compression, how does lossy RAW split up the Bayer image? If you were to compress the whole image with JPEG, it would be inefficient. JPEG could do it, but it relies on swaths of color for savings. Meanwhile a raw Bayer image is guaranteed to have every pixel different from the one one right next to it.
    JPEG compression uses DCT compression which does not rely on swaths of color. That would not be very efficient as human color perception is much worse than luminance perception.
    DCT compression compresses the luminance separate from the color information (usually in the form YCbCr). The luminance channel and the two color "channels" are compressed by calculating the correspondence to a range from low to high frequency patterns with a bias towards the low frequency patterns.

    For raw data there is only luminance data (on a fixed color pattern), so one DCT channel should be sufficient for compression. However a Bayer pattern in not balanced color wise (green is prevalent) and simply mapping DCT technology onto raw data would seem to introduce certain problems.
    Last edited by Cary Knoop; 04-19-2018 at 08:58 AM.


    Reply With Quote
     

  9. Collapse Details
    #79
    Default
    Quote Originally Posted by Cary Knoop View Post
    does not rely on swaths of color
    On swaths of tone in each channel, caused by swaths of color in the real world. You are right that JPEG compresses the colors separately. In fact I was the one who posted this set of videos about it, http://www.dvxuser.com/V6/showthread...n-Really-Works, which failed to create the viral sensation I had hoped ;)

    Still it would be a very wasteful not to split out the red, green, and blue pixels from a RAW sensor before compressing them. Otherwise the image is needlessly complicated:

    bayer.gif


    Reply With Quote
     

  10. Collapse Details
    #80
    Default
    Quote Originally Posted by Fader8 View Post
    @CPC:


    Everybody is talking about ProRes RAW. In my opinion it won't be too different except that it's containered. That's one of the downsides for me with cinema DNG; loads of small files are slower to copy, archive, work with on HDDs / HDD RAIDs. I get about half performance with my 4 x 3 TB RAID 0 when working with cDNG.

    If you want to make a smart move an gain some new customers; offer a container converion over Slim Raw. E.g. uncompressed DNG to 3:1 / 4:1 containerd DNG. I would pay for the upgrade :-)
    The problem comes from the lack of support for single container CinemaDNG in post production software. Even with MXF CinemaDNG support in slimRAW (there is MXF mapping defined in the CinemaDNG spec), the files would be unusable -- they can't be imported anywhere. A push in this direction should probably come from Blackmagic, cause they produce both CinemaDNG cameras and one of the major video production apps.


    Reply With Quote
     

Page 8 of 8 FirstFirst ... 45678

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •