RAW compression and food for thought

P

paulcurtis

Veteran
I was looking at some of the 2K RAW footage and seeing the mosquito noise around areas thinking this looks like compression artefacts. Then it dawned on me. They probably are.

2K 240fps through a 3G-HDSDI connection must be compressed. The connection is 2.9Gbit/s which is about 380MB/s so the math doesn't work.

This made me wonder about the 4K output, because a 4K frame at 12 bit (RAW) i think is around 13MB/frame. So uncompressed that's just about 30fps through that single connection. The 4K output is supposed to be 60fps. Hmm.

I presume the same is true of the AXR recorder as well, the bottle neck is the single connection.

I wonder how the signal is compressed, whether it's lossy (i guess so in the case of 240 fps RAW) or lossless (maybe for 4K).

Am i wrong, is my math wrong or is there something here that needs to be looked at?

curious...
Paul
 
one frame in sonys 4k mxf raw is about 4,7 MB, hence apparently only BW luminance information of each pixel needs to be recorded.

2k raw has 4 times less pixels than 4k so 240fps 2k and 60fps 4k have the same total amount of pixels per second.

The problems you witness are propably due to sony way of getting the raw 2k out of 4k sensor. if you could post a sample picture of the problem it would help to verify its source.
Are you looking at 2k raw recorded with sonys recorder?
 
Sami2,

Is my math wrong? I'm aware that the RAW is luma only, so i did 4096x2160 pixels (8,847,360) each pixel is 12 bits assuming Sony are packing the bits and not padding to 16bit. So times that number by 12 and that's how many bits the frame is. And divide by 8 for bytes, then divide by 1024 again for K which is 12,960K or 12MB a frame for a 12 bit RAW 4K.

If my assumption is correct then if the actual frame is 5MB it must have compression, because the RAW data stream is not 5MB.

Whether that compression is lossless or lossy is really my question...

i'll see if i can grab a still of what i mean.

And yes the example i'm giving above is from Filippos 2K O7Q short, although i have seen similar issues in the F55 2K RAW and the RAW via the AXR.

thanks
Paul
 
Last edited:
I was under the impression that about 2.5x compression was applied to FS700 RAW to fit the data over the SDI port. This appears to be about right. I wonder how it compares to the 3.6x (RAW-SQ) and 7x (RAW-Lite) compression that Sony's RAW R4/R5 decks use and whether there may be a generational loss if using the Sony R5 deck means compressing the bayer data once 2.5x to get it over SDI and then decompressing it and re-compressing it again 3.6x to go to AXSM card.

In any case, while 2.5x lossless compression is possible in general, in this case consistent 2.5x compression probably cannot be mathematically lossless because mathematically lossless bitrates are very much scene-dependent. I'm sure to achieve constant 2.5x compression, it must be lossy, but it may be able to be characterized as "visually lossless", though to me that always seems like an odd designation for RAW data since RAW data's visual form is very dependent on debayer-algorithm and post-processing.
 
Enclosed are some crops from Resolve. This is from a DNG file from the short film shot with the O7Q at 2K RAW.

Firstly i'm not sure Resolve is debayering correctly, the one on the left is with the DNG file being set to Rec709 colorspace the one on the right is BMD film colourspace.

The artefacts jump right out in Rec709 but the film 'version' is more subtle, however if you gain up the film version you can easily see the noise around the shoulder i'm talking about.

So the first observation is i'm sure the Rec709 can't be right, but that's the default in Resolve, so what particular flavour of DNG does the O7Q use? Secondly even if the main edge problems are resolved (pun sorta intended) then the mosquito noise is still quite apparent. This is perhaps the simple result of RAW compression.


2kraw.jpg

The 2K RAW file from CD (the slow mo fountain shot show the same thing too_)

2kraw2.jpg

You can see magenta/blue noise too in the sand coloured block [EDIT, the jpeg compression has hidden this, in the original i see coloured noise in the sand]. The glints on the water show artefacts too.

Now compare this to the O7Q 4K image down sampled to 2K

2kraw3.jpg

Wow, that is a stunning image. Noise minimised and coloured correctly. No issue with glints, edges of lines PERFECT, no aliasing noise. (the original 4K is pretty stunning too - no edge problems or aliasing, just a bit softer)

So, perhaps i'm debayering this 2K RAW stuff wrong, but that's my observation so far. Can anyone point me where i'm wrong?

cheers
Paul
 
Hi Paul, here is my math:

12-bit RAW is ((4096 * 2160* 12) / 8) * 24 = 339.738.624 bytes / second = 318 MB/s

So 318MB/s for 4k @ 24fps capture without compression.

80MB/s for 2k @ 24fps capture without compression.

800Mb/s for 2k @ 240fps capture without compression.

If 3.0Gb/s is the max for 3G-SDI then the minimum compression ratio has to be 6.4Gb / 3.0Gb = 2.13 : 1


I don't however think what you're seeing has anything to do with compression but EVERYTHING to do with how Sony is getting these high frame rates:

As per the 2k 120fps MXF file thread in the F3/F5/F55 section Sony recently confirmed to me that they are using "pixel grouping" to derive HD/2k images from a 4k sensor. If this is the case, then what I believe we are seeing here are the effects of this. Sony has reduced the spatial resolution in order to improve the sensitivity and noise acceptability required for shooting at these extremely high frame rates.

It's basically cheating because the sensor itself can not be read in its entirety at 240fps, or because the ASIC inside can not process the information at such high data rates.
 
Last edited:
paulcurtis said:
Enclosed are some crops from Resolve. This is from a DNG file from the short film shot with the O7Q at 2K RAW.

Firstly i'm not sure Resolve is debayering correctly, the one on the left is with the DNG file being set to Rec709 colorspace the one on the right is BMD film colourspace.

The artefacts jump right out in Rec709 but the film 'version' is more subtle, however if you gain up the film version you can easily see the noise around the shoulder i'm talking about.

So the first observation is i'm sure the Rec709 can't be right, but that's the default in Resolve, so what particular flavour of DNG does the O7Q use? Secondly even if the main edge problems are resolved (pun sorta intended) then the mosquito noise is still quite apparent. This is perhaps the simple result of RAW compression.


View attachment 77854

The 2K RAW file from CD (the slow mo fountain shot show the same thing too_)


So, perhaps i'm debayering this 2K RAW stuff wrong, but that's my observation so far. Can anyone point me where i'm wrong?

cheers
Paul
Click to expand...

I think you are seeing the results of excessive aliasing from line-column skipping to get 2k raw coupled with a not very sophisticated debayer process. Have you tried changing the debayer quality settings in Resolve?
 
I'm not sure if what we're looking at is post de-bayer issue (maybe some alternative settings will pretty it up) but as far is the methods performed on the sensor and in-camera we do not know 100% for sure the camera uses line-skipping. Sony recently confirmed to me that they are using pixel grouping (I call that pixel binning) on the F5/F55 to achieve HFR in HD/2k from 4k sensor so I would assume Sony uses the same technique for the FS700.

Having said that either method results in the same type of problem, ugly edges.

An anti-aliasing filter is going to be the only fix for this. Tweaks to the de-bayering algorithm might help but in the end garbage in garbage out.
 
starcentral said:
I'm not sure if what we're looking at is post de-bayer issue (maybe some alternative settings will pretty it up) but as far is the methods performed on the sensor and in-camera we do not know 100% for sure the camera uses line-skipping. Sony recently confirmed to me that they are using pixel grouping (I call that pixel binning) on the F5/F55 to achieve HFR in HD/2k from 4k sensor so I would assume Sony uses the same technique for the FS700.

Having said that either method results in the same type of problem, ugly edges.

An anti-aliasing filter is going to be the only fix for this. Tweaks to the de-bayering algorithm might help but in the end garbage in garbage out.
Click to expand...

AA filter is definitely the best approach, but non-real time high quality debayer algorithms can clean it up a lot. Love to get some 2k DNG sample frames to play with.
 
Razz16mm said:
I think you are seeing the results of excessive aliasing from line-column skipping to get 2k raw coupled with a not very sophisticated debayer process. Have you tried changing the debayer quality settings in Resolve?
Click to expand...

I tried tweaking what i was able too. I've only tried Resolve though.

As Dennis says, Sony do claim pixel binning.

Frankly the best solution is to only use 2K for >120fps, the rest of the time use 4K and downsample. In theory 4K continuous up to 60p (if we're lucky) then, on the 700 at least, 120fps 4K in burst mode.

I don't think there are *too* many people that need 240fps continuous. The 2K RAW seems like a speciality feature to me, not a day to day one.

I'm really not that fussed about it, i'm much more concerned about getting 4K out of the O7Q :)

cheers
Paul
 
paulcurtis said:
I tried tweaking what i was able too. I've only tried Resolve though.

As Dennis says, Sony do claim pixel binning.

Frankly the best solution is to only use 2K for >120fps, the rest of the time use 4K and downsample. In theory 4K continuous up to 60p (if we're lucky) then, on the 700 at least, 120fps 4K in burst mode.

I don't think there are *too* many people that need 240fps continuous. The 2K RAW seems like a speciality feature to me, not a day to day one.

I'm really not that fussed about it, i'm much more concerned about getting 4K out of the O7Q :)

cheers
Paul
Click to expand...

Still love to run a couple of those 2k dng frames through the 96 bit floating point raw processors I'm using to see if they can make a difference. Might be a useful alternate workflow to someone for some specialty shots if it helps.
 
Aa, that explains it all, you are looking at crappy frames from CD. 2k raws poor performance is addressed ages ago.

4k raw is brilliant without any compression artifacts. there could be a zip like compression, etc on the data as it is transfered to sonys interface. Does sony state somewhere that they are only using 380MB/s when transfering data from camcorder to the interface. They could use a bit more bandwidth, hence the signal camera outputs is not restricted by any standard, since it is sonys own link between camcorder and interface. So they could be pulling a lot more bandwidth out of the connection.
Has any seen any datailed specs on what the data actually is that is transfered between camcorder and the interface?

So i would not bother with math, but instead look at the results we are getting out of sonys own recorder in 4k. You can find quite a few 4k raw clips from that online and look for compression artifacts.

2k is ****, but when there is need for 240fps having the frame rate even at shitty 2k raw is awesome :)
 
starcentral said:
Hi Paul, here is my math:

12-bit RAW is ((4096 * 2160* 12) / 8) * 24 = 339.738.624 bytes / second = 318 MB/s

So 318MB/s for 4k @ 24fps capture without compression.

80MB/s for 2k @ 24fps capture without compression.

800Mb/s for 2k @ 240fps capture without compression.

If 3.0Gb/s is the max for 3G-SDI then the minimum compression ratio has to be 3.0Gb / .800Gb = 3.75:1
Click to expand...


Not quite. 12bit 2K 240fps RAW uncompressed is ~800MBps which is 6.4Gbps, and 6.4Gbps/3Gbps= 2.13 so if 3Gbps is the limit then 2K 240fps has to be compressed at least 2.13x to fit over 3G-SDI. Of course most likely it will need more compression after the extra headers and embedded sound datastreams are accounted for etc, but then again perhaps Sony is not completely limited by 3G-SDI protocols as the data they are sending may not be formatted for standardized SDI anyway? In either case, I'm pretty sure I heard some time back that ~2.5x compression was confirmed for the bayer data traveling out over SDI from the FS700.

Yes 2K is not perfect, but when no other camera offers even the FS700's 2K image quality for 120/240fps for even twice the price, its a bit hard to complain. Yes the 2K has some aliasing and more noise but it still looks pretty awesome and doesnt crop and its really not such a bad tradeoff if you need 240fps...
 
Last edited:
I wonder if Sony's RAW viewer application will get support for CD's DNG RAW files. That might help a bit since I imagine some of the issues here might just be interpretation of RAW data from the unknown FS700 camera. DNG may be supported, but from what I've seen, it goes a really long way if the software you are using to view/process your RAW knows what camera it came from and is designed to process data from that camera properly. I know I opened some FS700 DNG files a little while ago in Adobe RAW and everything looked wayy off. A little software support can go a long way and the 7Q isn't even out yet so hopefully they will test some workflows and maybe resolve/adobe/sony will add support for FS700/7Q DNG RAW.

Another reason nice looking 4k RGB/YUV/prores out of the box might help a lot since no doubt the 7Q's FS700 upgrade option will be specifically tweaked to process FS700 RAW properly.
 
nyvz said:
I wonder if Sony's RAW viewer application will get support for CD's DNG RAW files. That might help a bit since I imagine some of the issues here might just be interpretation of RAW data from the unknown FS700 camera. DNG may be supported, but from what I've seen, it goes a really long way if the software you are using to view/process your RAW knows what camera it came from and is designed to process data from that camera properly. I know I opened some FS700 DNG files a little while ago in Adobe RAW and everything looked wayy off. A little software support can go a long way and the 7Q isn't even out yet so hopefully they will test some workflows and maybe resolve/adobe/sony will add support for FS700/7Q DNG RAW.

Another reason nice looking 4k RGB/YUV/prores out of the box might help a lot since no doubt the 7Q's FS700 upgrade option will be specifically tweaked to process FS700 RAW properly.
Click to expand...
I can tell you that we use different deBayer algorithms for different RAW cameras. Yes, the FS700 record option is specifically tweaked for that camera's output. And as long as we think we can improve it the tweaking will never stop.
 
Mitch Gross said:
I can tell you that we use different deBayer algorithms for different RAW cameras. Yes, the FS700 record option is specifically tweaked for that camera's output. And as long as we think we can improve it the tweaking will never stop.
Click to expand...

Great to hear it. I'd be interested to hear the status updates on the debayer options. Last I checked the 7Q only monitored FS700 RAW with what looked like a pretty ugly nearest neighbor debayer algorithm and the challenge of debayering 2k 240fps was up in the air. Any progress being made on those fronts? Or at least reassuring optimism for a pre-orderer? I suppose i could just wait for the shipping product and eventual FS700 update to find out, but I'm curious what will be possible :)

Even if 240fps debayered realtime to a prores recording isnt possible, perhaps a buffered version? Hard to imagine most people rolling a long time on 240fps anyway... Just the added quality/formats and a little bit of a buffer length or readout waiting time that beats what the FS700 has built in could go a long way.
 
nyvz said:
Even if 240fps debayered realtime to a prores recording isnt possible, perhaps a buffered version? Hard to imagine most people rolling a long time on 240fps anyway... Just the added quality/formats and a little bit of a buffer length or readout waiting time that beats what the FS700 has built in could go a long way.
Click to expand...

I'm curious about this also. I specifically asked Mitch if they are going to do a 240fps 4:2:2 10-bit option and he said "Yes, it's coming soon." I'm wondering how they'll do it as 240fps RAW would just kill storage space like no tomorrow.
 
Gents, we have programmers working on it now. When I can tell you more I will.

We are improving our deBayer engine. We have one that currently functions at 2K 240fps RAW. It will ship as the current FS700 Record Option. But we are working to make it look that much better and to be more efficient coding. When we get any improvement done it will be a part of a free firmware update.

We are also working on our compression encoding. 2K 240fps CONTINUOUS to a quality compressed format is our goal. This is actively in development on a rigorous timeline. This will also be a part of a free firmware update.
 
You must log in or register to reply here.
Back
Top