A 4K Bayer array can be deBayered to 4K 444. If there's a restriction to 422 (1/2 horizontal color resolution) that's a performance limitation of the CD hardware/software. 6K doesn't do anything magical, but 8K does- allows a simple low pass filter to be performed yielding reduced aliasing. 6K can be resampled to 4K, however a more expensive filter is required (final result won't be as good as 8K source). Aliasing can also be reduced with a proper optical low pass filter.
Odyssey7Q + FS700 - 4K Modes
- Thread starter Dan Keaton
- Start date
Barry_Green
Moderator
That seems highly optimistic. 4K can't even be debayered to deliver 4K, much less 4K at 4:4:4... the normal res loss ranges from 33 down to about 20%, so a 4K sensor delivers typically around 3.2k to 3.4k of resolution. And color resolution is typically half the sensor res, so 2K's worth of color detail spread across a 4k frame that has an actual resolved detail of about 3.4k. Which means you usually have about 1.6 pixels worth of luminance information for every 1 pixel of color info.
How are you getting 4K 4:4:4 out of a 4K bayer sensor?
epsom salt
Well-known member
id prefer 4k
I used to be skeptical too until i deBayered 5D3 1920x1080 RAW with ACR. It's not full 1000+ lines, however it's pretty good. Some charts here: http://www.eoshd.com/content/10475/canon-1d-c-vs-5d-mark-iii-raw-and-c300-gh2-resolution-comparison
Note the C300 use 3.8K of green to get 1.9K final output (averaged/low-pass filtered- near optimal results). The 5D3 does pretty well with only 1.9K source Bayer pixels.
How is this possible? By using every single photosite/Bayer-pixel for luminance and chrominance. That is, multiple Bayer pixels are used to compute one output pixel color element. For example, a green element also uses red or blue elements along with filtering to compute the final green element output. The math is based on correlation, gradients, etc. (a variety of methods).
Compare the upper left image to the lower right image (Microsoft's 2004 version in the paper):
http://research.microsoft.com/pubs/102068/Demosaicing_ICASSP04.pdf
There are some edge artifacts, but it looks really good compared to the original, full resolution image.
I found a few other papers where some do a little better than the Microsoft paper, but appeared more expensive to compute, and some much more expensive to compute (the ones which looked almost perfect compared to the original).
The Microsoft version above might run on the CD embedded hardware; perhaps an even better version will work (or perhaps they have to use something simpler...). GPU versions are also common now (DaVinci Resolve hasn't matched Adobe's ACR in my tests; perhaps someone knows how to get Resolve to match or exceed ACR (or perhaps not available in the free version?)).
Agreed higher resolution sources will work better to reduce aliasing, however 444 is indeed possible given a decent enough deBayer algorithm.
Complex method: ftp://140.118.175.164/Demosaicing o...sks and adaptive Heterogeneity-projection.pdf
Faster method with excellent results: see page 24 to see how they get 444 and page 28 for image comparisons (excellent quality): http://www.eie.polyu.edu.hk/~enyhch...y_color_demosaicing_algorithm_based_on_IG.pdf
Last edited:
Barry_Green
Moderator
jcs... I don't think those papers are saying what you think they are...
You're looking at page 24 and seeing a filled-in block for red and apparently thinking that means that it's full-resolution red. That's not how it works.
Every demosaic results in a fully-populated red and a fully-populated blue and a fully-populated green. That's the purpose of a demosaic, is to reconstruct an RGB pixel value for each and every photosite. But those values are synthesized, they're not sampled. It's not actual resolution because it wasn't sampled individually at the sensor in the first place. The actual resolved detail is quite a bit lower.
Canon (with the C300) doesn't do a conventional demosaic on their 1080p at all. They do as close to 4:4:4 as you can get from a single sensor, specifically because they don't try to deliver 4K from it, they only deliver 1080p from it. So they have an individual sample for red, and an individual sample for blue, and actually two individual samples for green, for each and every destination pixel. So you have actual 4:4:4 color sampling for every destination pixel. And that works.
What you're talking about is fundamentally impossible. You cannot sample green from a red pixel. You cannot sample blue from a green pixel. The filters are placed over the pixels specifically to block the other spectrums of light and result only in the red, or the green, or the blue component from entering.
I think you should read some of Graeme Nattress's writings on demosaicing and color resolution, he's Red's engineer and writes the software behind Red's demosaicing and color science, and is one of the leading voices in demosaic technology today. And he gets about 78 to 80% resolved detail out of a 4K raw sensor, in terms of luminance. And he gets (last I checked) about 50% resolved color detail. Those are both quite high numbers and much better than the industry standard.
Nobody in the industry today can resolve 4K of detail off a 4K sensor -- and that's in terms of luminance, not even talking about color! Using all three colors together and combined to create luminance, the best of the best only delivers about 80%. When you get down to color, you're talking about a total source image of 2K of samples for red, and 2K of samples for blue. You cannot get 100% color fidelity and true color sampling out of a single Bayer sensor. It is not now possible, nor can it ever be possible... and I'll stick by that right up until Graeme proves me wrong. :thumbsup:
You're looking at page 24 and seeing a filled-in block for red and apparently thinking that means that it's full-resolution red. That's not how it works.
Every demosaic results in a fully-populated red and a fully-populated blue and a fully-populated green. That's the purpose of a demosaic, is to reconstruct an RGB pixel value for each and every photosite. But those values are synthesized, they're not sampled. It's not actual resolution because it wasn't sampled individually at the sensor in the first place. The actual resolved detail is quite a bit lower.
Canon (with the C300) doesn't do a conventional demosaic on their 1080p at all. They do as close to 4:4:4 as you can get from a single sensor, specifically because they don't try to deliver 4K from it, they only deliver 1080p from it. So they have an individual sample for red, and an individual sample for blue, and actually two individual samples for green, for each and every destination pixel. So you have actual 4:4:4 color sampling for every destination pixel. And that works.
What you're talking about is fundamentally impossible. You cannot sample green from a red pixel. You cannot sample blue from a green pixel. The filters are placed over the pixels specifically to block the other spectrums of light and result only in the red, or the green, or the blue component from entering.
I think you should read some of Graeme Nattress's writings on demosaicing and color resolution, he's Red's engineer and writes the software behind Red's demosaicing and color science, and is one of the leading voices in demosaic technology today. And he gets about 78 to 80% resolved detail out of a 4K raw sensor, in terms of luminance. And he gets (last I checked) about 50% resolved color detail. Those are both quite high numbers and much better than the industry standard.
Nobody in the industry today can resolve 4K of detail off a 4K sensor -- and that's in terms of luminance, not even talking about color! Using all three colors together and combined to create luminance, the best of the best only delivers about 80%. When you get down to color, you're talking about a total source image of 2K of samples for red, and 2K of samples for blue. You cannot get 100% color fidelity and true color sampling out of a single Bayer sensor. It is not now possible, nor can it ever be possible... and I'll stick by that right up until Graeme proves me wrong. :thumbsup:
Barry_Green
Moderator
Because the only way to get that 3200, is to start from 4000. This isn't an issue related to 4K, it's an issue inherent in all single-sensor camera systems. The single-sensor system, after undergoing a demosaic process, is going to yield somewhere around 80% of its nominal resolution. That said, sharper is sharper. 80% of 6K is sharper than 80% of 4K, which is still hugely sharper than 80% of 1080p...
Hey Barry- I write image and audio processing software during the day (filmmaking is a night and weekend thing). Currently writing a bunch of cool real-time video effects for iOS using the GPU (GLSL/shaders). Also a fan of real-time physics for games (wrote this (symplectic/energy accurate) physics engine from scratch: http://www.youtube.com/watch?v=DrCvjSXXgkk (ground is bicubic polynomial surfaces instead of polygons)). Perhaps boring to many folks here, but I enjoy understanding the math as well as the art.
I was not able to find any papers or math written by Graeme regarding the 3.2K from 4K Bayer resolution limit comment other than this forum post: http://www.reduser.net/forum/showthread.php?12026-Why-Bayer
I need to see the math describing exactly why 80% is the max believed possible resolution limit from a Bayer array.
In practice, I don't see this with 5D3 RAW 1920x1080 and ACR- it looks closer to 94% (about 900+ lines or ~1800 resolution vs. ~800 lines and ~1600 resolution from the proposed 80% limit).
See also this chart, comparing the superb 3.8K (sensor) C300 to the 5D3 1.9K RAW (I fully understand how the C300 is processed and not deBayered (wrote about this many times on this forum, including this thread)):
It's doing a lot better than 80% here (and in my own tests). Here Jason Green shoots a full test chart with 5D3 RAW (May 25th build- has some artifacts): https://vimeo.com/67090302. While there are clearly first artifacts appearing, line pairs can be seen all the way out to 1000 (5D3 is doing a decent job with filtering to prevent aliasing).
This might seem counterintuitive, but it is possible and it's one of the secrets to getting max information out of the Bayer array:
See page page 3: http://research.microsoft.com/pubs/102068/Demosaicing_ICASSP04.pdf
The other paper you referenced does similar processing.
Green from Red, Green from Blue, Red from Green, Red from Blue, and so on, depending on position in the Bayer array. The values are indeed reconstructed or synthesized, however it appears the practical limit is higher than 80%- perhaps 94% or more (perhaps not on embedded hardware yet). Those coefficients are multiplied by the cell values and added up then normalized to get the final color value. The coefficients are part of a Finite Impulse Response (FIR) filter (a Wiener filter: http://en.wikipedia.org/wiki/Wiener_filter, good a suppressing noise). They optimized these coefficients to produce the best results. How is this possible? Statistical correlation (they did all the hard work figuring out a decent way to implement it). See the Pixel Correlation section here: http://en.wikipedia.org/wiki/Demosaicing
Again, it is possible to get decent 444 from a Bayer array in post with a high-quality deBayer algorithm (as with ACR). Doing this well in real-time on embedded hardware is currently more challenging, but still possible.
I was not able to find any papers or math written by Graeme regarding the 3.2K from 4K Bayer resolution limit comment other than this forum post: http://www.reduser.net/forum/showthread.php?12026-Why-Bayer
I need to see the math describing exactly why 80% is the max believed possible resolution limit from a Bayer array.
In practice, I don't see this with 5D3 RAW 1920x1080 and ACR- it looks closer to 94% (about 900+ lines or ~1800 resolution vs. ~800 lines and ~1600 resolution from the proposed 80% limit).
See also this chart, comparing the superb 3.8K (sensor) C300 to the 5D3 1.9K RAW (I fully understand how the C300 is processed and not deBayered (wrote about this many times on this forum, including this thread)):
It's doing a lot better than 80% here (and in my own tests). Here Jason Green shoots a full test chart with 5D3 RAW (May 25th build- has some artifacts): https://vimeo.com/67090302. While there are clearly first artifacts appearing, line pairs can be seen all the way out to 1000 (5D3 is doing a decent job with filtering to prevent aliasing).
This might seem counterintuitive, but it is possible and it's one of the secrets to getting max information out of the Bayer array:
See page page 3: http://research.microsoft.com/pubs/102068/Demosaicing_ICASSP04.pdf
The other paper you referenced does similar processing.
Green from Red, Green from Blue, Red from Green, Red from Blue, and so on, depending on position in the Bayer array. The values are indeed reconstructed or synthesized, however it appears the practical limit is higher than 80%- perhaps 94% or more (perhaps not on embedded hardware yet). Those coefficients are multiplied by the cell values and added up then normalized to get the final color value. The coefficients are part of a Finite Impulse Response (FIR) filter (a Wiener filter: http://en.wikipedia.org/wiki/Wiener_filter, good a suppressing noise). They optimized these coefficients to produce the best results. How is this possible? Statistical correlation (they did all the hard work figuring out a decent way to implement it). See the Pixel Correlation section here: http://en.wikipedia.org/wiki/Demosaicing
Again, it is possible to get decent 444 from a Bayer array in post with a high-quality deBayer algorithm (as with ACR). Doing this well in real-time on embedded hardware is currently more challenging, but still possible.
Last edited:
paulcurtis
Veteran
+1 jcs
Barry, you're *technically* semantically correct but if you start with a large grid with a few holes and then fill in those holes with intelligent values based on the gradients around that hole you are creating resolution. If you just fill in those holes by copying the value next to it, you're not. It might not be real life resolution but you will get a pretty good approximation of what is really there in most cases. If an image changes over that single pixel then the OLPF should prevent it from being seen anyway. This can work well with contrasty edges etc,. but different debayers can approach this differently - and hence back to the original point of the only downside of not getting RAW from the 700 is the inability to do this.
The proof is in the pudding as they say and jcs provides some good examples. Ironically the worst case scenario for this is resolution charts as that's the whole point of their existence. In real world use it *can* and does make a difference to apparent resolution.
Frankly before people get uptight about the minutiae of resolution they should spend more time working on ways to make focusing a lens easier and more accurate, most resolution is lost with a soft shot, not a lack of resolution (from experience!). So more times focusing, less time processing raw....
cheers
paul
Barry, you're *technically* semantically correct but if you start with a large grid with a few holes and then fill in those holes with intelligent values based on the gradients around that hole you are creating resolution. If you just fill in those holes by copying the value next to it, you're not. It might not be real life resolution but you will get a pretty good approximation of what is really there in most cases. If an image changes over that single pixel then the OLPF should prevent it from being seen anyway. This can work well with contrasty edges etc,. but different debayers can approach this differently - and hence back to the original point of the only downside of not getting RAW from the 700 is the inability to do this.
The proof is in the pudding as they say and jcs provides some good examples. Ironically the worst case scenario for this is resolution charts as that's the whole point of their existence. In real world use it *can* and does make a difference to apparent resolution.
Frankly before people get uptight about the minutiae of resolution they should spend more time working on ways to make focusing a lens easier and more accurate, most resolution is lost with a soft shot, not a lack of resolution (from experience!). So more times focusing, less time processing raw....
cheers
paul
Barry_Green
Moderator
Semantics are sometimes important. I do understand and agree with the premise you're stating, but I have a couple of big problems with it too.
First, yes, you're synthesizing some resolution. But you're doing so by guessing. It is not sampled, it's guessed. It's synthesized. There's a difference -- and it's a massive huge difference when you're talking about 4:4:4 sampling, because you're not sampling it at all, you're guessing and interpolating and synthesizing.
Why does this matter? Because -- why does color sampling matter at all? It's well established that humans don't see color detail nearly as well as we see luminance detail. That's why so many of today's cameras use 4:2:0, where there's only one color sample for every four luminance samples, and -- frankly, it looks fine to most people. Lots of folks have done side-by-sides and determined that they can't really see any difference between 4:2:0 and 4:2:2. And, I'd dare say, there's a bigger perceivable difference between 4:2:0 and 4:2:2, than there is between 4:2:2 and 4:4:4.
So why does it matter? Why does anyone want 4:4:4 at all? Because not all footage is viewed just by eyeballs, lots of it is worked on in computers, and computers very definitely see the difference.
To argue that a bayer pattern can produce 4:4:4, and using your statements above, is no different than saying that putting 4:2:0 footage on the timeline and up-rezzing it to 4:4:4 is the same. Surely you would find that statement objectionable, right? 4:2:0 is not 4:2:2, and it's definitely not 4:4:4, yet when I work with it in the NLE, that's what the NLE does -- it has to up-rez to 4:4:4 to work on the footage.
But up-rezzing, and sampling, aren't the same thing. Not by a million miles. If they were, we never would be talking about 4K in the first place! We would just up-rez our HD and be fine. Or up-rez our SD.
Lest you think I'm being facetious, I'd like to ask you to really think about that for a second. Philosophically there's no difference between up-rezzing SD to HD, and up-rezzing (by demosaic'ing) bayer pattern data to 4:4:4. In both cases you're taking one sample and spreading it across 4 pixels.
Does mathematically up-rezzed SD to HD look better than just doubling the pixels? Sure. Does scaling up a still-frame of SD video in photoshop to 1920x1080 look better when you use bicubic interpolation than when you use duplication? Of course. But does either look like an actually-sampled 1920x1080 HD picture from a three-chip sensor and recorded to 4:4:4? Not at all.
4:4:4 means that you've sampled an individual color sample for each of the primary colors, at each pixel position. Bayer doesn't do that. It samples one color at each pixel position. Now, Paul mentioned "a large grid with a few holes" -- and that's the nature of the problem, except reversed. You don't have a large grid with a few holes, you have a large hole with a few pixels in it. 75% of the grid is empty. When you talk about red and blue, in each case, there is zero color information in 75% of the picture. You have to invent or synthesize 75% of the image, from a 25% source. That's not a trivial task. It's a very difficult task. In fact, I'd dare say it's impossible to reconstruct a 100% image, by guessing about what's missing.
So how do we get so close? Because, again, the eye is much less sensitive to color than it is to brightness. Using all the pixels together, you can get a really good representation of brightness (again, about 80% success in recreating what was originally there). But color? Nowhere near. Fortunately, we don't see color that distinctly, so the images still look good to us.
Can you make really good looking images off a Bayer-pattern sensor? Heck yes. Can you make native-resolution 4:4:4 off a Bayer-pattern sensor? Not possible.
Statistical reconstruction from a Bayer array is very different than up-rezzing an RGB image. Performing a high-quality statistical reconstruction of the Bayer array yields much better results vs. a fast/low-quality deBayer followed by any form of RGB up-rezzing (bilinear, bicubic, Lanczos, etc.).
Did you see the charts? The 1.9K 5D3 RAW with ACR compares very well against the C300 (which is 1000+ lines resolution). The full resolution chart I posted also shows line-pair differentiation all the way out to 1000 lines (though with first-artifacts starting sooner). Those are real-world results, which is backed up by the math. 444 means full color information in a YCbCr/YUV format (doesn't really apply to RGB). The charts and real world results show quite a bit better results than 80% (94+%). If all the Bayer cells weren't used to create the output, it would not be possible to get past 50% luminance and 25% chrominance. Take a look at the Microsoft paper and compare the original image compared to their reconstructed image. Very, very close in terms of quality.
Note that if multiple frames are used, the aliasing information can be used to create additional, real resolution: http://en.wikipedia.org/wiki/Superresolution. This is the next step in deBayering for video (there are papers which include SuperResolution in the Bayer reconstruction process). SuperResolution can reconstruct to higher resolution than originally sampled!
The Marvel-He-Cutler (Microsoft) deBayer has been efficiently ported to GPUs: http://graphics.cs.williams.edu/papers/BayerJGT09/
For those still with us, here's a cool program that includes some of the most advanced, recent deBayer methods: http://www.rawtherapee.com/
From forum posts, some of these should be able to do better than Adobe ACR (haven't tried it yet).
Did you see the charts? The 1.9K 5D3 RAW with ACR compares very well against the C300 (which is 1000+ lines resolution). The full resolution chart I posted also shows line-pair differentiation all the way out to 1000 lines (though with first-artifacts starting sooner). Those are real-world results, which is backed up by the math. 444 means full color information in a YCbCr/YUV format (doesn't really apply to RGB). The charts and real world results show quite a bit better results than 80% (94+%). If all the Bayer cells weren't used to create the output, it would not be possible to get past 50% luminance and 25% chrominance. Take a look at the Microsoft paper and compare the original image compared to their reconstructed image. Very, very close in terms of quality.
Note that if multiple frames are used, the aliasing information can be used to create additional, real resolution: http://en.wikipedia.org/wiki/Superresolution. This is the next step in deBayering for video (there are papers which include SuperResolution in the Bayer reconstruction process). SuperResolution can reconstruct to higher resolution than originally sampled!
The Marvel-He-Cutler (Microsoft) deBayer has been efficiently ported to GPUs: http://graphics.cs.williams.edu/papers/BayerJGT09/
For those still with us, here's a cool program that includes some of the most advanced, recent deBayer methods: http://www.rawtherapee.com/
From forum posts, some of these should be able to do better than Adobe ACR (haven't tried it yet).
alaskacameradude
Veteran
Anyone else remember the discussions about the HVX 200 and the 'pixel interpolation' that it did?
This reminds me a bit of that.
This reminds me a bit of that.
Barry_Green
Moderator
Yes, I saw them, and the charts EOSHD posted are not really useful for much because they only show down to 600 lines. Any decent HD camera nowadays should be able to cleanly resolve that. It's the area from 600 to 1000 where the funky stuff starts up.
In technique, yes. But in terms of the task that's being asked to be performed, no, they're basically exactly the same.
It's much easier to visualize if you just get down to the brass tacks. Ignore combining the patterns at all and look at what you're starting with. You have (in HD) 960x1080 of green. You have 960x540 of blue. You have 960x540 of red. By the time you're done, you have to have 1920x1080 of green, 1920x1080 of red, and 1920x1080 of blue. You do have to up-rez 960x540 of blue, and 960x540 of red, to get 1920x1080 of each.
Obviously the techniques used in recombining and reconstructing colors are going to differ from those used to strictly "scale up" an RGB image. But in concept, you're having to reach the same end goal -- take a smaller sample of data, and try to create a larger structure of data out of it.
Yes, and we've had this discussion before, and we disagree 180 degrees about what constitutes resolved detail. A camera can only resolve up until the point that it cannot. Any "resolved line pairs" after that point are, inherently, not actual resolved detail, they are aliases. Fake detail. Not legitimate sampling.
The point at which a camera first fails to resolve the black and white lines accurately, is the limit of its resolving capability. Whether or not you can discern individual black and white lines after that point is irrelevant, because that is not resolved detail, it's aliasing.
Barry_Green
Moderator
It should, because "pixel shift" (aka "spatial offset") and demosaic'ing are actually quite related.
In the HVX, they used three chips that where spatially offset to give an overlapping 1920x1080 sampling points out of 960x540 sensors. Then they used a somewhat-similar-to-demosaic process to combine the sensors into delivering an image that was, effectively, about 1440x810 in resolution.
Interestingly, if you take 1920x1080 single Bayer sensor and count on losing about 25% of the resolution to demosaicing, you end up with ... drum roll please ... 1440 x 810.
So yes, it's a comparable technique, with comparable results. Suitable for the task at hand (considering, especially, that the recording codec was 1280x1080), but not on par with the true 3-progressive-chip 1920x1080 systems more common today.
The difference is, nobody was claiming you could get 1920x1080 @ 4:4:4 out of the spatial-offset system. All that was claimed is that the images were "HD", which they were.
capt chuck
Well-known member
I've got a technical background, but this discussion is starting to make my head hurt. So much so I think I need some deBayer aspirin 
lol Chuck.I need some deBayer aspirin
Barry- no one is saying that 100% full luma and chroma are possible; I'm saying we can store useful information in 444 vs tossing 1/2 or 3/4 color information if we use a high quality deBayer. One point of this discussion is hoping that CD will use a modern high quality deBayer (and perhaps let us know which method / theory they are using).
When first artifacts occur on a test chart- that's a phase issue. Frequency folding doesn't happen until later. That's why there's a discussion of first artifacts through extinction of detail. The Arri Alexa has a strong OLPF which quickly blurs detail right after first artifacts- very little aliasing is recorded. I haven't seen that on any other camera. Comparing Fourier transforms of square waves (charts) might make it easier to compare resolution vs subjective eyeballing charts.
At the end of the day, my 5D3 RAW footage cuts perfectly with my FS700 footage (422 ext. or 420 internal), though the 5D3 colors are much better (14-bit). Waiting for 7Q footage to see if it can outperform 5D3 RAW for HD material.
Barry_Green
Moderator
Okay, then I'm officially confused. What I'm arguing against is the notion that you could get 4:4:4 images out of a Bayer sensor. And 4:4:4 means full resolution images in each color channel. Are you saying that's not what you're asserting?