Alexa / RED / Super 16 comparison footage

[Halsu]

Ok, waking up. s16 makes me the happiest, alexa is the first cam i've seen footage off that threatens that happiness...

... but the red? what happened to that window? surely exposing for the window can result in not too noisy shadows? did u not expose for the window? I mean I'm not into red, but still I'm sure it can't pale that bad. It looks like an hd-fake-shoulder next to those two...

reading over to see if i missed any exposure details...

Red was indeed overexposed it seems. Based on the experience i have with Red footage, it shouldn't have had any trouble with a scenes like that. Likely when exposing for midtones at ISO 320 the highlights got truncated, while there almost certainly was tons of "unused" dynamic range in the shadows. Based on my own experience with Red one MX, it could have been easily stopped town i.e. two stops to protect the highlights (to ISO 1280), and still give almost perfectly noiseless results.

 

[Halsu]

You're correct. I didn't expose for the highlights. That was my choice to do so, to expose them all at the same level. I could have exposed the RED for the windows and traded some shadows for highlights, but that wouldn't be a fair point of comparison would it.

In my opinion (like you remember from our discussion elsewhere), it would have been fair. In fact, that would have been the reasonable way to conduct such a test, if it's to be called a dynamic range test. Expose for highlights, see how much you can get from the shadows with an acceptable quality.

An easy way to do that would have been to simply change the ISO on the Red to 800 or higher. That way, assuming you use the default Redgamma2, you would have gotten the preseverved highs AND proper midtone reproduction, both in your on set monitor and in post, without a need to dig into color correction.

 

[Barry_Green]

Barry, how do you account for the discrepancies between this camera test and other camera tests performed by Red and Red One MX owners who find that the MX sensor had around 13.5 stops of dynamic range? We are having a pretty lengthy discussion about it on RedUser.

Are we talking about the SCCE test? Because this thread was about the John Brawley footage, right?

I'm gonna presume you're meaning the SCCE test?

The testing would/should be conducted simultaneously, using the same methodology, on the same chart, viewed on the same monitor. I don't know why some people could get 13.5 and others got 11.9, but I am pretty confident that when the exact same people are conducting the exact same test at the exact same time on the exact same chart on the exact same equipment, that the likelihood is that any observed differences would likely be consistent. If the SCCE team got 11.9 and the Red guys got 13.5, then it would seem reasonable to assume that the guys who got 13.5 out of the MX would get about 16 out of the Alexa under the same scenarios.

If you look closely at the Dynamic Range tests, especially the highlights test, the scenes are inconsistently lit from shot to shot.

The scenes were not inconsistently lit. They were lit identically, as I understand it. There was one case where there was a change in lighting, which was the fruits and vegetables scene, where the lighting was changed after the Alexa shot, and they 'fessed up to that as a mistake.

Lot of love for Steve and the ASC members who spent a lot of time on this test, but there are some major mistakes in this test, and I am surprised we could find these kinds of mistakes from people with their resumes.

While it would be pretty much impossible to conduct a test of this magnitude with zero mistakes, I would certainly not agree that there are "major mistakes" here, and if you're saying "Steve" as in Steve Weiss from Zacuto, then you have to understand that Steve Weiss was not involved in the test at all. This test was conducted by Bob Primes ASC, with Nancy Schreiber ASC and ... another ASC I don't remember, and 140 other people.

All Steve Weiss did was shoot a documentary of it.

Don't make any judgements based off of the Zacuto video. Either try to attend a screening of, or wait for the blu-ray of, the actual SCCE test. You can follow it on www.thescce.org.

 

[Jared Caldwell]

Are we talking about the SCCE test? Because this thread was about the John Brawley footage, right?

I'm gonna presume you're meaning the SCCE test?

The testing would/should be conducted simultaneously, using the same methodology, on the same chart, viewed on the same monitor. I don't know why some people could get 13.5 and others got 11.9, but I am pretty confident that when the exact same people are conducting the exact same test at the exact same time on the exact same chart on the exact same equipment, that the likelihood is that any observed differences would likely be consistent. If the SCCE team got 11.9 and the Red guys got 13.5, then it would seem reasonable to assume that the guys who got 13.5 out of the MX would get about 16 out of the Alexa under the same scenarios.


The scenes were not inconsistently lit. They were lit identically, as I understand it. There was one case where there was a change in lighting, which was the fruits and vegetables scene, where the lighting was changed after the Alexa shot, and they 'fessed up to that as a mistake.


While it would be pretty much impossible to conduct a test of this magnitude with zero mistakes, I would certainly not agree that there are "major mistakes" here, and if you're saying "Steve" as in Steve Weiss from Zacuto, then you have to understand that Steve Weiss was not involved in the test at all. This test was conducted by Bob Primes ASC, with Nancy Schreiber ASC and ... another ASC I don't remember, and 140 other people.

All Steve Weiss did was shoot a documentary of it.

Don't make any judgements based off of the Zacuto video. Either try to attend a screening of, or wait for the blu-ray of, the actual SCCE test. You can follow it on www.thescce.org.

Damn, Barry. I was just trying to ask your opinion, no need to be snarky with me here.

Yes, I am talking about the SCCE. I apologize if I didn't make that clear before, I didn't mention it because I thought quoting your post containing SCCE would show what I meant. Sorry for the miscommunication there.

I'm not saying for certain that if the Red guys measured 13.5 on the MX that they would not have measured 16 out of the Alexa, but I am not convinced. I suspect that camera operation is the culprit for the discrepancies. The Red One and the Alexa are different cameras, and to me, it doesn't seem likely that when other tests have shown a .5 to 1 stop difference in dynamic range between the Alexa and the MX, that the SCCE measurements would be right. Maybe they are, and that was the basis of my question to you earlier. Do you wonder why there is a discrepancy between the two (when other tests show the contrary), or do you just accept the test for what it is?

As for the inconsistent lighting, it is a big deal to me. Nobody's perfect, but it weakens the test (especially when this "shootout" has been touted as being one of the most extensive and professional tests that have been executed with modern d-cinema cameras). Maybe we can agree to disagree. I'm fine with that on this point.

I know that Steve (Weiss) did not create the test, I would just liked to have seen more information on the methodologies used in this segment of the shootout within his documentary. Maybe I should have been more clear on that as well.

I am only judging this segment of the shootout, and it's fine that we have a difference of opinion on what a "major mistake" is. It's only my opinion. The main area I was asking your opinion on initially doesn't require me to watch it on Blu ray, charts with those numbers are clear enough already in compressed web video.

The main question I was really asking was how do you account for the discrepancies with the dynamic range of the Red One MX to other dynamic range tests that have been conducted that contradict these finding?

I thought that asking you about it would be beneficial, especially after reading your informative posts for years. I wasn't expecting a slightly condescending response in return. I am just trying to use a critical eye when going over camera tests, and I thought you might have some info to give. That's all I meant to do bud.

 

[Barry_Green]

Damn, Barry. I was just trying to ask your opinion, no need to be snarky with me here.

Did I come across as snarky? Apologies, I certainly didn't mean to be!

I do seem to come across as a bit defensive about the SCCE, and I wasn't even involved in it. But having done more than my share of multi-camera evaluations, and being usually skewered for it, and then seeing just how meticulous and thorough these folks were, yeah, I do bristle a bit when it's suggested that these guys screwed it up somehow, because I think they did it as good as it could possibly be done.

But I certainly didn't mean to come across as attitudinal, so sorry about that!

I'm not saying for certain that if the Red guys measured 13.5 on the MX that they would not have measured 16 out of the Alexa, but I am not convinced. I suspect that camera operation is the culprit for the discrepancies. The Red One and the Alexa are different cameras, and to me, it doesn't seem likely that when other tests have shown a .5 to 1 stop difference in dynamic range between the Alexa and the MX, that the SCCE measurements would be right. Maybe they are, and that was the basis of my question to you earlier. Do you wonder why there is a discrepancy between the two (when other tests show the contrary), or do you just accept the test for what it is?

I am inclined to take this test as being completely accurate. It is difficult to do a proper comparison between two disparate systems because it's hard to keep all the millions of variables straight. However, these folks brought in experts who were dedicated to each system, and had so many additional pairs of eyes doing checks and cross-checks, and I have talked to a few of the folks who have testified of just how meticulous they were throughout the process (especially, especially in light of the complaints that Red lodged against the PGA Camera Assessment Series) that I really, really doubt that they screwed anything up, most especially when it comes to the Red.

Red has made claims throughout the years, such as that "film skews just like our rolling shutter", which was conclusively proven at the SCCE to just not be true.

Dynamic range really depends on the chart you're using to test. You could use a cheapo Stouffer, which I bought and discarded because it wasn't accurate and was designed for scanners, not cameras. You could use a DSC 13-stop or 17-stop chart, which is what I have now. Or you could use the chart that the SCCE used, which I'd never seen or used before. But the crucial thing is to use the same chart, at the same time, in the same conditions, on the same monitor, with the same folks viewing it. Put all that together all at the same time, and you're much more likely to get accurate answers.

I know that Steve (Weiss) did not create the test, I would just liked to have seen more information on the methodologies used in this segment of the shootout within his documentary. Maybe I should have been more clear on that as well.

All the information, and more, will be forthcoming. The SCCE is going to publish a white paper that documents everything. I would again reiterate, I wouldn't judge anything based off of the Zacuto documentary. The real test is a totally different presentation. I haven't even watched the Zacuto doc, because I've seen the SCCE presentation itself repeatedly, and that's what I consider the real test results.

The SCCE has said that they will make all their results public, and even that it will be released into the public domain.

The main question I was really asking was how do you account for the discrepancies with the dynamic range of the Red One MX to other dynamic range tests that have been conducted that contradict these finding?

Again, all I can say is that what matters is to have competent administrators over each camera, use the same equipment, and let the chips fall where they may. Red was repeatedly invited to supply a camera and a tech, and they chose not to. If they wanted to be able to ensure that their camera was represented best, they should have made the effort. As it is, there are lots and lots of highly qualified Red DIT's out there, and I would be startled into next week if it turns out that they didn't properly and accurately represent what the Red One MX is capable of delivering.

I wasn't expecting a slightly condescending response in return.

Again, apologies, that was never meant.

 

[Jared Caldwell]

Thanks for the clarification, Barry. If you didn't intend to come across like that, then I will just chalk it up to the general terseness of the Internet.

As an aside, I very much respect the professionals that worked to make this test happen (and I respect the tests and posts you have made in the past as well). Anytime you put your work out there, the potential for criticism is always out there, especially with acquisition of quantitative data. I am just concerned, because I see a big difference between 13.5 stops of dynamic range previously reported and the number obtained by the SCCE. Something is off somewhere (I don't know where and/or who), and I am just trying to investigate to satisfy my own curiosity. I don't mean to hold the participants in the test to anything below the highest esteem for their efforts, but as a scientist, I'm still going to investigate the data, you know?

Thanks for your response. I await the white papers.

 

[Halsu]

The testing would/should be conducted simultaneously, using the same methodology, on the same chart, viewed on the same monitor.

One thing is sure, one shouldn't use the real time output of the camera.

Tests like this should be conducted from the camera files (they likely were), and those files can (one would venture to say should) be examined using i.e. levels adjustment to bring out the contrast in both highlights and the shadows separately to really see what's going on. The idea is to see what is the absolute maximum brightness variation that the camera can register between clipping and noise floor. That usually can't be determined from a regular video monitor, with regular viewing settings.

That's also not necessarily a direct indication of what the usable range is - as long as a brightness difference can be spotted, it's counted. This doesn't necessarily tell how visually pleasing (or ugly) a real life image is when the range is pressed.

I don't know why some people could get 13.5 and others got 11.9, but I am pretty confident that when the exact same people are conducting the exact same test at the exact same time on the exact same chart on the exact same equipment, that the likelihood is that any observed differences would likely be consistent. If the SCCE team got 11.9 and the Red guys got 13.5, then it would seem reasonable to assume that the guys who got 13.5 out of the MX would get about 16 out of the Alexa under the same scenarios.

I think this is a bit of a mistake - there was a comparison between Red Epic and Alexa on the Red forums, and i think folks are talking about that, the numbers seem to match. It's the same sensor, but Epic saves 16 bit RAW, Red One MX "only" 12 bit. It would be logical that R1:s format is the limiting factor as a 12 bit linear file can only hold 12 stops (11.9 measured stops would make sense here). I may be wrong with this though, this is my own pondering, not confirmed by Red or anyone else for that matter.

If the above is correct, it would partly explain why there's so little noise in even the darkest blacks with Red One - some of the sensor's noise is below the file format's ability to save. In practice, the difference lies in the very darkest portions of any scene, and in a vast majority of cases, those differences will be hidden more or less completely by color correction anyway.

Here's the chart and the thread:

http://reduser.net/forum/showthread.php?47529-EPIC-and-Alexa

If i'm correct, Red One's chart should look the same as Epics, except for the darkest step or two.

 

[Razz16mm]

Epic has two generations newer electronics. DR isn't all about the sensor. S/N ratio in the associated electronics is just as important. I tend to agree that 12 stops is about right for R1 MX conversion. The original M sensor hit about 10 stops+ with the same FPGA based processors. Alexa is 3 year newer design and electronics. Epic should hit an honest 14 stops as the software matures.
This is not to say that any of these cameras wouldn't look better in these tests if tweaked and shot to their individually optimum settings.

 

[Halsu]

Epic has two generations newer electronics. DR isn't all about the sensor. S/N ratio in the associated electronics is just as important.

As far as i know, in CMOS sensors the signal is usually amplified and digitized in the sensor itself - the sensors output data, not analog voltage.

 

[S. Matthews]

tests and ego's very hard things to manage! and to trust!

 

[Rick Burnett]

As far as i know, in CMOS sensors the signal is usually amplified and digitized in the sensor itself - the sensors output data, not analog voltage.

This is usually true, but doesn't mean it is always true. However, for performance reasons, it would make NO sense to do this off the CMOS sensor, as the speed gain of doing it on the same CMOS would be MUCH faster than off chip. One of the reasons people started using CMOS was to make a cheaper sensor since you could put all the digital and analog support circuitry on the same die. You cannot do this with CCD technology.

 

[Ian-T]

tests and ego's very hard things to manage! and to trust!

True in a lot of cases. It seems John did an honest test though based on what he was looking for in his type of work.

 

[Halsu]

I found a suitable high contrast R1MX/Alexa comparison online, which included the camera originals. I did some tests with those, and here's what i came up with:

http://reduser.net/forum/showthread.php?59507-Camera-shootout-2011/page25#244

To cut it short, the two can be matched more or less exactly to each other except for the darkest two stops or so, where Alexa has the edge. To match the two, Red should be exposed 2 stops darker than Alexa (i.e. with ND).

 

[Robert Ruffo]

Alexa seems to be what the producer's like. It's a bit more expensive than the RED...which is also more expensive than the previous season's format...

jb

have you used Red Color 2? Do you really know how to post it? Our MX footage looks WAY better than your test.

 

[Robert Ruffo]

I found a suitable high contrast R1MX/Alexa comparison online, which included the camera originals. I did some tests with those, and here's what i came up with:

http://reduser.net/forum/showthread.php?59507-Camera-shootout-2011/page25#244

To cut it short, the two can be matched more or less exactly to each other except for the darkest two stops or so, where Alexa has the edge. To match the two, Red should be exposed 2 stops darker than Alexa (i.e. with ND).

Exactly 1.268 stosp according to our tests, not 2. Remember Red noise is easier to clean up because it is much higher res which noise reducers like.

 

[Robert Ruffo]

Damn, Barry. I was just trying to ask your opinion, no need to be snarky with me here.

Yes, I am talking about the SCCE. I apologize if I didn't make that clear before, I didn't mention it because I thought quoting your post containing SCCE would show what I meant. Sorry for the miscommunication there.

I'm not saying for certain that if the Red guys measured 13.5 on the MX that they would not have measured 16 out of the Alexa, but I am not convinced. I suspect that camera operation is the culprit for the discrepancies. The Red One and the Alexa are different cameras, and to me, it doesn't seem likely that when other tests have shown a .5 to 1 stop difference in dynamic range between the Alexa and the MX, that the SCCE measurements would be right. Maybe they are, and that was the basis of my question to you earlier. Do you wonder why there is a discrepancy between the two (when other tests show the contrary), or do you just accept the test for what it is?

As for the inconsistent lighting, it is a big deal to me. Nobody's perfect, but it weakens the test (especially when this "shootout" has been touted as being one of the most extensive and professional tests that have been executed with modern d-cinema cameras). Maybe we can agree to disagree. I'm fine with that on this point.

I know that Steve (Weiss) did not create the test, I would just liked to have seen more information on the methodologies used in this segment of the shootout within his documentary. Maybe I should have been more clear on that as well.

I am only judging this segment of the shootout, and it's fine that we have a difference of opinion on what a "major mistake" is. It's only my opinion. The main area I was asking your opinion on initially doesn't require me to watch it on Blu ray, charts with those numbers are clear enough already in compressed web video.

The main question I was really asking was how do you account for the discrepancies with the dynamic range of the Red One MX to other dynamic range tests that have been conducted that contradict these finding?

I thought that asking you about it would be beneficial, especially after reading your informative posts for years. I wasn't expecting a slightly condescending response in return. I am just trying to use a critical eye when going over camera tests, and I thought you might have some info to give. That's all I meant to do bud.

As mentioned, it's 1 and one quarter stops, less if you use noise reduction because you can't noise reduce 1080p for 1080p as aggressively as you can 4K. Other people here thought it was more like 0.75 stops. Metering for 300 ASA is simply the wrong way to shoot Red. It is 800-1000 iso. People also fail to understand that you don't really shoot any ISO at all since nothing is baked in. In other words, they simply metered the shot wrong - what you see is what Red was able to recoup despite bad shooting procedure. You wouldn't meter 1000 iso film at 50 would you? Same principle.

You then can apply some pretty aggressive curves in post - with Arri not so much - I mean 800iso metered Red footage borders on HDR if handled with very low contrast in post.

 

[Halsu]

Exactly 1.268 stosp according to our tests, not 2.

How did you get that number?? In the Gunleik's footage i used, the clipping point was very close on both cameras, when using 2 stops more ND on the Red.

Remember Red noise is easier to clean up because it is much higher res which noise reducers like.

What i found when examining the noise in the very deepest shadows was that, surprisingly to me, once the exposure was compensated by 2 stops in order to protect the highlights, Red's noise was mushed (very likely by the compression) to the extent that it was much coarser than Alexa's even after being scaled down to 1080p. As said earlier, i didn't shoot the footage, and i don't know what compression settings were used on the Red.

 

[Patro]

I know I'm late on this thread, but...

I know I'm late on this thread, but...

Sorry to disappoint you. As I mentioned in the blog, this was not graded or prettied up. Just shooting with what was there.

also,

From the SCCE.

jb

So I saw this chart, and I'd just like to add one I found from the "ultimage camera shootout" one year.

​

Notice that while the Alexa have similar dynamic ranges, middle exposure falls into a different spot on each. Film is able to reach into the highlights and retain detail, while sometimes losing the shadows to black. The Alexa seems kind of centered, so it can reach down into the shadows and pull more detail. However, it still has a problem blowing out highlights. Some people prefer one, while others prefer the other. It's kind of about what look you want. Personally, I'd rather deal with clipped shadows than clipped highlights. I just think a white spot on someone's face looks bad.

Anyway, this is why you can overexpose film so much without ruining it completely. I wouldn't dare overexpose digital, as it clipps to white so easily. I've recently started playing around with a GH2 in preparation for a short film my friends and I are making, and I've noticed that the highlights are easily a problem. I also have a Krasnogorsk-3 that I plan to shoot a film on, and the test footage I shot blew away my expectations. Film is so gorgeous (especially when projected) and so forgiving. Anyway, thanks for listening to me ramble. The Alexa and some others are impressive cameras, just not quite the same as film though.

 

[Rick Burnett]

What these charts aren't showing is the curve characteristic of the sensitivity of the capture. Film reactivity as you reach overexposure slows down due to the fact that photons might be hitting portions that are already exposed whereas with digital, it doesn't work like that. That is one of the reasons they look different because digital doesn't work like that at all, instead, after the image is captured from the sensor it is then (unless you are staying raw) converted into a curve that matches what the film sees but not what the sensor actually saw.

 

[mail4joeg]

Well that explains a lot, as for why I like Alexa/Film and dislike the RedMX. Pretty much what I expected to see. Alexa is the super clean version of film. Film has the color, character, charm and grain we expect.