Inside DNG Files

[Gene Sung]

Real world tests have it that the FS700 will give you 12 stops of range, though. Especially with needing to keep down the noise in the blacks in post. Technically Sony is right but at the end of the day it is best for ourselves, our videos, and our clients if we are just honest with what the camera can really do.

But if you can clearly see 14+ stops on that chart, what does that say? From direct real world tests I've seen between the 5D3 RAW (DXO rated at 12 stops) and FS700 S-Log 2 is clearly at the very least 1 stops more DR than 5D3, which would put it at 13+. I'll find the test when I get home. FS700 S-Log definitely has more DR than 12 stop rated 5D3 RAW in real world tests.

The weird thing with S-Log is that the DR seems to be really influenced by the amount of light. In daytime, the DR is really great but inside and in low light, the darks get noisy fast. S-Log 2 DR, at least on the FS700 seems to have 2 personalities where as the DR from Cinegammas seem a bit more consistent from good light to bad light. It's like, the DR from S-Log just drops a lot more than Cinegammas when going from light to dark, if that makes sense. That's just based on my own observations and not charts.

 

[Colinelves]

But if you can clearly see 14+ stops on that chart, what does that say? From direct real world tests I've seen between the 5D3 (DXO rated at 12 stops) and FS700 S-Log 2 is clearly at the very least 1 stops more DR than 5D3, which would put it at 13+. I'll find the test when I get home. FS700 S-Log definitely has more DR than 5D3 in real world tests.

I believe you've said before that you have mic greater tolerance than most people for noise - so that's where you get the extra range from: it all depends upon how much noise you can tolerate in which bits of your image. Paul's data does show more discernible stops but a) they're represented by very very few code values (so you will see them on a chart, but in the real world of subtle gradients, they'll have a lot of banding on them) and b) they're really far into the noise floor - so using them means living with a lot of noise.

In terms of practical real world use, I'd say it's good to know that, generally, you have 12 good, usable clean stops - meaning you should light for this range when you can, but if you really really need a couple more and you'd rather live with video noise and blocky blacks than video clipping - you have it if you need it.

Personally I'm not a fan of the CD LUT for raw as it assumes the 2000 base ISO that Sony has determined for slog2 to 'maximise' DR, but this, of course, relies on these dodgy shadow stops: meaning you get noisy footage (or must adjust for an overexposed LUT).

 

[Gene Sung]

Is there any way for CD to tweak the S-Log profile so it handles low light better? Like I said, in bright daylight, the dynamic range just seems insane and the footage is so clean if you expose to the middle or right, but once you get indoors or low light, the blacks are noise city with any underexposure.

Out doors S-Log, I don't ever have to do much to the blacks, but indoor shots I really have to bring the blacks down to hide the noise. Much more so than Cinegammas. S-Log 2 is almost like Jekyll and Hyde.

 

[Gene Sung]

I believe you've said before that you have mic greater tolerance than most people for noise - so that's where you get the extra range from: it all depends upon how much noise you can tolerate in which bits of your image. Paul's data does show more discernible stops but a) they're represented by very very few code values (so you will see them on a chart, but in the real world of subtle gradients, they'll have a lot of banding on them) and b) they're really far into the noise floor - so using them means living with a lot of noise.

Got it, I get what you guys are saying now.

 

[Marino215]

I have the AE plugin Dark Energy by cinnafilm and using that I can get a stop or two of extra DR without a hitch, especially indoors.
I never used Neat video but can say that Dark energy is fantastic on fs700 footage.

 

[paulcurtis]

Hi Paul,

Quick question for you. If 12-bit linear is limited to 12 stop, how are people like Art Adams able to shoot 14 stops on his tests? Are they false stops? If you look at his chart, you can clearly see 14, even the beginning of a 15th stop. I'm confused a bit by the various reports as Art says about 13 useable stops in his article. Also, Sony specs the FS700 at 14 stops. Really confusing.

View attachment 89910

http://www.dvinfo.net/article/acquisition/sonyxdcam/fs700-the-4k-story-from-my-perspective.html/4

Also, many people are claiming the A7s as having around 14 stops of DR. I would image that the A7s S-Log 2 would be similar to the FS700s. So does the A7s not have linear RAW?

It's a very fair point. What i'm seeing is that the source RAW data is 12 bit Linear.

When that is colour matrixed to real world colours, the RAW RGB (overly green) is combined into real world RGB. However remember that green is more sensitive (so being brighter it has more steps as it is further up the linear graph), so you're mixing those 12 bit RAW colours in a way that is showing more tonal range overall. This is key and i'm not sure Razz16 is seeing that. Perhaps a new graph is in order

I've also been concerned that the transmission wedge i use is wrong. However i can count down my values and see that every 3 steps the level halves, which is what i would expect. I've also done tests by stopping down or using shutter speed. However i think that lens apertures are rarely very accurate anyway.

In terms of Art Adams (and others tests) for over all range there are a couple of variables in play

a) Do these tests use highlight recovery to extend the top range. You can see from the image on the blog post that green clips a stop earlier than the other colours. Depending on your workflow this may get clipped or that last stop can be 'made up'. Step wedges are the ideal situation to see highlight recovery. So figure an extra stop because of that. The real world somewhat less so.

b) When do you feel there is no more data and it's all noise, that's a very personal thing. Tim Dashwood did a test with (i think 17 stops) of ND and if you download the DNG you can make out the different between the black surround and ALL the stops. So does that mean 17 stops? Well no, because IMHO those last 6 stops were pretty much the same. But they were there but just noise. It seems very difficult to get the 700 to output actual black. If there is *any* light then it appear to record a signal.

The A7s does not have 14 stops in video. I don't even know where that comes from. Well mine doesn't and i think macgregor in a different thread has reached the same conclusion. I will do the same tests with it and also do it via RAW stills as well (where there probably is 14 stops).

Slog2 can only hold 13 stops anyway. Maybe 13.5. Slog3 can hold more.

My point though is that in the real world NONE of these cameras perform like the specs being thrown around. Look at dpreviews camera tests and compare with the 700, you'll see the 700 does an incredible job. So all things being relative the 700 is very good. Not an Alexa but no one should claim that it is (but then some alexa footage is super noisy and i know people who have had real issues with it in post)

cheers
Paul

 

[Mitch Gross]

One thing rarely accounted for in such tests is color temperature. Even Kodak B&W film stocks had different sensitivities when lit by daylight then lit by tungsten.

All silicon-based sensors are blue-starved. They work more efficiently the higher the color temperature goes. This is why in the early days of the RED camera some DPs took to adding a light blue filter in front of the camera to shift the overall balance. In daylight you are actually filling the three color channels fairly evenly. In tungsten the blue channel is quite low, and invariably this is where you see noise first.

So the issue of sensitivity and dynamic range is not - so to speak - so black & white.

 

[Gene Sung]

One thing rarely accounted for in such tests is color temperature. Even Kodak B&W film stocks had different sensitivities when lit by daylight then lit by tungsten.

All silicon-based sensors are blue-starved. They work more efficiently the higher the color temperature goes. This is why in the early days of the RED camera some DPs took to adding a light blue filter in front of the camera to shift the overall balance. In daylight you are actually filling the three color channels fairly evenly. In tungsten the blue channel is quite low, and invariably this is where you see noise first.

So the issue of sensitivity and dynamic range is not - so to speak - so black & white.

thanks for the explanation Mitch, that makes a lot of sense. Seems like S-Log 2 is particularly susceptible to color temperature sensitivities, or at least compared to Cinegammas.

 

[Razz16mm]

I've mentioned before but please can you send me one of these DNGs then i can have a look at the actual data values in the file. The ones i'd looked at previously were 12 bit straight. And as i mentioned this is a quote from the engineer himself:

>Yes- the max range is about 12 stops.
>It is fairly linear across the range. Our optical black is around 30 so this is the start, so the first 4 stops are not quite 0 to 16.
>There is a slight toe but not too much in the darks.

In the face of evidence suggesting otherwise and a direct quote from the people responsible i don't know how you can carry on saying this!!

Thanks everyone else for the nice compliments, i do have a lot more to write up.

I've also noticed that the images don't link on iDevices, so i will need to make some changes to the site - each image has a bigger version which has a lot more detail.

cheers
Paul

Because every raw processor I open a D16 file in sees the image data value range as 16 bit linear, not 12 bit.

In Scratch I can manually choose the debayer algorithm and enter the data range for the debayer process to interpret the file by. Max white data point and min black data point.
If I enter a max white point data value of 4096 for 12 bits this is what I get.

If I enter a max white point data value of 65535 for 16 bit, I get this:

In both cases black level is set to 20. Bilinear algorithm.

 

[Razz16mm]

Same image as linear light sensor value without display gamma correction:

4096 WP

65535 WP

 

[paulcurtis]

David (Razz?)

Can you send me that DNG and i can send you back the raw data as a 16 bit tiff from the SDK.

All your images are post debayered, you really want to be looking at this stuff before the debayer if you want to see what the camera output. Once it's in another colourspace (sRGB or whatever) then all bets are off because the debayering will linearise to 16 bit regardless of whether that is a simple scale from 12 or a more complex LUT transform. The images you posted are not the raw data.

cheers
Paul

 

[cuervo]

Paul, Just got around to reading your post. Pretty much confirms the approach I've been using to expose 1-2 stops over what a lightmeter would predict. Not exactly ASA 800, more like 600, I would say, for Slog2. But, in general, the technical info you present is very informative, so thanx. Switching between O7Q (4K to REC709) and (4K to Slog2) is rather problemattic as each profile demands a different exposure philosophy. But, at least, now I understand why. And, indeed, I've found wide variation in post workflow results. I'd be very interested to see what you come up with in your next blog. I've been liking the ACES workflow, because it's fairly quick. But, when I have the time to fiddle around, I seem to always do better manually correcting my images, no LUTs, as they all seem to do things to the image I don't like.

 

[cpc]

It is important to understand that the linearization table (as in the BM cameras) is not required in order to have more DR than seems possible with the available bits. You can have 14 stops of DR in a 12-bit DNG file. This is achieved by simply not having a linear signal in the first place. Sony have been doing this regularly with the raw files of their photo cameras. They often use 12 bits (with no linearization table) but the dynamic range is actually higher because there are non-linearities in the range, usually at the black end. Sony Nex-6, Nex-7, etc have DR exceding 12 stops but stored in 12-bit raw files with no tables.
Also cameras from other manufacturers have non-linearities in the upper end: you will often note the specified white level in Canon raw files is significantly lower than the actual possible range (as defined by the number of bits) because the ADC becomes non-linear when approaching the channel saturation points.

@Razz: your Scratch example seems like a Scratch idiosyncrasy, they possibly normalize everything to 16 bits in their UI. Every D16 file I've seen is simple 12-bit linear data.

 

[Razz16mm]

David (Razz?)

Can you send me that DNG and i can send you back the raw data as a 16 bit tiff from the SDK.

All your images are post debayered, you really want to be looking at this stuff before the debayer if you want to see what the camera output. Once it's in another colourspace (sRGB or whatever) then all bets are off because the debayering will linearise to 16 bit regardless of whether that is a simple scale from 12 or a more complex LUT transform. The images you posted are not the raw data.

cheers
Paul

https://www.dropbox.com/s/fogyllzag5gk6rq/BLX-0025-20140622-0000001.dng

Interested to see what you find.

Edit: This frame pushes the full DR of the camera pretty hard. The DR of the scene by the light meter between the specular highlights and the black at the bottom of the tea kettle is a bit over 13 stops. I'm clipping 1 or 1-1/2 stops of specular highlights so 11-12 stops of scene DR in the captured image. The interior is exposed about 2 stops under 0 EV compared to the exterior reading.

https://www.dropbox.com/s/ftog0z9f60flc8d/BLX-0003-20140702-0000001.dng

 

[paulcurtis]

Looking at the first one i can see that it's a little under exposed, the black point is 20, as DBX have told be but the white point is reduced which is odd. So the range is from 20 to 3450, it is 12 bit for sure as DBX have always maintained.



Taking a sample of the brightest part of the frame i see that the raw value max out at 2500. So there's another stop of exposure that could have been used. You can tell from the image just by looking that it is not using clipping anywhere.

In your second DNG, the white point reports as being higher - so perhaps different firmware versions.

The values run from 30 to 4095 which uses the full 12 bit range



I can't really tell from these images whether the camera is straight linear as we'd need to use a step wedge to test but DBX said that it is and i don't see any reason not to believe them.

So i would maintain as i've said, that the raw camera data is 12 bit.

A close up on the over image image stats:



The average value for the scene do imply linear, most of the image data is pretty low down the scale.

Once something debayers it in 16 bit space then it becomes quite tough to tell what the source was. But just because it ends up in 16 bit space it doesn't mean that it started off as 16 bit values.

cheers
Paul

 

[paulcurtis]

It is important to understand that the linearization table (as in the BM cameras) is not required in order to have more DR than seems possible with the available bits. You can have 14 stops of DR in a 12-bit DNG file. This is achieved by simply not having a linear signal in the first place. Sony have been doing this regularly with the raw files of their photo cameras. They often use 12 bits (with no linearization table) but the dynamic range is actually higher because there are non-linearities in the range, usually at the black end. Sony Nex-6, Nex-7, etc have DR exceding 12 stops but stored in 12-bit raw files with no tables.
Also cameras from other manufacturers have non-linearities in the upper end: you will often note the specified white level in Canon raw files is significantly lower than the actual possible range (as defined by the number of bits) because the ADC becomes non-linear when approaching the channel saturation points.

@Razz: your Scratch example seems like a Scratch idiosyncrasy, they possibly normalize everything to 16 bits in their UI. Every D16 file I've seen is simple 12-bit linear data.

I'm not sure the DNG format supports anything other than a straight Linear encoding OR a customer encoding with a 1D LUT in each file, does it?

Of course you could encoded whatever you like in the source file which is why the highlight and shadow have a slight curve on the 700. But the key point i'm making is sure, you can just about see more shadow stops but they're not exactly useful.

AFAIK Sony RAW (in terms of stills) using some form of delta compression to maximise the range. In some cases i believe the raw files are actually 8 bit. But this is totally different from what we're working with, so we can't really draw comparisons.

cheers
Paul

 

[Egg Born Son]

We need an emoticon with an exploding head!

Enjoying this thread and your article although it may be a while before I'm able to fully comprehend it. Your writing style is easy to follow so I am at least getting the gist of it and hopefully I'll be taking something out of it I can use if not fully understand. Thanks.

 

[Razz16mm]

Looking at the first one i can see that it's a little under exposed, the black point is 20, as DBX have told be but the white point is reduced which is odd. So the range is from 20 to 3450, it is 12 bit for sure as DBX have always maintained.

View attachment 90001View attachment 90002

Taking a sample of the brightest part of the frame i see that the raw value max out at 2500. So there's another stop of exposure that could have been used. You can tell from the image just by looking that it is not using clipping anywhere.

In your second DNG, the white point reports as being higher - so perhaps different firmware versions.

The values run from 30 to 4095 which uses the full 12 bit range

View attachment 90003

I can't really tell from these images whether the camera is straight linear as we'd need to use a step wedge to test but DBX said that it is and i don't see any reason not to believe them.

So i would maintain as i've said, that the raw camera data is 12 bit.

A close up on the over image image stats:

View attachment 90004

The average value for the scene do imply linear, most of the image data is pretty low down the scale.

Once something debayers it in 16 bit space then it becomes quite tough to tell what the source was. But just because it ends up in 16 bit space it doesn't mean that it started off as 16 bit values.

cheers
Paul

No question the DNG file is 12 bit linear data. But I don't see it corresponding to a 12 bit linear light value range. Your quote from the DB engineer mentioned a "slight toe" at the bottom, which could be done in either the analog or digital domain on the D16, essentially similar to what a log recording curve does in video, but not so extreme. Raising the first stage 16 bit deep shadow values into the 12 bit table for better shadow resolution. Knee and toe compression have been common in video cameras since the early 80's.

The kitchen shot was close to optimum exposure for the lighting conditions to preserve the full range of scene values. I downloaded Rawdigger to take a look too. There is roughly an equal percentage of white clipped highlights and black clipped shadow and a minimal amount of both. The beauty of raw is the ability to push the midrange scene values where you want them in post while retaining full highlight and shadow details.

The firmware is the same in both cases, but the dog shot was at 100 ISO and the kitchen shot at 200 ISO camera settings. Changing ISO on the D16 is not like most CMOS cameras. Everything changes, analog gain and slopes, shutter clock timing, AD conversion. I haven't done formal chart tests yet but the ISO speeds were factory calibrated to a Sekonic light meter and seem to match my old Sekonic L398 readings quite well. I expose the D16 by meter, not by monitor, using the same methods I would use for film.

Thanks for turning me on to Raw digger by the way. That is going to be a very useful tool for profiling the camera.

 

[paulcurtis]

No question the DNG file is 12 bit linear data. But I don't see it corresponding to a 12 bit linear light value range. Your quote from the DB engineer mentioned a "slight toe" at the bottom, which could be done in either the analog or digital domain on the D16, essentially similar to what a log recording curve does in video, but not so extreme. Raising the first stage 16 bit deep shadow values into the 12 bit table for better shadow resolution. Knee and toe compression have been common in video cameras since the early 80's.

The kitchen shot was close to optimum exposure for the lighting conditions to preserve the full range of scene values. I downloaded Rawdigger to take a look too. There is roughly an equal percentage of white clipped highlights and black clipped shadow and a minimal amount of both. The beauty of raw is the ability to push the midrange scene values where you want them in post while retaining full highlight and shadow details.

The firmware is the same in both cases, but the dog shot was at 100 ISO and the kitchen shot at 200 ISO camera settings. Changing ISO on the D16 is not like most CMOS cameras. Everything changes, analog gain and slopes, shutter clock timing, AD conversion. I haven't done formal chart tests yet but the ISO speeds were factory calibrated to a Sekonic light meter and seem to match my old Sekonic L398 readings quite well. I expose the D16 by meter, not by monitor, using the same methods I would use for film.

Thanks for turning me on to Raw digger by the way. That is going to be a very useful tool for profiling the camera.

The only way you're going to be able to judge scene linear is using a transmission wedge or similar so you can plot the samples like i did for the 700 from the raw data. These stouffler wedges aren't that expensive and so long as you can backlight without hotspots they're pretty easy to set up. You can see from the graph i created that the 700 is indeed linear, the light is exponential. Yes there is a small curve on the top and bottom but nothing that would make any real world difference.

Pushing the mid range whilst keeping black and white is just doing a gamma on the image and that's not usually what you want to be doing. The beauty of RAW is that you can photometrically change exposure because the values are scene referred. An exposure change in linear is a simple gain operation assuming a linear grading environment.

When RAW files are debayered the application doing the work needs to know what kind of encoding the DNGs are in otherwise the image produced isn't technically accurate - most higher end workflows ACES and similar rely on knowing that the source files are tied to scene referred values. If a camera manufacturer decides to stick a massive S curve on the raw output and not tell anyone that will cause headaches in post. BMD do this but supply the LUT. Arri do this but do it in LogC. Neither DBX nor the 700 do and if you think the DBX has a whacking great toe, enough to make a difference then DBX should be embedding a 1D LUT in the file. But i don't think it does, i don't have a DBX to test though so can't say for sure.

RawDigger is very useful for poking around, glad you like it. Tell Illias (the developer) that i'm trying to publicise it! But look carefully at your shadow values, that's always the issue in RAW

cheers
Paul

 

[paulcurtis]

We need an emoticon with an exploding head!

Enjoying this thread and your article although it may be a while before I'm able to fully comprehend it. Your writing style is easy to follow so I am at least getting the gist of it and hopefully I'll be taking something out of it I can use if not fully understand. Thanks.

Thank you, that's very kind of you.

cheers
Paul