Magic Lantern enables 14-bit RAW CinemaDNG on 5D

alex1 said:
Has the 55 second RAW limit be fixed? Because it really isn't that useful except for making montage or comparison videos.
Click to expand...

This is a complicated question because the RAW limit exists for different reasons, some of which have been addressed, others not.

First reason: File system file size limits (i.e. FAT32 @ ~4gb). This has been addressed. On the mk3, I think it's as simple as formatting the card with exFAT. On the mk2 (what I use), there's a new build of the RAW record module ( http://www.magiclantern.fm/forum/index.php?topic=5655.0 ) that automatically splits the RAW recording into another file once it reaches about 3.7gb. Once you're back on your computer, you combine the two files (they will have names like M0000001.RAW, M0000001.R01, M0000001.R02) into one large file that you then feed to RAW2dng. I use the cat command in Terminal on my Mac to do it (cat raw1.raw raw1.r01 raw1.r02 > raw1-combined.raw).

Second reason: Exceeding the CF card's buffer. This still happens if the resolution you choose exceeds the capabilities of your card/camera to keep up. The mk3 seems to be able to write to the CF card faster than the mk2 even with the same CF card, so it's possibly hardware limited. So depending on what settings and camera and card, you might still hit recording time limits. But the ML interface is very good at letting you preview the speeds needed for each setting and benchmarking your card to see what it can handle. Plus there is a setting now that ends the recording as soon as the first dropped frame occurs, so you don't waste time on bogus takes.

Things are changing fast but in short, yes, it's getting a lot more usable now than just a week ago.
 
alex1 said:
Has the 55 second RAW limit be fixed? Because it really isn't that useful except for making montage or comparison videos.
Click to expand...

There is no 4Gb file limit on an ex-fat formatted card on builds from May 19 onwards. I've recorded a single 1080p file that filled a 64Gb card. raw2dng can convert the files on all 3 platforms.
 
And, if you fill up your card or get corrupt .raw files (well ones with no footer) here's how to fix them:
I started this thread over on the ML forums; I did 32GB/16Gb/8GB/4GB footer examples. if you need more file sizes let me know.

http://www.magiclantern.fm/forum/index.php?topic=5732.msg41171#msg41171

For example, tonight I stuck in a freshly formatted exFAT 32GB Lexar card, disabled the 4GB limit, and let it run until it shut off "automagically". In ML's current state, since the footer wasn't written to disk (card) the file wouldn't unpack. When you dropped the .raw onto raw2dng.exe you'd get "Error: This ain't a lv_rec Raw file".
The above mentioned thread fixes that problem. If you can copy and paste, you can do it. I know nothing about HEX and figured this out on my own.

Shawn
 
Squig Ive only been popping in and out of this thread so you or other may have covered this - but stability at the client second cam to a C300 stable?

Looking for something small form factor thats light on the load, body will also fit in my C300 carry on bag etc.
 
Noel Evans said:
Squig Ive only been popping in and out of this thread so you or other may have covered this - but stability at the client second cam to a C300 stable?

Looking for something small form factor thats light on the load, body will also fit in my C300 carry on bag etc.
Click to expand...

Hey mate, I took the kids to Taronga on the weekend (vimeo pending) and the 17/5 build worked without a hitch shooting 1920x840 on a Toshiba 64Gb 1066x card (It can manage 1920x1080 reliably). I filled the card with 76 files that ran over 15min. The 17/5 build only records up to 4Gb files but the 19/5 build will record unlimited file sizes and appears to be good as far as artifacts go but I haven't given it a good workout yet. The last file will be corrupted but it sounds like sparky has come up with a fix. The biggest issue is post workflow, if you just shoot like you're shooting H.264 and nail your exposure then you can do a prores transcode from ACR/AE with no issues, it's only when you start fuc#ing with the sliders in ACR that it becomes a problem which means no flat/log looking transcode. The prores transcode takes 5x longer than H.264.
 
I was able to process the DNG frames made by the Magic Lantern software from the RAW data file in my developmental version of my free de-Bayer program after making some changes to support the 14bit packed data format. I need some questions answered to directly support the raw file the camera saves, but that should be possable. This suport is planded for my de-Bayer program version 0.09 which is not released yet but is planed to be when I get some details worked out, one of which is the Linux version release.

Someone on CML sent me some DNG frames 1920x840 and I processed those into an uncompressed AVI file, graded and set to play at 24fps, its of a street in late afternoon with a car driving down the street away from the camera. It looks as one would expect, similar to maybe being shot on the KineRAW-S35 (tm) camera looking at the uncompressed AVI. It looks like he used a HDSLR lens, so there is some softness on the leafs on the tree, maybe it would look crisper shot with a 35mm movie camera lens, but usable for making a feature film maybe if they can implement 2048x858 (maybe by using single pixel mode rather than binning, sum, or skip-pixel modes)?

He tells me he can shoot 1920x1080 now and made a black frame set for checking the FPN levels.

Right now he is shooting some test DNG to see what impact the user settings have on the analog gain applied to the sensor. Obviously you would want minimum analog gain set on the sensor for shooting raw, that is you set the camera to 4100K ISO 160, and expose the sensor at about ISO 320 to 1280, so the monitoring table will show its underexposed, then process the raw for best results in software like mine that uses 16bit LUT to avoid adding processing errors and histogram gaps etc.

One question I had was can Magic Lantern develop software that lets the monitoring table in the camera be adjusted independent of the analog gain settings in the sensor, so when there is enough light the analog gain can be set low as possable for white balance, and then the monitoring can be adjusted for the 2 to 5 stop under exposure of the actual 14bit sensor data to hold the highlights more than the H.264 recording option would allow because of H.264 shadow detail noise and block limitations.

If anyone that can answer technical questions from Magic Lantern reads this, please email me at: tempnulbox (at) yahoo (dot) com with "about Magic Lantern 14 bit RAW support" in the subject line, so I can resolve support issues for your raw file format.

You may be able to see the last test marked M19*.avi in the shared CML FTP folder, members of CML will know what that means. Note that you may not be able to play an uncompressed AVI on a MAC, it should play on a very fast PC from RAID SSD using media player or other windows programs like VirtualDub (tm) etc.
 
squig, overall very nice.. one thing I' ve noticed, some chroma clipping on the girl's hat 0.38 etc. light blue color seem go give the sensor some hard time.
How did you expose footage, and how developed, curious to know, AE to prores workflow, I assume? Thanks again for all your efforts to keep this thread alive and active!
 
OlegKalyan said:
squig, overall very nice.. one thing I' ve noticed, some chroma clipping on the girl's hat 0.38 etc. light blue color seem go give the sensor some hard time.
How did you expose footage, and how developed, curious to know, AE to prores workflow, I assume? Thanks again for all your efforts to keep this thread alive and active!
Click to expand...

The chroma is probably film convert, the Fuji stock I used is quite saturated.

I exposed to the left and still clipped the highlights in some shots.

I opened the DNGs in ACR/AE, made highlight, white balance, and shadow adjustments, applied a cinestyle type LUT, added a film convert stock in AE, rendered out to prores4444, and cut it in FCPX.
 
Tnanks again, how, do you color balance shots, while shooting?
How much is left to color balance in FCP, do you mostly do it in AE, making log kind of look first?
 
OlegKalyan said:
Tnanks again, how, do you color balance shots, while shooting?
How much is left to color balance in FCP, do you mostly do it in AE, making log kind of look first?
Click to expand...

I was chasing the kids around so no time to nail exposure and WB, just focus and framing. I mostly had WB set @ 5600. I did most of the corrections in ACR/AE but did some more work on the exposure and WB in FCPX. The giraffes needed a lot of work, I think I need an IR filter. It was a pretty rushed job, no time to perfect a non paying gig. Kholi reckons it's a bit too green, I agree. Yeah I made a log/cinestyle LUT in ACR and then chose the cinestyle profile in film convert. A proper film convert LUT will be a great time saver.
 
The Cineform option is looking really good for fast turnaround. A mac version would be nice, Dave?
 
Dan Hudgins said:
expose the sensor at about ISO 320 to 1280...
Click to expand...

When shooting RAW you should just use 100,200,400,800, 1600 etc... Those intermediate ISOs are digital pushes (like in stills mode), great for H264 because you can't adjust them much but with RAW they just take away a bit of DR.
 
jcs said:
Really fast way to convert RAW into something that can be watched (results not as good as ACR+AE, but very fast): RAW to Cineform tool. http://www.magiclantern.fm/forum/index.php?topic=5479.50
I already had Cineform NEO 3D installed and it worked OK (de-Bayered. The non-de-Bayered output was stretched).
Click to expand...

A great option for fast-turnaround projects, indeed: at over 100 frames per second, it's basically just like converting the .raw file to a stream of .dng files!"
"If you convert from your Compact Flash source, the conversion will be at read speed with very little CPU usage"

So, with something like Advanced Detail 1 debayering, how does detail look compared to the original H.264 video? It should be somewhere between that soft ugly thing we used to love but don't want anymore, and the sparp, clean beauty of ACR
http://www.similaar.com/foto/blackmagic-workflow/index.html
 
Samuel H said:
So, with something like Advanced Detail 1 debayering, how does detail look compared to the original H.264 video? It should be somewhere between that soft ugly thing we used to love but don't want anymore, and the sparp, clean beauty of ACR
http://www.similaar.com/foto/blackmagic-workflow/index.html
Click to expand...

So far the biggest part of the magic of this RAWvolution (1st in this context? (AOT food) :) ) is ACR: excellent noise reduction, sharpening, and color controls. Looking forward to more C300 style 2x oversampled Bayer sensors with simple processing to get ultra-high quality (not a typical de-Bayer method). Haven't used Resolve for de-Bayering- how does it compare to ACR?

The ultimate ACR is a (from scratch) custom physics-based racing game I wrote a few years ago: http://www.youtube.com/watch?v=ydlEEVrY3qg (physics computed on cubic curved surfaces) :)
 
JCS - I bumped your now 1 year old thread about the 5d3 having 800-900 lines of resolution. I would love to see a redux of this with the 5d3 hack...:)
 
You must log in or register to reply here.
Back
Top