Rolling shutters and matchmoving

[modernPrimitive]

Hi,

Just wondering if anyone here has had problems matchmoving footage from rolling shutter CMOS cams like the EX1 or Red. I know image distortion is only evident to the human eye during fast whip pans but surely even slower pans will cause enough distortion on tracking points to cause problems with solves (or am I talking crap)?

I'm thinking of getting the EX1 or the HPX170 and since the work I will be doing will be vfx intensive I think this is going to be a huge deciding factor.

[triplej96]

Rolling shutters are bad for matchmoving. Read this.

http://www.ssontech.com/phpBB2/viewtopic.php?t=478&postdays=0&postorder=asc&start=0

[modernPrimitive]

Yeah it make sense when you understand how matchmoving algorithms work and how distortion of the image can affect this. However there must be some work- around.

I don't understand why Peter Jackson used the RED on District 9 (just gone into post production). I know the movie's going to be CG heavy as the director Neil has a vfx intensive background so I wonder how they got around this? Also looking at Peter and Neil's tests done for Halo (on Youtube) I'm assuming that they' used some intensive handheld style cinematography with lots of cam movement, whip pans etc. (assuming of course I know what District 9 is all about - well I'm pretty sure of the basic storyline anyway).

[eoffermann]

I'll try to summarize as quickly as possible because I could write pages and pages on this - these notes should give you a starting point for additional research.

First: there is no workaround! You just deal with it - and you *definitely* avoid CMOS for vfx work.

The Red isn't so bad, and is so far the best CMOS camera I've seen (most are abjectly horrible: don't start wondering if maybe X or Y won't have this problem... most have it MUCH worse than the Red)

The company I work with just completed work on a spot for Nike (http://www.youtube.com/watch?v=jlXRengzZoc) shot on the Red and while the temporal lag is impossible to *really* compensate for and made it complicated in ways that would never have happened in film or CCD-based HD, the lag was minimal enough that most people would never have seen the issues our critical eyes see. Thank goodness there's such a talented team there that made it possible: but I'd strongly discourage individual artists or small boutiques from placing their hopes in a CMOS camera for a vfx-heavy spot. Last year's "Leave Nothing" spot for Nike (http://www.youtube.com/watch?v=GX_5tzwVz3I) would have been unbelievably painful on CMOS footage.

You can almost compensate for temporal lag in the comp if it's a nodal move: ie., the camera is on a tripod. You really just have to hope for the best with moving camera work. Renderers aren't currently capable of delivering temporally offset scanlines and since your image data is damaged in unpredictable ways before you ever receive it (including losing data that can't be recovered because it doesn't exist in the frame any more) ,you'll never get a flawless track.

In a RED camera, using the 2k mode, you get faster dumps of the CMOS framebuffer than you'd get at 4k resolution - which reduces the temporal lag in the resulting image. This is probably the best way to optimize RED footage for tracking later. The lag is still there, but for many shots it will be minor enough that you can cover the issues by the time it gets to comp.

[modernPrimitive]

thanks for the info eoffermann.

Out of interest have you found that matchmoving is much more difficult even with slow camera movements on the RED or is the problem generally just limited to faster movements.

Interestingly I did see someone's reel (from CGtalk) that demonstrated vertical distortion compensation on a typical nodal pan on a rolling shutter - not sure how this was done whether it was comped manually with some sort of distortion grid or something. One should then be able to apply a reversed distortion mesh to the CG elements. (kinda like some of the lens distortion correction tools available for matchmoving). Someone will probably come out with a plugin aimed at this.

[modernPrimitive]

Okay, I read some interesting info on the RED. Apparently the RED has eliminated quite a lot of skew with sofware upgrades (apparently a lot better than the EX1 - according to RED).

The new Mysterium X sensor should bring the rolling shutter down to an almost film-like skew - let's hope. Sounds like they're aiming to simulate film rather than eliminate skew entirely, though they might have a simulated "global shutter" as an optional feature on the following generation of sensors. If they can do that then there should be no problems with the RED. Will have to see if Sony and the others can do the same.

[Anhar Miah]

I've saved some bad matchmoves by "tweaking" it by hand, sadly sometimes you have to go old school.

I've also found that high shutter speeds help, (blur = loss in tracking points, i.e results in sliding etc).

I'd also recommend this:

"Matchmoving Invisible Art Camera Tracking "

http://www.amazon.com/Matchmoving-Invisible-Art-Camera-Tracking/dp/0782144039

Anhar

[USLatin]

I am curious to hear about matchmoving for Red One with the current build 17 or 18... I will need some of this work done for my feature.

In comparison to the EX1/3 the amount of skew is a whole different ballpark, but still there and visible when you run with the camera. That is not something I plan on doing, and for the VFX shots I'll pay close attention to my pan speeds.

Anyone with first hand experience?

[triplej96]

USlatin I've heard it suppose to be better with current build but haven't tried my self. I might get some Red One footage to play with soon I'll let you know when I try to matchmove it.

[USLatin]

I would need particle animation work and I wouldn't want to have to make the camera boring... besides a lot of it would be in the third act and past the point when "shit hits the fan" (gotta love those technical script writing terms) :)