I second that... really.
"Deskew" proposed opensource Rolling shutter correction software
- Thread starter Anhar_Miah
- Start date
I second that... really.
Anhar_Miah
Veteran
Sadly the problem is (once you watch frame by frame) is that the distortion is not a constant, it varies with angluar speed (camera pan) thus the corrective transforms need to be in line with "scale" of the pan, however we don't have the information about the pan, a few ideas are:
(1) Fit an accelerator sensor on the camera and record the information and use this to do the correction
(2) Video analysis, using optical flow to determin mean velocity.
(3) Manual, hand correction.
@Artscroll
Sadly I have zero experience in developing plugins for NLE/AE (but its open for anyone to give it a go), I'm a software developer, a standalone application is what this project was intended to be.
Anhar
I guess that all CMOS companies are praying for a mathematician to solve the Rolling Shutter issue. is a Time Displacement artifact as Karswall said but there is no known solution to it
As you said, the present solution is to eliminate geometrically the RS artifact coming from lens POV by tracking its movements in all axis. The few Deskew plugins available estimate its movement by basic 2D tracking (works if the entire frame is focused, if not many things are moving around), but without the Z axis the wobble can get even stranger (I tried)
By the way, when someone will implement a timecode based accelerometer for 3D tracking?
And to not say about fast movements inside the frame, no program can calculate if something is distorted (skewed or crushed) without information of all axis of every moving thing inside the frame. This issue is almost hopeless
I heard that Weta didn't go for the Red because of the Rolling Shutter, because it gives serious problems with 3D tracking software. If they, or Sony, or Dalsa, even NASA, didn't overcome it, I don't know who can
Someone on another thread also suggested to use Time Displacement in AE in the same fashion as Karswall (with a gradient ramp), but it doesnt work initially because the Time Displacement effect in AE just copy and paste pixels from another frames; new pixels must be interpolated and blended inside the frame in a Twixtor way, a new frame recreated between the orginal two frames
But then it would be needed a 360 shutter, a continuous CMOS scan without time gaps for this to work flawlessly without morph artifacts; the D90 is not the case and most CMOS based cams use fast shutters to minimize the distortion --it also gives less motion blur creating that strobe Private Ryan filming--
As you said, the present solution is to eliminate geometrically the RS artifact coming from lens POV by tracking its movements in all axis. The few Deskew plugins available estimate its movement by basic 2D tracking (works if the entire frame is focused, if not many things are moving around), but without the Z axis the wobble can get even stranger (I tried)
By the way, when someone will implement a timecode based accelerometer for 3D tracking?
And to not say about fast movements inside the frame, no program can calculate if something is distorted (skewed or crushed) without information of all axis of every moving thing inside the frame. This issue is almost hopeless
I heard that Weta didn't go for the Red because of the Rolling Shutter, because it gives serious problems with 3D tracking software. If they, or Sony, or Dalsa, even NASA, didn't overcome it, I don't know who can
Someone on another thread also suggested to use Time Displacement in AE in the same fashion as Karswall (with a gradient ramp), but it doesnt work initially because the Time Displacement effect in AE just copy and paste pixels from another frames; new pixels must be interpolated and blended inside the frame in a Twixtor way, a new frame recreated between the orginal two frames
But then it would be needed a 360 shutter, a continuous CMOS scan without time gaps for this to work flawlessly without morph artifacts; the D90 is not the case and most CMOS based cams use fast shutters to minimize the distortion --it also gives less motion blur creating that strobe Private Ryan filming--
Lee Wilson
Veteran
This was also my first idea looking at the problem.
No !
The distortion is vertical and constant !!
The reset time on the CMOS is constant, the scan time is constant everything is constant !
When the camera is fixed (no movement) the distortion is still there, there is still a constant time delay between the top and bottom of the frame, but it is not detectable, for it to be seen we must swing the camera around, but it is always there and always constant (have I mentioned it s constant yet?). :cheesy:
The distortion does not 'switch on' when you move the camera.
What Karsvall said is correct (and in theory would work as a solution).
Lee Wilson
Veteran
Not at all, the problem is trivial (mathmetically) - it requires absolutely no calculation as the 'constant' :cheesy: distortion is a known value.
You first must increase temporal resolution then you can use time displacement with a gradient map.
Barry_Green
Moderator
But the distortion isn't constant. It's only constant if your subject has no motion within the frame. But if your subject is moving left and the camera's panning right, you're going to end up with a situation that cannot be corrected for.
Furthermore, you can't even begin to try to correct for wobble. Wobble is when the direction of the camera changes within a frame. Pan left and then pan right, and you'll end up with skewing in two directions in one frame. Or tilt up and then tilt down, you'll have scrunched and stretched footage going on in the same frame.
Something like a fixed-rate pan skew corrector seem simple, but the real-world implications make it probably impossible. You might be able to correct for a few simple situations but to try to correct for all the rolling shutter artifacts would make for an astonishingly difficult task.
Furthermore, you can't even begin to try to correct for wobble. Wobble is when the direction of the camera changes within a frame. Pan left and then pan right, and you'll end up with skewing in two directions in one frame. Or tilt up and then tilt down, you'll have scrunched and stretched footage going on in the same frame.
Something like a fixed-rate pan skew corrector seem simple, but the real-world implications make it probably impossible. You might be able to correct for a few simple situations but to try to correct for all the rolling shutter artifacts would make for an astonishingly difficult task.
Lee Wilson
Veteran
The distortion is constant, the distortion is a temporal delay between the top and the bottom of the frame.
This is visible when there is movement, but without movement it is still there.
Of course it can !
There are not a whole host of disparate artifacting distortions due to the CMOS refresh rate, there is just one, a temporal delay between the top and bottom of the sensor.
Distortion = temporal delay between the top and bottom of the sensor.
Wobble = temporal delay between the top and bottom of the sensor.
Squashed image temporal delay between the top and bottom of the sensor.
Stretched image = temporal delay between the top and bottom of the sensor.
It is all the same, name it how you like, it is a constant value. Wobble will be as easy to cure as skew.
Do this thought experiment.
Instead of moving the camera around, let's suppose it is stuck on a tripod and not touched.
1) Now run through the frame carrying a vertical pole, what do we see ? Skew.
2) Now run through the frame carrying a vertical pole, but this time stop half way through run back the other way, what do we see ? 'wobble'.
Conclusion, we changed nothing on the camera, we didn't even touch it and yet we separately generated skew and then wobble, they are one and the same thing.
Nonesense ! :cheesy: There is only one rolling shutter artefact, a temporal delay between the top and bottom of the sensor.
Last edited:
Guys, you're talking about different things. Sometimes when people can't express themselves stringently you have to guess what they mean. Of course the distortion is constant, temporally, but it varies within the frame. For someone who doesn't understand what time mapping or temporal resolution means it doesn't help to say it a hundred times. So, there's really no way of correcting the artifacts on a frame by frame basis, that's correct, but in the time domain the problem becomes trivial, also correct. The problem is "just" to increase the temporal resolution without too many artifacts. If twixtor could be controlled by a gradient map for example the problem would be solved, right? Though won't it look strange if the top and bottom will have time remapping artifacts (there will always be some) and the middle not? Perhaps worse than wobble even? /matt
Lee Wilson
Veteran
Honestly, on my lovely cat's life, distortion does not vary within the frame.
Our perception of it may vary (swing the camera around a lot and we can see it clearly) - but believe me it is constant, it is always happening in a predictable way and at a constant rate - and this is important to undertstand when looking for a solution.
Distortion does not increase nor decrease when something goes quickly/slowly/back and forth through the frame.
It's just temporal distortion :cheesy: !!! ;P
Absolutely right !
You are spot on in saying that the artefacts from time stretch techniques is a very real issue, but the theory is sound.
You don't need to control Twixtor (or any other time stretch technique - optical flow / time warp etc etc) with a gradient map - the time displacement (using a gradient map) can be done separately (afterwards) from the time stretching.
There will be various issues to address, but we are at this stage talking about a hypothetical / proof of concept and of course there will be numerous things to iron out.
sure it does. Depends on which domain you're looking at. Don't make the same mistake as the guy you're discussing with, just the other way around.
wouldn't that mean having to increase the temporal resolution infinitely, or at least one entire extra frame per line resulting in around 20,000 fps? Sure you have the hd space?
/matt
We are? I thought the theory was pretty solid, i need a working fix. /matt
Anhar_Miah
Veteran
Lee with all due respect, even though the CMOS scan rate may indeed be a constant, what is not constant is the environment it moves in, thus objects move in space at different rates, in simple terms the faster an objects moves the greater the distortion, its a little more complicated than that.
If you still don't believe me then thats ok, why not do your own tests and create a simulation/application and show your results like I have done.
If you can do that I'll be the first to say I'm wrong.
Anhar
Lee Wilson
Veteran
Context: distortion does not vary within the frame.
Without wishing to traduce the conversation to "yes" - "no" - "yes" - "no" . . etc etc. . .
No ! :cheesy:
You are never looking at anything other than a CMOS chip scanning too slowly to give you a temporally consistent image - the kind of image a global shutter would render.
All this CMOS rolling shutter nomenclature is illusory - wobble, skew, squash etc etc it is all the same thing, the cure can be seen more clearly when you reduce the problem to it's basics.
When you say [distortion varies within the frame] [depending] 'on which domain you're looking at'. (apologies for the less than verbatim quote!) - can you clarify what you mean, can you spell out which 'domains' have a (temporally) variable rate of distortion ?
I haven't put in enough time on the maths as yet, but yes you are heading in the right direction, the temporal resolution would have to be very high - it will be a function of the vertical res and the frame rate.
And bear in mind a well written solution would only be increasing the temporal resolution for a single horizontal line of the frame at a time rather than the whole frame.
Without wishing to traduce the conversation to "yes" - "no" - "yes" - "no" . . etc etc. . .
No ! :cheesy:
You are never looking at anything other than a CMOS chip scanning too slowly to give you a temporally consistent image - the kind of image a global shutter would render.
All this CMOS rolling shutter nomenclature is illusory - wobble, skew, squash etc etc it is all the same thing, the cure can be seen more clearly when you reduce the problem to it's basics.
When you say [distortion varies within the frame] [depending] 'on which domain you're looking at'. (apologies for the less than verbatim quote!) - can you clarify what you mean, can you spell out which 'domains' have a (temporally) variable rate of distortion ?
I haven't put in enough time on the maths as yet, but yes you are heading in the right direction, the temporal resolution would have to be very high - it will be a function of the vertical res and the frame rate.
And bear in mind a well written solution would only be increasing the temporal resolution for a single horizontal line of the frame at a time rather than the whole frame.
Last edited:
Lee Wilson
Veteran
The theory is solid, it's application less so.
24p 35mm camera. :shocked:
rogar-BigGrin(DBG):cheesy:Lee Wilson
Veteran
No respect required, be honest/rude/frank, the solution is nearer that way.
'The environment 'it' (the CMOS sensor) moves in' is irrelevant, the camera cannot tell the different between being shaken around whilst filming or being pointed at a cinema screen that is having projected onto it some footage taken on a camera being shaken around !
"thus objects move in space at different rates" - this appears to be some kind of non-sequitur, but I will put it down to being a misunderstanding :2vrolijk_08:
The camera does not care, cannot tell, cannot judge, has no clue, cannot sense, does not see or cannot predict what is going on in the world.
It records whatever falls on it's sensor.
It records it at a constant rate and with a constant temporal (geometric) distortion.
This constant temporal distortion is read by us as skew/wobble/stretch etc etc depending on what is going on in the frame. Crucially the camera does not make this distinction, between skew and wobble, losts of distortion or no distortion - it just constantly records what it is pointed at with a constant temporal distortion from the top of the sensor to the bottom of the sensor.
Now you can call this what you wish, you may wait for a van to whizz by and shout "Skew"! - then wait for a man to run in and then out of frame to shout "wobble!" - or for someone to jump up and down and shout "Squash!" - but believe me when I say the sensor is indifferent to these apparent 'changes' - it is all just a time delay from the top to the bottom of the sensor - and that is all that needs to be addressed for a solution.
I lack your skills.
You would be the second. :cheesy:
Anhar_Miah
Veteran
I think you have misunderstood the problem, I'm speaking from experience and if it was that simple to fix these distortion I (along with everyone else) would have done it already!
I'm sorry I don't have the time to go over in deatil why its not a constant (the distortion NOT the scan delay), I'm sure there are many DVX members with more knowledge on this subject matter than me can explain it far better than me to you.
Anhar
Lee Wilson
Veteran
'Experience' you say !
Well then, now that you have played the 'experience' card I have nothing to say to that. :cheesy:
I think you might have employed some very basic logical fallacies, I have not claimed the implementation of a solution was simple, I have claimed the problem was simple and that following this simple understanding of the problem would likely yield a solution.
This is like someone explaining calculus to you and you retorting: "if this is so simple why don't you build a calculator". :shocked:
Now there is a surprise
"I know you are wrong but I have not got the time / am not able to / you would not understand my explantion"The scan delay is the distortion. The two are not separate. When you see wobble, squeeze, skew etc you are just watching the scan delay - always the same, at the same rate, in the same direction, with the same results, regardless at how you may wish to label or name them.
Even if the camera is still, it is still happening, but because the camera is still you cannot see it, the sensor is still reading one end of the sensor and then (over time) working its way to the other end - this does not stop when the car or person in the frame decides to stop moving - even though it looks like when the person stops the camera decides to stop distorting. It is illusory.
Which means, as much as these things may present themselves as separate entities, a solution to the scan delay is a solution to all rolling shutter artefacts (regardless of attempts to present them as discrete).
Last edited: