"Deskew" proposed opensource Rolling shutter correction software

[Anhar Miah]

Hello fellow DVXers;

I've been thinking alot about the issue of rolling shutter (skewing artifacts) and methods that could be taken to correct/reduce them.

Naturally as already posted many times before some techinal papers on the matter on this subject has been seen already.

I decided to attempt to develop an opensource project for such a post correction utility (depending on the demand)

So here is a simulation of what is possible, the software results should be exactly the same:

Frame 12 original:

http://i42.photobucket.com/albums/e320/Anhar/skew.png

Frame 12 Corrected:

http://i42.photobucket.com/albums/e320/Anhar/corrected.png

Video:

http://www.youtube.com/watch?v=e1RY6uY_ZOs

Anhar,

[danimal84]

hey,

ive been looking into a lot for any way to correct in post the rolling shutter issue.

i came across the article that had that footage of the before and after of panning that christmas tree and read the article about the correction used to apply to fix it. but in that article if i remember corectly, while it did correct the "skew" in the christmas tree and other parts of the image in that lab, did it not do this by "skewing" the entire frame where as the geometry has been fixed to not have a skew issue, the frame has been skewed so the only way to use this would be to crop inside the frame so that you do not have the skewed edges?

correct me if im wrong, im all about trying to fix this issue but from what i remember that you would lose your working resolution and have to crop in order to use the fixed footage?

[Andrew Brinkhaus]

What would the workflow be like with this software? As we all know the D90 doesn't have a great compression ratio as it is, so furthering compression would be less than ideal... What would the output format be? Or is this something that could be applied in an NLE as a plug-in?

Regardless at this stage that looks very promising. Now we just need a de-jello'ing software. :beer:

[shinyrobot]

You should talk to Joe, from Joe's Filters. He makes incredible Final Cut Pro filters and could totally do something like this as a plug in.

http://www.joesfilters.com/

[Anhar Miah]

correct me if im wrong, im all about trying to fix this issue but from what i remember that you would lose your working resolution and have to crop in order to use the fixed footage?

If the theory is correct, it may be possible to do it crop free; I will attempt a crop free simulation tommorrow.



What would the workflow be like with this software? As we all know the D90 doesn't have a great compression ratio as it is, so furthering compression would be less than ideal... What would the output format be? Or is this something that could be applied in an NLE as a plug-in?


It will hopefully be a standalone application, you should be able to open any video file your machine supports (AVI) and save using any compression you wish. Perhaps saving the D90 to some intermediate lossless codec would be the best way, then using the same lossless codec during render

As for a NLE plugin, if anyone is willing to port it then yes, being opensource anyone can look at the code and is free to port it.

[Andrew Brinkhaus]

Looking forward to your crop free test, Anhar.

[Chris_TC]

That's a pretty significant improvement! Can this also reduce the jello or just the skew?

[Anhar Miah]

That's a pretty significant improvement! Can this also reduce the jello or just the skew?

If by "Jello" you mean the effect of "stretching" that occurs to the image as the camera is moved up and down or tilted up and down, then at present the answer is no.

However if the first project goes smoothly, then for the next version I will have a look at it and see if its possible to fix, I would imagine that it would probably involve resizing the image in the Y axis, and introducing regions of the last frame that is equal in proportion to the missing amount as produced due to the reduction.

Anhar,

[theabsurdman]

good idea to seek a software solution for this.

there already exists a filter for virtualDub which can correct rolling shutter skew: deshaker by gunnar thalin -- http://www.guthspot.se/video/deshaker.htm

i did a little test of it recently using some of kholi's wobblier footage.

before:
http://www.youtube.com/watch?v=y484jV_2IfA&fmt=6

after:
http://www.youtube.com/watch?v=ye3Ag8wTzvg&fmt=6

his filter has the option of covering the frame rotation crop by either zooming the frame or (better) filling-in the gaps using previous and succeeding frames. there are a lot of config options available. for the above example, i used the previous/next method.

perhaps you could email him and get his input on the problem.

[Anhar Miah]

Results from non crop simulated tests:

I found the results to be poor, it performed worst on very large skews, contray to the theory of using the last frame or pixels from previous frames in order to "fill" in the missing gaps, I found that the current frame worked far better, from my eyeball guestimate I would say that "alignment" seemed to be out by 0.5 to 1.0 pixels.

Addings some blur on the edges may fix this, this would have to be done in the coding.

NOTE: the thin black line on the top left shouldn't be there, this is due to having a line extra pixel border (croping the video would remove this, I didn't crop the original video accurately)

http://i42.photobucket.com/albums/e320/Anhar/NonCrop_test.png

[Thebes]

How do people shooting with film cameras with horizontal rolling shutters deal with them? I know vertical rolling shutters are less of an issue, but there are some of each, and vertical ones still induce stretching and compression, right?

IIRC, you pan slowly and blow things out of focus if you have to pan too fast... I feel the rolling shutter skew is less of an issue than the jello-vision. Aesthetically I don't like the skew, but its not horrifically offensive unless you shoot to emphasise it- unlike the wobble artifacts which turn it into instant garbage... as I understand it those can't be removed since camera motion was not constant during the frame's shutter.

[Anhar Miah]

How do people shooting with film cameras with horizontal rolling shutters deal with them? I know vertical rolling shutters are less of an issue, but there are some of each, and vertical ones still induce stretching and compression, right?

IIRC, you pan slowly and blow things out of focus if you have to pan too fast... I feel the rolling shutter skew is less of an issue than the jello-vision. Aesthetically I don't like the skew, but its not horrifically offensive unless you shoot to emphasise it- unlike the wobble artifacts which turn it into instant garbage... as I understand it those can't be removed since camera motion was not constant during the frame's shutter.


I think they used a film camera aimed at a calibrated TV tube:

http://ieba.wordpress.com/2007/10/12/rolling-shutter-pick-the-right-tool-for-the-job/

(scroll down)

[danimal84]

here are some links dealing with the correcting rolling shutter issue:

abstract: http://mpac.ee.ntu.edu.tw/Exhibition/rolling-shutter.php

PDF overview: http://140.117.166.1/eehome/ISCOM2005/SubmitPaper/UploadPapers/ISCON05_00159.pdf or view it as html here (http://74.125.45.104/search?q=cache:QMW75iQr020J:140.117.166.1/eehome/ISCOM2005/SubmitPaper/UploadPapers/ISCON05_00159.pdf+Analysis+and+Compensation+of+Rol ling+Shutter+Effect&hl=en&ct=clnk&cd=1&gl=us)

[Anhar Miah]

EDIT::

I far as I understand it, both the "Jello" AND "Skew" is due to rolling shutter, the only difference is that one effects the capture in the X axis, the other effects the capture in the Y axis.

Part of my current thinking is to utilise 2d motion tracking to generate the motioin in both X and Y, intergrated over time should give the velocity vector (or motion vector) using this it should be possible to correct both the skew and "jello" artifact, sadly cropping would be required.

[Anhar Miah]

here are some links dealing with the correcting rolling shutter issue:

abstract: http://mpac.ee.ntu.edu.tw/Exhibition/rolling-shutter.php

PDF overview: http://140.117.166.1/eehome/ISCOM2005/SubmitPaper/UploadPapers/ISCON05_00159.pdf or view it as html here (http://74.125.45.104/search?q=cache:QMW75iQr020J:140.117.166.1/eehome/ISCOM2005/SubmitPaper/UploadPapers/ISCON05_00159.pdf+Analysis+and+Compensation+of+Rol ling+Shutter+Effect&hl=en&ct=clnk&cd=1&gl=us)

Thank you thats were I got the videos from :D

[timmytimetravel]

Would the smoothcam plugin from final cut 6 address either problem to any extent? probably not...

[Daniel Aragão]

what about the stair stripping artifact, those lines are ugly...is there a way to resolve this problem?

thanks

[Anhar Miah]

Yes, anti-aliasing plugins for AE exists, one such is produced already by Algolith called AlgoSuite:


Review:

http://digitalcontentproducer.com/videoencodvd/revfeat/video_algolith_algosuite/

Company website

http://www.algolith.com/home/index.html

{Looks like they have a hardware version}

[maccesch]

I might have overlooked it, but ist there already a project site, where I can browse the code? I'm very eager to see it :-)

[Karsvall]

Why not just timeremap it with -time on the top and +time on the bottom of the frame?
There is a plugin in AE that can "splice" time like that using a gradient as map. And when using ae´s new (a kind of morph) timeremap-algoritm it might make a good result?
Just an idea....

[Nicky]

Why not just timeremap it with -time on the top and +time on the bottom of the frame?
There is a plugin in AE that can "splice" time like that using a gradient as map. And when using ae´s new (a kind of morph) timeremap-algoritm it might make a good result?
Just an idea....

I second that... really.

[Carl]

Sounds good,
any way or plugin that work direct in FCP for this?


Thanks

/ C

:thumbup:

[Artscroll]

I just found out about this thread. Anhar, please keep developing the application. I am a Canon HV20 user and this would be a superb plug-in to have.

[Anhar Miah]

Why not just timeremap it with -time on the top and +time on the bottom of the frame?
There is a plugin in AE that can "splice" time like that using a gradient as map. And when using ae´s new (a kind of morph) timeremap-algoritm it might make a good result?
Just an idea....

Sadly the problem is (once you watch frame by frame) is that the distortion is not a constant, it varies with angluar speed (camera pan) thus the corrective transforms need to be in line with "scale" of the pan, however we don't have the information about the pan, a few ideas are:

(1) Fit an accelerator sensor on the camera and record the information and use this to do the correction

(2) Video analysis, using optical flow to determin mean velocity.

(3) Manual, hand correction.

@Artscroll

Sadly I have zero experience in developing plugins for NLE/AE (but its open for anyone to give it a go), I'm a software developer, a standalone application is what this project was intended to be.

Anhar

[beckspace]

I guess that all CMOS companies are praying for a mathematician to solve the Rolling Shutter issue. is a Time Displacement artifact as Karswall said but there is no known solution to it

As you said, the present solution is to eliminate geometrically the RS artifact coming from lens POV by tracking its movements in all axis. The few Deskew plugins available estimate its movement by basic 2D tracking (works if the entire frame is focused, if not many things are moving around), but without the Z axis the wobble can get even stranger (I tried)

By the way, when someone will implement a timecode based accelerometer for 3D tracking?

And to not say about fast movements inside the frame, no program can calculate if something is distorted (skewed or crushed) without information of all axis of every moving thing inside the frame. This issue is almost hopeless


I heard that Weta didn't go for the Red because of the Rolling Shutter, because it gives serious problems with 3D tracking software. If they, or Sony, or Dalsa, even NASA, didn't overcome it, I don't know who can


Someone on another thread also suggested to use Time Displacement in AE in the same fashion as Karswall (with a gradient ramp), but it doesnt work initially because the Time Displacement effect in AE just copy and paste pixels from another frames; new pixels must be interpolated and blended inside the frame in a Twixtor way, a new frame recreated between the orginal two frames

But then it would be needed a 360 shutter, a continuous CMOS scan without time gaps for this to work flawlessly without morph artifacts; the D90 is not the case and most CMOS based cams use fast shutters to minimize the distortion --it also gives less motion blur creating that strobe Private Ryan filming--

[Lee Wilson]

Why not just timeremap it with -time on the top and +time on the bottom of the frame?
There is a plugin in AE that can "splice" time like that using a gradient as map. And when using ae´s new (a kind of morph) timeremap-algoritm it might make a good result?
Just an idea....


This was also my first idea looking at the problem.


Sadly the problem is (once you watch frame by frame) is that the distortion is not a constant, it varies with angluar speed (camera pan) thus the corrective transforms need to be in line with "scale" of the pan,

No !

The distortion is vertical and constant !!

The reset time on the CMOS is constant, the scan time is constant everything is constant !

When the camera is fixed (no movement) the distortion is still there, there is still a constant time delay between the top and bottom of the frame, but it is not detectable, for it to be seen we must swing the camera around, but it is always there and always constant (have I mentioned it s constant yet?). :cheesy:

The distortion does not 'switch on' when you move the camera.

What Karsvall said is correct (and in theory would work as a solution).

[Lee Wilson]

And to not say about fast movements inside the frame, no program can calculate if something is distorted (skewed or crushed) without information of all axis of every moving thing inside the frame. This issue is almost hopeless

Not at all, the problem is trivial (mathmetically) - it requires absolutely no calculation as the 'constant' :cheesy: distortion is a known value.


Someone on another thread also suggested to use Time Displacement in AE in the same fashion as Karswall (with a gradient ramp), but it doesnt work initially because the Time Displacement effect in AE just copy and paste pixels from another frames; new pixels must be interpolated and blended inside the frame in a Twixtor way, a new frame recreated between the orginal two frames

You first must increase temporal resolution then you can use time displacement with a gradient map.

[Barry_Green]

But the distortion isn't constant. It's only constant if your subject has no motion within the frame. But if your subject is moving left and the camera's panning right, you're going to end up with a situation that cannot be corrected for.

Furthermore, you can't even begin to try to correct for wobble. Wobble is when the direction of the camera changes within a frame. Pan left and then pan right, and you'll end up with skewing in two directions in one frame. Or tilt up and then tilt down, you'll have scrunched and stretched footage going on in the same frame.

Something like a fixed-rate pan skew corrector seem simple, but the real-world implications make it probably impossible. You might be able to correct for a few simple situations but to try to correct for all the rolling shutter artifacts would make for an astonishingly difficult task.

[Lee Wilson]

But the distortion isn't constant.

The distortion is constant, the distortion is a temporal delay between the top and the bottom of the frame.

This is visible when there is movement, but without movement it is still there.


It's only constant if your subject has no motion within the frame. But if your subject is moving left and the camera's panning right, you're going to end up with a situation that cannot be corrected for.

Of course it can ! :)


Furthermore, you can't even begin to try to correct for wobble.

There are not a whole host of disparate artifacting distortions due to the CMOS refresh rate, there is just one, a temporal delay between the top and bottom of the sensor.

Distortion = temporal delay between the top and bottom of the sensor.
Wobble = temporal delay between the top and bottom of the sensor.
Squashed image temporal delay between the top and bottom of the sensor.
Stretched image = temporal delay between the top and bottom of the sensor.

It is all the same, name it how you like, it is a constant value. Wobble will be as easy to cure as skew.

Do this thought experiment.

Instead of moving the camera around, let's suppose it is stuck on a tripod and not touched.

1) Now run through the frame carrying a vertical pole, what do we see ? Skew.

2) Now run through the frame carrying a vertical pole, but this time stop half way through run back the other way, what do we see ? 'wobble'.

Conclusion, we changed nothing on the camera, we didn't even touch it and yet we separately generated skew and then wobble, they are one and the same thing.



You might be able to correct for a few simple situations but to try to correct for all the rolling shutter artifacts would make for an astonishingly difficult task.

Nonesense ! :cheesy: There is only one rolling shutter artefact, a temporal delay between the top and bottom of the sensor.

[mattsand]

Guys, you're talking about different things. Sometimes when people can't express themselves stringently you have to guess what they mean. Of course the distortion is constant, temporally, but it varies within the frame. For someone who doesn't understand what time mapping or temporal resolution means it doesn't help to say it a hundred times. So, there's really no way of correcting the artifacts on a frame by frame basis, that's correct, but in the time domain the problem becomes trivial, also correct. The problem is "just" to increase the temporal resolution without too many artifacts. If twixtor could be controlled by a gradient map for example the problem would be solved, right? Though won't it look strange if the top and bottom will have time remapping artifacts (there will always be some) and the middle not? Perhaps worse than wobble even? /matt

[Nicky]

I hope RS could be corrected like Lee said - I wanna try slowing down jello footage with twixtor and rebuilding the vertical columns of pixels using that AE time gradient map?

Ill let u guys know how it goes if I get a chance today.

[Lee Wilson]

Guys, you're talking about different things. Sometimes when people can't express themselves stringently you have to guess what they mean. Of course the distortion is constant, temporally, but it varies within the frame.

Honestly, on my lovely cat's life, distortion does not vary within the frame.

Our perception of it may vary (swing the camera around a lot and we can see it clearly) - but believe me it is constant, it is always happening in a predictable way and at a constant rate - and this is important to undertstand when looking for a solution.

Distortion does not increase nor decrease when something goes quickly/slowly/back and forth through the frame.




For someone who doesn't understand what time mapping or temporal resolution means it doesn't help to say it a hundred times.

It's just temporal distortion :cheesy: !!! ;P




So, there's really no way of correcting the artifacts on a frame by frame basis, that's correct, but in the time domain the problem becomes trivial, also correct. The problem is "just" to increase the temporal resolution without too many artifacts. If twixtor could be controlled by a gradient map for example the problem would be solved, right?

Absolutely right !

You are spot on in saying that the artefacts from time stretch techniques is a very real issue, but the theory is sound.

You don't need to control Twixtor (or any other time stretch technique - optical flow / time warp etc etc) with a gradient map - the time displacement (using a gradient map) can be done separately (afterwards) from the time stretching.



Though won't it look strange if the top and bottom will have time remapping artifacts (there will always be some) and the middle not? Perhaps worse than wobble even? /matt

There will be various issues to address, but we are at this stage talking about a hypothetical / proof of concept and of course there will be numerous things to iron out.

[mattsand]

Honestly, on my lovely cat's life, distortion does not vary within the frame. sure it does. Depends on which domain you're looking at. Don't make the same mistake as the guy you're discussing with, just the other way around.


the time displacement (using a gradient map) can be done separately (afterwards) from the time stretching
wouldn't that mean having to increase the temporal resolution infinitely, or at least one entire extra frame per line resulting in around 20,000 fps? Sure you have the hd space? :-)
/matt

[mattsand]

There will be various issues to address, but we are at this stage talking about a hypothetical / proof of concept and of course there will be numerous things to iron out.

We are? I thought the theory was pretty solid, i need a working fix. /matt

[Anhar Miah]

No !

The distortion is vertical and constant !!

The reset time on the CMOS is constant, the scan time is constant everything is constant !

Lee with all due respect, even though the CMOS scan rate may indeed be a constant, what is not constant is the environment it moves in, thus objects move in space at different rates, in simple terms the faster an objects moves the greater the distortion, its a little more complicated than that.

If you still don't believe me then thats ok, why not do your own tests and create a simulation/application and show your results like I have done.

If you can do that I'll be the first to say I'm wrong.

Anhar

[Lee Wilson]

Context: distortion does not vary within the frame.


sure it does. Depends on which domain you're looking at.


Without wishing to traduce the conversation to "yes" - "no" - "yes" - "no" . . etc etc. . .

No ! :cheesy:


You are never looking at anything other than a CMOS chip scanning too slowly to give you a temporally consistent image - the kind of image a global shutter would render.

All this CMOS rolling shutter nomenclature is illusory - wobble, skew, squash etc etc it is all the same thing, the cure can be seen more clearly when you reduce the problem to it's basics.


When you say [distortion varies within the frame] [depending] 'on which domain you're looking at'. (apologies for the less than verbatim quote!) - can you clarify what you mean, can you spell out which 'domains' have a (temporally) variable rate of distortion ?






wouldn't that mean having to increase the temporal resolution infinitely, or at least one entire extra frame per line resulting in around 20,000 fps?

I haven't put in enough time on the maths as yet, but yes you are heading in the right direction, the temporal resolution would have to be very high - it will be a function of the vertical res and the frame rate.

And bear in mind a well written solution would only be increasing the temporal resolution for a single horizontal line of the frame at a time rather than the whole frame.

[Lee Wilson]

We are? I thought the theory was pretty solid

The theory is solid, it's application less so.


i need a working fix. /matt

24p 35mm camera. :shocked::Drogar-BigGrin(DBG):cheesy:

[Lee Wilson]

Lee with all due respect, even though the CMOS scan rate may indeed be a constant, what is not constant is the environment it moves in, thus objects move in space at different rates, in simple terms the faster an objects moves the greater the distortion, its a little more complicated than that.

No respect required, be honest/rude/frank, the solution is nearer that way.

'The environment 'it' (the CMOS sensor) moves in' is irrelevant, the camera cannot tell the different between being shaken around whilst filming or being pointed at a cinema screen that is having projected onto it some footage taken on a camera being shaken around !

"thus objects move in space at different rates" - this appears to be some kind of non-sequitur, but I will put it down to being a misunderstanding :2vrolijk_08:

The camera does not care, cannot tell, cannot judge, has no clue, cannot sense, does not see or cannot predict what is going on in the world.

It records whatever falls on it's sensor.

It records it at a constant rate and with a constant temporal (geometric) distortion.

This constant temporal distortion is read by us as skew/wobble/stretch etc etc depending on what is going on in the frame. Crucially the camera does not make this distinction, between skew and wobble, losts of distortion or no distortion - it just constantly records what it is pointed at with a constant temporal distortion from the top of the sensor to the bottom of the sensor.

Now you can call this what you wish, you may wait for a van to whizz by and shout "Skew"! - then wait for a man to run in and then out of frame to shout "wobble!" - or for someone to jump up and down and shout "Squash!" - but believe me when I say the sensor is indifferent to these apparent 'changes' - it is all just a time delay from the top to the bottom of the sensor - and that is all that needs to be addressed for a solution.



If you still don't believe me then thats ok, why not do your own tests and create a simulation/application and show your results like I have done.

I lack your skills.


If you can do that I'll be the first to say I'm wrong.

You would be the second. :cheesy:

[Anhar Miah]

it is all just a time delay from the top to the bottom of the sensor - and that is all that needs to be addressed for a solution.

I think you have misunderstood the problem, I'm speaking from experience and if it was that simple to fix these distortion I (along with everyone else) would have done it already!

:)

I'm sorry I don't have the time to go over in deatil why its not a constant (the distortion NOT the scan delay), I'm sure there are many DVX members with more knowledge on this subject matter than me can explain it far better than me to you.

Anhar

[Lee Wilson]

I think you have misunderstood the problem, I'm speaking from experience and if it was that simple to fix these distortion I (along with everyone else) would have done it already!

'Experience' you say !

Well then, now that you have played the 'experience' card I have nothing to say to that. :cheesy:

I think you might have employed some very basic logical fallacies, I have not claimed the implementation of a solution was simple, I have claimed the problem was simple and that following this simple understanding of the problem would likely yield a solution.

This is like someone explaining calculus to you and you retorting: "if this is so simple why don't you build a calculator". :shocked:


I'm sorry I don't have the time to go over in deatil why its not a constant (the distortion NOT the scan delay),


Now there is a surprise :) "I know you are wrong but I have not got the time / am not able to / you would not understand my explantion"

The scan delay is the distortion. The two are not separate. When you see wobble, squeeze, skew etc you are just watching the scan delay - always the same, at the same rate, in the same direction, with the same results, regardless at how you may wish to label or name them.

Even if the camera is still, it is still happening, but because the camera is still you cannot see it, the sensor is still reading one end of the sensor and then (over time) working its way to the other end - this does not stop when the car or person in the frame decides to stop moving - even though it looks like when the person stops the camera decides to stop distorting. It is illusory.

Which means, as much as these things may present themselves as separate entities, a solution to the scan delay is a solution to all rolling shutter artefacts (regardless of attempts to present them as discrete).

[mattsand]

can you spell out which 'domains' have a (temporally) variable rate of distortion ?
none, which is exactly the point. *you're* the one talking only about the temporal domain, while the guy you're trying to convince is locked in 2d space. am i really the only one who sees that these are two entirely different things? in the temporal domain the distortion is constant and predictable, but in 2d space it varies. simple as that. i'd almost go as far as saying that there are no artifacts in temporal space due to the rolling shutter, but that the rolling shutter manifests itself as artifacts in 2d space. i think that's pretty much what an artifact is.

/matt

[mattsand]

thus objects move in space at different rate
but not in time. they are all 1/24s older in the next frame regardless of speed.

/matt

[mattsand]

instead of discussing just think of it this way: for any given pixel p1 what we know is that it's captured x seconds after pixel p2. this relationship is constant since the shutter "rolls" at a constant speed. if we can somehow calculate where p1 was x seconds ago we can reconstruct an image where all pixels were captured at the same time. this is due to the "constant distortion" in time. however, this has to be adaptive since all pixels can be moving at different speeds, this in turn is due to the "variable" nature of the image. see? nobody's right and nobody's wrong. you're both trying to solve the exact same problem but without even acknowledging the other way of seeing the problem.

/matt

[mattsand]

a well written solution would only be increasing the temporal resolution for a single horizontal line of the frame at a time rather than the whole frame.
exactly, which is why i suggested the possibility of controlling twixtor using a gradient. i assume twixtor calculates a motion vector for each pixel in the image, so it shouldn't be very difficult to ask for a different time for each line or even each pixel. maybe it's possible for multiple instances of twixtor to use the same motion tracking data, then we could "simply" apply 720 instances and crop them to one line each. as a proof of concept maybe 10 regions with 80 pixels and some edge blur would suffice just to see if it works? i don't have twixtor so i can't try it.

/matt

[Lee Wilson]

none, which is exactly the point. *you're* the one talking only about the temporal domain, while the guy you're trying to convince is locked in 2d space. am i really the only one who sees that these are two entirely different things? in the temporal domain the distortion is constant and predictable, but in 2d space it varies. simple as that. i'd almost go as far as saying that there are no artifacts in temporal space due to the rolling shutter, but that the rolling shutter manifests itself as artifacts in 2d space. i think that's pretty much what an artifact is.

/matt


Agreed !

All these wobble, skew, lots of distortion, no distortion, squeeze - etc - artefacts are temporal epiphenomena or emergent from the temporal distortion, you only have to keep the camera still to watch them subside - but that does not mean that suddenly (because you keep the camera still) that the inherent distortion suddenly switches off.

You are viewing one thing - always - temporal distortion - the fact that we give names to things we recognise as being different does not mean they are different beyond our visual recognition.

[Lee Wilson]

if we can somehow calculate where p1 was x seconds ago we can reconstruct an image where all pixels were captured at the same time. this is due to the "constant distortion" in time. however, this has to be adaptive since all pixels can be moving at different speeds, this in turn is due to the "variable" nature of the image.


There is no need to calculate where anything was.

Each horizontal line of pixels is temporally offset from the previous by a constant amount - regardless of what is going on in the frame, regardless of whether the camera is on a tripod looking at a bowl of fruit or being thrown out of a hot air baloon.

Content is irrelevant, regardless of how tempting it is to see an image with a large skew as different from a static locked off shot with zero distortion - they are the same, subject to the same rules, being recorded on the same sensor with the same distortion.

No calculation is needed, this distortion is a known (knowable) value.

[Lee Wilson]

as a proof of concept maybe 10 regions with 80 pixels and some edge blur would suffice just to see if it works? i don't have twixtor so i can't try it.


Done it already (using timewarp in AE) - and I used a similarly crude vertical resolution like you suggest (480p 20x24 pixel slices) - it, of course, works as you would expect.

[Lee Wilson]

I think we are going around in circles, when I get the chance (maybe tonight) I will illustrate the basic points I am establish with a diagram or two, that might clarify what it is I am saying.

[Nicky]

Done it already (using timewarp in AE) - and I used a similarly crude vertical resolution like you suggest (480p 20x24 pixel slices) - it, of course, works as you would expect.

Nice Lee... Please elaborate on your "AE timewarp greyramp" method.

Im actually trying to do it too!

[Anhar Miah]

I have not claimed the implementation of a solution was simple, I have claimed the problem was simple and that following this simple understanding of the problem would likely yield a solution.

Lee, its good that your standing firm with your theory but;

Sadly the problem is not as simple as you think you understand it, here-in lies most of the confusion (on your part),

And the fact of the matter is other than just taking about theories, I have actually gone out there, tested it, done it, got the t-shirt and I'm telling you from legitmate experience (since I've at least attempted a correction) that your simple idea just simply doesn't work, in fact a constant correction was one of the first test I tried, guess what it failed.

You have a theory fine, thats totally fine, now please go out there and put it to the test, then come back with the results and tell me I'm wrong. I'll be more then happy to listen to the results, because theories are only as good as the assumptions/understanding they are built on,

and frankly without sounding dis-respectful your understanding/assumptions are simply incorrect, yes I could go in depth, give you a detailed point by point breakdown as to why the distortion are not constant, but that costs time, time I can spend doing other things.

And no its not a "get out excuse", in fact I don't need an excuse I've already shown the results.

Anhar

[Anhar Miah]

Sorry for the double post,

If you say that a "constant" correction is all that is needed to solve the distortion, what do you think would happen when you apply this constant correction to a static image?

you would end up distorting it, see the problem now?

Let me repeat my self in case you missed it, constant correction was the first test I ever did, and it failed.

Anhar

[Karsvall]

In theory, I don´t understand why it shouldn´t work.
But then I´m not an export of rolling shutters.

Does it scan whole lines from top down to the bottom? Or does it scan pixel for pixel from upper left (like you read a book)?
What´s the speed? Does it scan from top to bottom and then directly from top to bottom again so that the actual time of the bottom of the frame is aprox. the same as the top of the next frame? (It can not be so slow, can it?)

If so, you could apply this technic. I believe. At least in theory.

The problem is that you first have to slow down the clip as many times as there is lines in the clip (720 times) and then time displace it from top to bottom. And there is not a plugin in the world that can make that kind of slomo with good results. And the render would take for ever.

And yes, every frame would in some way be distorted - but It would work. I believe.

But, if the rolling shutter is faster (it has to be a lot faster than above) - you have to slomo even more...

Bye the way, Why do they call these cmos cameras progressive. They are not.

And why do you not see these artifacts in stills?

[bronxjragon]

this thread is half fact, half ego trip

[mattsand]

I could go in depth, give you a detailed point by point breakdown as to why the distortion are not constant
i believe what lee says is correct but i'd love for him, and me, to be proven wrong. can you at least give us a couple of the points from the list?

lee simply says that the first row is always captured x milliseconds before the last row, where x is constant for a given camera. it never changes, and certainly not depending on the subject, that would be magic wouldn't it?

/matt

[mattsand]

If you say that a "constant" correction is all that is needed to solve the distortion, what do you think would happen when you apply this constant correction to a static image?
you would get the same static image of course. all pixels in a static image were in the same position a few milliseconds ago so no transformation would be applied. lee is really lousy at explaining because he doesn't see that you're talking about a different domain, just as you have no idea that he does. the difference is he's talking about the domain where this problem originates and can easily be solved while you're talking about a domain where it can't.


you would end up distorting it, see the problem now?
the problem is that you think it would get distorted, it wouldn't. we're, again, talking about a constant correction in temporal space. you know what temporal space is, don't you?


Let me repeat my self in case you missed it, constant correction was the first test I ever did, and it failed.
so because you didn't know how to do it right, or even what you were doing, it can't be done? you don't even know what temporal space is, so how on earth could you have tried it? is that one of the points on your list?

/matt

[mattsand]

The problem is that you first have to slow down the clip as many times as there is lines in the clip (720 times) and then time displace it from top to bottom. And there is not a plugin in the world that can make that kind of slomo with good results. And the render would take for ever.
that's what i was talking about. you only have to find the motion vectors for each pixel. you don't have to actually do all the transformations, just one per line. the other 719 lines won't be used so why render them?

/matt

[Karsvall]

you need them all to make the whole frame. 719 lines have to be time displaced.

[Anhar Miah]

you would get the same static image of course. all pixels in a static image were in the same position a few milliseconds ago so no transformation would be applied. lee is really lousy at explaining because he doesn't see that you're talking about a different domain, just as you have no idea that he does. the difference is he's talking about the domain where this problem originates and can easily be solved while you're talking about a domain where it can't.
/matt

Perphaps I'm worse than Lee at explaining, from what I understand Lee and you are saying is:

(a) CMOS scan delay is fixed (because its scans at a fixed rate) etc, (I agree here)

Thus the logic is:

(b) If we apply a constant shift (to counteract the delay) then we can correct that pixel shift caused by the delay ? (I disagree here, since it depends on the motion of the camera, larger motion = larger distortation, thus the correction transforms have to natually be larger)

Clearly as you already mentioned, you can see that to apply a constant correction to a stationary camera would cause a distoration; in your words:

(c) "no transformation would be applied"

but then this contradicts the point (b) about having a constant correction, now if I've misunderstood (b) then please explain further,

and if your saying (b) is not constant as you yourself have highlighted in situation (c) then we have something that is not constant (in terms of correction).

Anhar

[beckspace]

wow, suddenly a lot of talking about this

meanwhile I was trying the Time Displacement approach to see if it was possible

I used, lets say, a low Time Resolution of just 30 frames (slowed down on Twixtor from 24 fps to 720fps, the ideal for 720p would be 17.280fps) and used a gradient ramp at 16 bpc and Sapphire's Time Displacement that handles maps of 16bpc and smooths between the time slices (not perfect but helps)

I used a handheld shot to test this

http://i36.tinypic.com/29e6byo.gif

as you can see a Low 'time resolution' creates horizontal waves in the displacement process. i believe that a higher temporal resolution would solve it

Of course, this process is not practical, impossible to create 17K fps even from rocks, not to say to blend it. but there is a short cut to it, must be...


EDIT: the render (48mb) comparing the Original and Time Remapped at a decent resolution >>>>>>>>>>> http://www.megaupload.com/?d=40MP7BTR

the values:

slomo to 720fps
http://i37.tinypic.com/2qko0og.png


the time displacement and gradient ramp:
http://i33.tinypic.com/29kumxe.png


Retimed to 24fps (with Motion Blur :))
http://i36.tinypic.com/309lq2w.png

[mattsand]

you need them all to make the whole frame. 719 lines have to be time displaced.

Altogether. Not for every line. There's absolutely no need to time shift the other lines when considering the current line. /matt

[mattsand]

if I've misunderstood (b) then please explain further,



What do you think i've been doing the last few pages? I've written *exactly* what you just said and i've explained why it's flawed, why lee doesn't understand and why you don't understand him. I'm not taking sides, just trying to be the annoying know-it-all. This is high school logic and math. Stop treating it like it's something you need to explain to people. Take the next step and consider temporal space and time shifting instead.
/matt

[bronxjragon]

What do you think i've been doing the last few pages? I've written *exactly* what you just said and i've explained why it's flawed, why lee doesn't understand and why you don't understand him. I'm not taking sides, just trying to be the annoying know-it-all. This is high school logic and math. Stop treating it like it's something you need to explain to people. Take the next step and consider temporal space and time shifting instead.
/matt

thanks God someone said it

[Anhar Miah]

Ok my sincere apologies if my posts have come off sounding like I’m lecturing people, that was not my intention, perhaps we have all been speaking on different wavelengths.


Anyhow, I think I now understand what you and Lee are saying:

Pixels are delayed in time, thus shifting time in the opposite direction (or interpolating it) can correct the distortion created by the CMOS scan delay.* ?


If that is the case, then in theory I'd agree with you, in practice you will run into two problems:

(a) it will still be variable (the time shifting, due to the panning of the camera, faster pans = larger time corrections) (as I understand it.

(b) Depth, objects in the foreground will have greater distortion than those in the background, using a constant shift can;t correct this, since you need to shift pixels accordance to depth (z axis) but a 2d image has no depth information.


Again I'm sorry to anyone who feels I'm lecturing them, I just think this idea about a simple fix is very very misleading, and again if anyone does have a solution then I'm all for it, I'll be first in line to say I'm wrong and go the extra mile and code it if I can.

Anhar

*(and I did test alone these lines, but without using interpolation, basically I pulled pixels by a constant amount from the previous frame)

[beckspace]

I just think this idea about a simple fix is very very misleading


No way that is a simple fix, there is no Matte (for the ramp) option on Twixtor for the time displacement (the ideal), this fix must be written from scratch.

And still the main problem for this fix is the pixel morph itself. If the shutter is too fast the morph will distort the results.

[Nicky]

I really think Lees method will work... can someone with time please try it already :)

I really did have a shot at it yesterday

[beckspace]

can someone with time please try it already :)


I've already posted one crude time displacement test, download it, tell me what you think

anyway, look at this little example with time distortion with the proper vertical resolution (30 lines) for a temporal resolution of 30 frames

http://i35.tinypic.com/2ajyxhi.gif (http://i35.tinypic.com/2ajyxhi.gif)

it works, but needs another workaround because of the astronomical render time and morph distortions

[mattsand]

very nice. thank you. are you using twixtor? if there was just a way of only analyzing each frame for motion vectors once, then apply the time shift using that analysis to each line it should be much faster. presumably the most time is spent analyzing the image and calculating the motion?

as you know i develop plugins myself, but i base mine on an understanding of what needs to be done since i'm a filmmaker myself, not complex algorithms. if somebody who's into motion tracking could just write a piece of software that generates motion data for all pixels in the frame i could easily implement the rest. for this purpose i think any simple motion tracker like the ones used for stabilizers would do, maybe with some edge detection added. twixtor uses more advanced techniques to detect objects and overlapping motion, but i don't think that's really necessary for this.

/matt

[Karsvall]

You should be able to make an expression (or script) in ae that automatically makes precomps that only is one pixel high (one for each line in the clip) and then process twixtor on them with different data. In twixtor you can set which framenumber to show (like time remap but with frames that not exsist) that use the same slomo.

But It still going to be too slow. And another problem is that I believe that it´s going too look kind of smeary and without any sharpness. And it might be hard for twixtor to analyze something that´s only one pixel high. I think it needs more information.

[mattsand]

And it might be hard for twixtor to analyze something that´s only one pixel high
what i'm saying is that the analysis can be done on the entire frame, and once for the entire frame. there's no need to analyze the entire frame for every line, nor to analyze only the current line. for any given pixel if you have its accurate motion vector you can easily calculate where it was x ms ago. (actually it's the reverse transform we're after but it's the same basic idea)

/matt

[Anhar Miah]

Matt can you not access the data from motion trackers in AE (and just ignore all Y movement data) then just sum and average to get a global velocity vector?

Is there not an export feature in AE for the tracking data?

Anhar

[mattsand]

huh? a global vector won't work for the very reason you've been pushing back and forth for 7 pages already. what's needed is the motion vector for each pixel in the image. and ignoring the y data would be a very bad idea too.

[Anhar Miah]

I was not entirely sure what you was after hence the suggestion,

But if your after the motion vector for every pixel then that can be done by optical flow algorithms & analysis, but thats quite a bit more complex to code, and CPU intensive.

Anhar

[mattsand]

I know, that's why i want somebody else to do it. :-)

[Thebes]

I wonder if, rather than coding a new motion tracking algorithm, someone might build this on top of some of the avisynth / virtual mod stuff. I know that there is a filter that creates tracking data and saves it as part of a two pass image stabilizer, but I forget what its called.

It might be easiest to have someone else do it if there were some form of payment or bounty... I'd be willing to contribute some cash for an AE or avisynth filter if it decreased rolling shutter artifacts to 5DM2 levels and took less than 20ghz-hours per minute of footage to process.

The theory seems sound, but I wonder how much computing power would be needed, right now with retiming to, say, 720fps.... seems I'd need a render farm.

[beckspace]

very nice. thank you. are you using twixtor? if there was just a way of only analyzing each frame for motion vectors once, then apply the time shift using that analysis to each line it should be much faster. presumably the most time is spent analyzing the image and calculating the motion?
/matt

thanks Matt. I've pasted all Twixtor values that I used in the rock test post. Twixtor can export motion vectors for each pixel at 16bpc, take a look here

www.revisionfx.com/support/faqs/motion_vector_FAQs/motion_vector_math/ (http://www.revisionfx.com/support/faqs/motion_vector_FAQs/motion_vector_math/)

[mattsand]

Very cool. I'm on it. Anybody got a twixtor licence to donate for the purpose? Or maybe the trial works?

[Nicky]

Matt - I have twixtor let me know if theres anything you need cos I really need this to come true.

That example looks really good Im wondering if this will actually work :) :)

[Nicky]

Beckspace - How are you doing that exactly?

[beckspace]

Beckspace - How are you doing that exactly?


In the hard way

1) AE: Slomo on Twixtor (in any composition size, if you make a slomo 24fps >> 720fps you will be able to fragment the whole wobble in 30 horizontal micro-wobbled slices, 1440fps with 60 slices and go on... maybe there is a confusion point around 100 slices with Twixtor's motion blur at 1280x720, being very optimistic)

2) Gradient Map on Photoshop (TIFF black bottom to white up at 16bits 1280x720)

3) AE: Time Displacement**: pushed the rendered slomo (interpreted as 24fps) into a new composition; pushed the photoshop gradient to there and applied Time Displacement effect in the slomo with the gradient as a Matte; math: slomofps/24fps=X frames to be displaced (black as (-) time or white as (+) time)

4) AE: Twixtor again: speedup the displaced slomo render to the original 24fps (I don't know the best way to do this, but twixtor gives motion blur)

**I used Sapphire plugin instead of AE's built in because it can soft the slices whereas AE cant

[Car3o]

yeah, i'd just rather keep the camera still than have to do all that.

[beckspace]

it was just a test to see if it was possible, not a workflow

But even on a tripod people move up and down, cars get distorted, this fix would not only resolve the wobble, would resolve all other inframe rolling shutter distortions too. It wasnt possible until this idea came along

[mattsand]

Twixtor can export motion vectors for each pixel
how do i do that? maybe it's a limitation of the fxplug demo version i'm using but i can only find a way to import motion vectors from a third party, not export them...

/matt

[mattsand]

it says it only works in host apps that support 16 bits but i'm using motion which does, and it's set up that way too...

/matt

[xxxxx1]

so just to clarify, what's happening here is that you:

1) identify the fixed formula giving CMOS delay effects for each pixel

then, for each frame, you:

2) retrieve the coordinate change for each pixel since previous frame from the twixtor-calculated motion vector
3) apply the fixed transformation for this pixel location based on 1)

-- (then the next frame based on the corrected values?)

so the hard work is in 2), being done by the twixtor motion vector calculations (via block motion compensation?), and the accuracy depends on how well they cope with the pre-correction CMOS deformation when following shapes?

thus, we could experiment with various motion compensation algorithms (so long as we could extract the motion vectors per-pixel) and expect to see various degrees of efficacy?

[mattsand]

Exactly. I think tracking each point one frame further to calculate acceleration then applying a bilinear correction would be fine. The time delay is fixed and only has to be calculated once. The hard part is probably where to find the missing pixels at the edges during pans and when objects move across backgrounds (blending? Forward compensation?). Not even twixtor's motion compensator succeeds with that in all situations. /matt

[Karsvall]

It´s worse than that...

If the cam pans in one direction and some objects are moving in another direction (that matt wrote) it´s going to look like crap. Twixtor can´t fix that automatically, you have to give twixtor mattes and stuff like that so it knows what is what. And if you have to fix that for each clip it´s not fun anymore.
And twixtors frames is not allways perfect :-) Sometimes you can slo things alot and with great results but sometimes not at all. You get wobbly smeary edges. Remember "kais power tools" :-) ?

What I´m saying is that all this work, but for now (sadly) - just in theory.

[mattsand]

i'm starting to wonder if the best bet isn't simply to use a global correction, stabilizer style, optionally letting it vary linearly from top to bottom. after all the main artifact we're after is the jello, and that's caused by camera movement not subject movement. deshaker for virtualdub does a great job of exporting global motion vectors in a simple format, and it even logs the difference due to a rolling shutter. sure, this will seriously screw up the geometry of objects that move across the frame in a different direction from the pan, but then again isn't that already the case?

/matt

[xxxxx1]

probably worth the effort to try all options. sensor cycles are 18-24mo, and it would require substantial engineering to add many more read-out channels or adopt a live MOS/global shutter approach; I doubt it could be fixed by the next revision, unless they anticipated this level of interest.

[beckspace]

maybe is a stupid question, but why there are no Rolling Shutter artifacts while taking photos?

the SLR photographic shutter is mechanical but I'm guessing that's just part of explanation, because whatever the exposition is there are no CMOS scanning artifacts

Apart of the mechanical vs electrical shutter, there are different CMOS behaviors between Photo and Video?

[mattsand]

an electronic shutter works by resetting the sensor, line by line, waiting the required shutter time, then reading it out, line by line. since you can only read one line at a time the next line is still being exposed while you read the current one. the cmos retains the image after it's been exposed though, so you with a mechanical shutter you reset the sensor, open the shutter, close it, then read it out and voila, no rolling shutter. (mechanical shutters roll too though, just much much faster)

/matt

[Walterson]

I agree with Lee that it is a constant delay and the difference in the skew has to do with the speed at which objects pass the censor either running by a static camera, or static objects in front of a quick pan, or both camera and object in motion.

However, this can not be fixed with a time displacement filter because the image information is not present in the previous frames. The censor delay is obviously very miniscule amounts of time. And by telling the computer to take the bottom part of the previous frames does not mean it will line up and create a non-skewed frame. We are only shooting 24 or 30 fps in most cases not 720fps so that we can have a 720 line displacement for each frame.

I loaded up a clip in FCP and just tried to make a skewed tree trunk straight again by using mattes and layering the same clip on top of itself over and over. The reality is that by displacing (even every line of 720 lines) simply cannot work because the information is not there. The previous frames are just more skewed imagery.

The only way to fix a rolling shutter problem would be by a faster sensor or a complicated bit of software that can somehow analyze and adjust the skew of each frame and that sounds just about impossible.

Sorry, I am not a programmer, but I was intrigued by this older thread as I am about to get a D90 and was looking into stuff about the rolling shutters.

i enjoyed reading the thread, thanks guys.