"Deskew" proposed opensource Rolling shutter correction software

Anhar_Miah

Anhar_Miah

Veteran
Hello fellow DVXers;

I've been thinking alot about the issue of rolling shutter (skewing artifacts) and methods that could be taken to correct/reduce them.

Naturally as already posted many times before some techinal papers on the matter on this subject has been seen already.

I decided to attempt to develop an opensource project for such a post correction utility (depending on the demand)

So here is a simulation of what is possible, the software results should be exactly the same:

Frame 12 original:

skew.png


Frame 12 Corrected:

corrected.png


Video:

http://www.youtube.com/watch?v=e1RY6uY_ZOs

Anhar,
 
hey,

ive been looking into a lot for any way to correct in post the rolling shutter issue.

i came across the article that had that footage of the before and after of panning that christmas tree and read the article about the correction used to apply to fix it. but in that article if i remember corectly, while it did correct the "skew" in the christmas tree and other parts of the image in that lab, did it not do this by "skewing" the entire frame where as the geometry has been fixed to not have a skew issue, the frame has been skewed so the only way to use this would be to crop inside the frame so that you do not have the skewed edges?

correct me if im wrong, im all about trying to fix this issue but from what i remember that you would lose your working resolution and have to crop in order to use the fixed footage?
 
What would the workflow be like with this software? As we all know the D90 doesn't have a great compression ratio as it is, so furthering compression would be less than ideal... What would the output format be? Or is this something that could be applied in an NLE as a plug-in?

Regardless at this stage that looks very promising. Now we just need a de-jello'ing software. :beer:
 
danimal84 said:
correct me if im wrong, im all about trying to fix this issue but from what i remember that you would lose your working resolution and have to crop in order to use the fixed footage?
Click to expand...

If the theory is correct, it may be possible to do it crop free; I will attempt a crop free simulation tommorrow.


Andrew Brinkhaus said:
What would the workflow be like with this software? As we all know the D90 doesn't have a great compression ratio as it is, so furthering compression would be less than ideal... What would the output format be? Or is this something that could be applied in an NLE as a plug-in?
Click to expand...


It will hopefully be a standalone application, you should be able to open any video file your machine supports (AVI) and save using any compression you wish. Perhaps saving the D90 to some intermediate lossless codec would be the best way, then using the same lossless codec during render

As for a NLE plugin, if anyone is willing to port it then yes, being opensource anyone can look at the code and is free to port it.
 
Chris_TC said:
That's a pretty significant improvement! Can this also reduce the jello or just the skew?
Click to expand...

If by "Jello" you mean the effect of "stretching" that occurs to the image as the camera is moved up and down or tilted up and down, then at present the answer is no.

However if the first project goes smoothly, then for the next version I will have a look at it and see if its possible to fix, I would imagine that it would probably involve resizing the image in the Y axis, and introducing regions of the last frame that is equal in proportion to the missing amount as produced due to the reduction.

Anhar,
 
good idea to seek a software solution for this.

there already exists a filter for virtualDub which can correct rolling shutter skew: deshaker by gunnar thalin -- http://www.guthspot.se/video/deshaker.htm

i did a little test of it recently using some of kholi's wobblier footage.

before:
http://www.youtube.com/watch?v=y484jV_2IfA&fmt=6

after:
http://www.youtube.com/watch?v=ye3Ag8wTzvg&fmt=6

his filter has the option of covering the frame rotation crop by either zooming the frame or (better) filling-in the gaps using previous and succeeding frames. there are a lot of config options available. for the above example, i used the previous/next method.

perhaps you could email him and get his input on the problem.
 
Results from non crop simulated tests:

I found the results to be poor, it performed worst on very large skews, contray to the theory of using the last frame or pixels from previous frames in order to "fill" in the missing gaps, I found that the current frame worked far better, from my eyeball guestimate I would say that "alignment" seemed to be out by 0.5 to 1.0 pixels.

Addings some blur on the edges may fix this, this would have to be done in the coding.

NOTE: the thin black line on the top left shouldn't be there, this is due to having a line extra pixel border (croping the video would remove this, I didn't crop the original video accurately)

NonCrop_test.png
 
How do people shooting with film cameras with horizontal rolling shutters deal with them? I know vertical rolling shutters are less of an issue, but there are some of each, and vertical ones still induce stretching and compression, right?

IIRC, you pan slowly and blow things out of focus if you have to pan too fast... I feel the rolling shutter skew is less of an issue than the jello-vision. Aesthetically I don't like the skew, but its not horrifically offensive unless you shoot to emphasise it- unlike the wobble artifacts which turn it into instant garbage... as I understand it those can't be removed since camera motion was not constant during the frame's shutter.
 
Thebes said:
How do people shooting with film cameras with horizontal rolling shutters deal with them? I know vertical rolling shutters are less of an issue, but there are some of each, and vertical ones still induce stretching and compression, right?

IIRC, you pan slowly and blow things out of focus if you have to pan too fast... I feel the rolling shutter skew is less of an issue than the jello-vision. Aesthetically I don't like the skew, but its not horrifically offensive unless you shoot to emphasise it- unlike the wobble artifacts which turn it into instant garbage... as I understand it those can't be removed since camera motion was not constant during the frame's shutter.
Click to expand...


I think they used a film camera aimed at a calibrated TV tube:

http://ieba.wordpress.com/2007/10/12/rolling-shutter-pick-the-right-tool-for-the-job/

(scroll down)
 
EDIT::

I far as I understand it, both the "Jello" AND "Skew" is due to rolling shutter, the only difference is that one effects the capture in the X axis, the other effects the capture in the Y axis.

Part of my current thinking is to utilise 2d motion tracking to generate the motioin in both X and Y, intergrated over time should give the velocity vector (or motion vector) using this it should be possible to correct both the skew and "jello" artifact, sadly cropping would be required.
 
I might have overlooked it, but ist there already a project site, where I can browse the code? I'm very eager to see it :-)
 
Why not just timeremap it with -time on the top and +time on the bottom of the frame?
There is a plugin in AE that can "splice" time like that using a gradient as map. And when using ae´s new (a kind of morph) timeremap-algoritm it might make a good result?
Just an idea....
 
You must log in or register to reply here.
Back
Top