"Deskew" proposed opensource Rolling shutter correction software

Anhar Miah said:
if I've misunderstood (b) then please explain further,
Click to expand...

What do you think i've been doing the last few pages? I've written *exactly* what you just said and i've explained why it's flawed, why lee doesn't understand and why you don't understand him. I'm not taking sides, just trying to be the annoying know-it-all. This is high school logic and math. Stop treating it like it's something you need to explain to people. Take the next step and consider temporal space and time shifting instead.
/matt
 
mattsand said:
What do you think i've been doing the last few pages? I've written *exactly* what you just said and i've explained why it's flawed, why lee doesn't understand and why you don't understand him. I'm not taking sides, just trying to be the annoying know-it-all. This is high school logic and math. Stop treating it like it's something you need to explain to people. Take the next step and consider temporal space and time shifting instead.
/matt
Click to expand...

thanks God someone said it
 
Ok my sincere apologies if my posts have come off sounding like I’m lecturing people, that was not my intention, perhaps we have all been speaking on different wavelengths.


Anyhow, I think I now understand what you and Lee are saying:

Pixels are delayed in time, thus shifting time in the opposite direction (or interpolating it) can correct the distortion created by the CMOS scan delay.* ?


If that is the case, then in theory I'd agree with you, in practice you will run into two problems:

(a) it will still be variable (the time shifting, due to the panning of the camera, faster pans = larger time corrections) (as I understand it.

(b) Depth, objects in the foreground will have greater distortion than those in the background, using a constant shift can;t correct this, since you need to shift pixels accordance to depth (z axis) but a 2d image has no depth information.


Again I'm sorry to anyone who feels I'm lecturing them, I just think this idea about a simple fix is very very misleading, and again if anyone does have a solution then I'm all for it, I'll be first in line to say I'm wrong and go the extra mile and code it if I can.

Anhar

*(and I did test alone these lines, but without using interpolation, basically I pulled pixels by a constant amount from the previous frame)
 
Anhar Miah said:
I just think this idea about a simple fix is very very misleading
Click to expand...


No way that is a simple fix, there is no Matte (for the ramp) option on Twixtor for the time displacement (the ideal), this fix must be written from scratch.

And still the main problem for this fix is the pixel morph itself. If the shutter is too fast the morph will distort the results.
 
I really think Lees method will work... can someone with time please try it already :)

I really did have a shot at it yesterday
 
Nicky said:
can someone with time please try it already :)
Click to expand...

I've already posted one crude time displacement test, download it, tell me what you think

anyway, look at this little example with time distortion with the proper vertical resolution (30 lines) for a temporal resolution of 30 frames



it works, but needs another workaround because of the astronomical render time and morph distortions
 
Last edited:
very nice. thank you. are you using twixtor? if there was just a way of only analyzing each frame for motion vectors once, then apply the time shift using that analysis to each line it should be much faster. presumably the most time is spent analyzing the image and calculating the motion?

as you know i develop plugins myself, but i base mine on an understanding of what needs to be done since i'm a filmmaker myself, not complex algorithms. if somebody who's into motion tracking could just write a piece of software that generates motion data for all pixels in the frame i could easily implement the rest. for this purpose i think any simple motion tracker like the ones used for stabilizers would do, maybe with some edge detection added. twixtor uses more advanced techniques to detect objects and overlapping motion, but i don't think that's really necessary for this.

/matt
 
You should be able to make an expression (or script) in ae that automatically makes precomps that only is one pixel high (one for each line in the clip) and then process twixtor on them with different data. In twixtor you can set which framenumber to show (like time remap but with frames that not exsist) that use the same slomo.

But It still going to be too slow. And another problem is that I believe that it´s going too look kind of smeary and without any sharpness. And it might be hard for twixtor to analyze something that´s only one pixel high. I think it needs more information.
 
Karsvall said:
And it might be hard for twixtor to analyze something that´s only one pixel high
Click to expand...
what i'm saying is that the analysis can be done on the entire frame, and once for the entire frame. there's no need to analyze the entire frame for every line, nor to analyze only the current line. for any given pixel if you have its accurate motion vector you can easily calculate where it was x ms ago. (actually it's the reverse transform we're after but it's the same basic idea)

/matt
 
Matt can you not access the data from motion trackers in AE (and just ignore all Y movement data) then just sum and average to get a global velocity vector?

Is there not an export feature in AE for the tracking data?

Anhar
 
huh? a global vector won't work for the very reason you've been pushing back and forth for 7 pages already. what's needed is the motion vector for each pixel in the image. and ignoring the y data would be a very bad idea too.
 
I was not entirely sure what you was after hence the suggestion,

But if your after the motion vector for every pixel then that can be done by optical flow algorithms & analysis, but thats quite a bit more complex to code, and CPU intensive.

Anhar
 
I wonder if, rather than coding a new motion tracking algorithm, someone might build this on top of some of the avisynth / virtual mod stuff. I know that there is a filter that creates tracking data and saves it as part of a two pass image stabilizer, but I forget what its called.

It might be easiest to have someone else do it if there were some form of payment or bounty... I'd be willing to contribute some cash for an AE or avisynth filter if it decreased rolling shutter artifacts to 5DM2 levels and took less than 20ghz-hours per minute of footage to process.

The theory seems sound, but I wonder how much computing power would be needed, right now with retiming to, say, 720fps.... seems I'd need a render farm.
 
mattsand said:
very nice. thank you. are you using twixtor? if there was just a way of only analyzing each frame for motion vectors once, then apply the time shift using that analysis to each line it should be much faster. presumably the most time is spent analyzing the image and calculating the motion?
/matt
Click to expand...

thanks Matt. I've pasted all Twixtor values that I used in the rock test post. Twixtor can export motion vectors for each pixel at 16bpc, take a look here

www.revisionfx.com/support/faqs/motion_vector_FAQs/motion_vector_math/
 
Matt - I have twixtor let me know if theres anything you need cos I really need this to come true.

That example looks really good Im wondering if this will actually work :) :)
 
Nicky said:
Beckspace - How are you doing that exactly?
Click to expand...


In the hard way

1) AE: Slomo on Twixtor (in any composition size, if you make a slomo 24fps >> 720fps you will be able to fragment the whole wobble in 30 horizontal micro-wobbled slices, 1440fps with 60 slices and go on... maybe there is a confusion point around 100 slices with Twixtor's motion blur at 1280x720, being very optimistic)

2) Gradient Map on Photoshop (TIFF black bottom to white up at 16bits 1280x720)

3) AE: Time Displacement**: pushed the rendered slomo (interpreted as 24fps) into a new composition; pushed the photoshop gradient to there and applied Time Displacement effect in the slomo with the gradient as a Matte; math: slomofps/24fps=X frames to be displaced (black as (-) time or white as (+) time)

4) AE: Twixtor again: speedup the displaced slomo render to the original 24fps (I don't know the best way to do this, but twixtor gives motion blur)

**I used Sapphire plugin instead of AE's built in because it can soft the slices whereas AE cant
 
You must log in or register to reply here.
Back
Top