"Deskew" proposed opensource Rolling shutter correction software

[mattsand]

can you spell out which 'domains' have a (temporally) variable rate of distortion ?

none, which is exactly the point. *you're* the one talking only about the temporal domain, while the guy you're trying to convince is locked in 2d space. am i really the only one who sees that these are two entirely different things? in the temporal domain the distortion is constant and predictable, but in 2d space it varies. simple as that. i'd almost go as far as saying that there are no artifacts in temporal space due to the rolling shutter, but that the rolling shutter manifests itself as artifacts in 2d space. i think that's pretty much what an artifact is.

/matt

 

[mattsand]

thus objects move in space at different rate

but not in time. they are all 1/24s older in the next frame regardless of speed.

/matt

 

[mattsand]

instead of discussing just think of it this way: for any given pixel p1 what we know is that it's captured x seconds after pixel p2. this relationship is constant since the shutter "rolls" at a constant speed. if we can somehow calculate where p1 was x seconds ago we can reconstruct an image where all pixels were captured at the same time. this is due to the "constant distortion" in time. however, this has to be adaptive since all pixels can be moving at different speeds, this in turn is due to the "variable" nature of the image. see? nobody's right and nobody's wrong. you're both trying to solve the exact same problem but without even acknowledging the other way of seeing the problem.

/matt

 

[mattsand]

a well written solution would only be increasing the temporal resolution for a single horizontal line of the frame at a time rather than the whole frame.

exactly, which is why i suggested the possibility of controlling twixtor using a gradient. i assume twixtor calculates a motion vector for each pixel in the image, so it shouldn't be very difficult to ask for a different time for each line or even each pixel. maybe it's possible for multiple instances of twixtor to use the same motion tracking data, then we could "simply" apply 720 instances and crop them to one line each. as a proof of concept maybe 10 regions with 80 pixels and some edge blur would suffice just to see if it works? i don't have twixtor so i can't try it.

/matt

 

[Lee Wilson]

none, which is exactly the point. *you're* the one talking only about the temporal domain, while the guy you're trying to convince is locked in 2d space. am i really the only one who sees that these are two entirely different things? in the temporal domain the distortion is constant and predictable, but in 2d space it varies. simple as that. i'd almost go as far as saying that there are no artifacts in temporal space due to the rolling shutter, but that the rolling shutter manifests itself as artifacts in 2d space. i think that's pretty much what an artifact is.

/matt

Agreed !

All these wobble, skew, lots of distortion, no distortion, squeeze - etc - artefacts are temporal epiphenomena or emergent from the temporal distortion, you only have to keep the camera still to watch them subside - but that does not mean that suddenly (because you keep the camera still) that the inherent distortion suddenly switches off.

You are viewing one thing - always - temporal distortion - the fact that we give names to things we recognise as being different does not mean they are different beyond our visual recognition.

 

[Lee Wilson]

if we can somehow calculate where p1 was x seconds ago we can reconstruct an image where all pixels were captured at the same time. this is due to the "constant distortion" in time. however, this has to be adaptive since all pixels can be moving at different speeds, this in turn is due to the "variable" nature of the image.

There is no need to calculate where anything was.

Each horizontal line of pixels is temporally offset from the previous by a constant amount - regardless of what is going on in the frame, regardless of whether the camera is on a tripod looking at a bowl of fruit or being thrown out of a hot air baloon.

Content is irrelevant, regardless of how tempting it is to see an image with a large skew as different from a static locked off shot with zero distortion - they are the same, subject to the same rules, being recorded on the same sensor with the same distortion.

No calculation is needed, this distortion is a known (knowable) value.

 

[Lee Wilson]

as a proof of concept maybe 10 regions with 80 pixels and some edge blur would suffice just to see if it works? i don't have twixtor so i can't try it.

Done it already (using timewarp in AE) - and I used a similarly crude vertical resolution like you suggest (480p 20x24 pixel slices) - it, of course, works as you would expect.

 

[Lee Wilson]

I think we are going around in circles, when I get the chance (maybe tonight) I will illustrate the basic points I am establish with a diagram or two, that might clarify what it is I am saying.

 

[Nicky]

Done it already (using timewarp in AE) - and I used a similarly crude vertical resolution like you suggest (480p 20x24 pixel slices) - it, of course, works as you would expect.

Nice Lee... Please elaborate on your "AE timewarp greyramp" method.

Im actually trying to do it too!

 

[Anhar_Miah]

I have not claimed the implementation of a solution was simple, I have claimed the problem was simple and that following this simple understanding of the problem would likely yield a solution.

Lee, its good that your standing firm with your theory but;

Sadly the problem is not as simple as you think you understand it, here-in lies most of the confusion (on your part),

And the fact of the matter is other than just taking about theories, I have actually gone out there, tested it, done it, got the t-shirt and I'm telling you from legitmate experience (since I've at least attempted a correction) that your simple idea just simply doesn't work, in fact a constant correction was one of the first test I tried, guess what it failed.

You have a theory fine, thats totally fine, now please go out there and put it to the test, then come back with the results and tell me I'm wrong. I'll be more then happy to listen to the results, because theories are only as good as the assumptions/understanding they are built on,

and frankly without sounding dis-respectful your understanding/assumptions are simply incorrect, yes I could go in depth, give you a detailed point by point breakdown as to why the distortion are not constant, but that costs time, time I can spend doing other things.

And no its not a "get out excuse", in fact I don't need an excuse I've already shown the results.

Anhar

 

[Anhar_Miah]

Sorry for the double post,

If you say that a "constant" correction is all that is needed to solve the distortion, what do you think would happen when you apply this constant correction to a static image?

you would end up distorting it, see the problem now?

Let me repeat my self in case you missed it, constant correction was the first test I ever did, and it failed.

Anhar

 

[Karsvall]

In theory, I don´t understand why it shouldn´t work.
But then I´m not an export of rolling shutters.

Does it scan whole lines from top down to the bottom? Or does it scan pixel for pixel from upper left (like you read a book)?
What´s the speed? Does it scan from top to bottom and then directly from top to bottom again so that the actual time of the bottom of the frame is aprox. the same as the top of the next frame? (It can not be so slow, can it?)

If so, you could apply this technic. I believe. At least in theory.

The problem is that you first have to slow down the clip as many times as there is lines in the clip (720 times) and then time displace it from top to bottom. And there is not a plugin in the world that can make that kind of slomo with good results. And the render would take for ever.

And yes, every frame would in some way be distorted - but It would work. I believe.

But, if the rolling shutter is faster (it has to be a lot faster than above) - you have to slomo even more...

Bye the way, Why do they call these cmos cameras progressive. They are not.

And why do you not see these artifacts in stills?

 

[bronxjragon]

this thread is half fact, half ego trip

 

[mattsand]

I could go in depth, give you a detailed point by point breakdown as to why the distortion are not constant

i believe what lee says is correct but i'd love for him, and me, to be proven wrong. can you at least give us a couple of the points from the list?

lee simply says that the first row is always captured x milliseconds before the last row, where x is constant for a given camera. it never changes, and certainly not depending on the subject, that would be magic wouldn't it?

/matt

 

[mattsand]

If you say that a "constant" correction is all that is needed to solve the distortion, what do you think would happen when you apply this constant correction to a static image?

you would get the same static image of course. all pixels in a static image were in the same position a few milliseconds ago so no transformation would be applied. lee is really lousy at explaining because he doesn't see that you're talking about a different domain, just as you have no idea that he does. the difference is he's talking about the domain where this problem originates and can easily be solved while you're talking about a domain where it can't.

you would end up distorting it, see the problem now?

the problem is that you think it would get distorted, it wouldn't. we're, again, talking about a constant correction in temporal space. you know what temporal space is, don't you?

Let me repeat my self in case you missed it, constant correction was the first test I ever did, and it failed.

so because you didn't know how to do it right, or even what you were doing, it can't be done? you don't even know what temporal space is, so how on earth could you have tried it? is that one of the points on your list?

/matt

 

[mattsand]

The problem is that you first have to slow down the clip as many times as there is lines in the clip (720 times) and then time displace it from top to bottom. And there is not a plugin in the world that can make that kind of slomo with good results. And the render would take for ever.

that's what i was talking about. you only have to find the motion vectors for each pixel. you don't have to actually do all the transformations, just one per line. the other 719 lines won't be used so why render them?

/matt

 

[Karsvall]

you need them all to make the whole frame. 719 lines have to be time displaced.

 

[Anhar_Miah]

you would get the same static image of course. all pixels in a static image were in the same position a few milliseconds ago so no transformation would be applied. lee is really lousy at explaining because he doesn't see that you're talking about a different domain, just as you have no idea that he does. the difference is he's talking about the domain where this problem originates and can easily be solved while you're talking about a domain where it can't.
/matt

Perphaps I'm worse than Lee at explaining, from what I understand Lee and you are saying is:

(a) CMOS scan delay is fixed (because its scans at a fixed rate) etc, (I agree here)

Thus the logic is:

(b) If we apply a constant shift (to counteract the delay) then we can correct that pixel shift caused by the delay ? (I disagree here, since it depends on the motion of the camera, larger motion = larger distortation, thus the correction transforms have to natually be larger)

Clearly as you already mentioned, you can see that to apply a constant correction to a stationary camera would cause a distoration; in your words:

(c) "no transformation would be applied"

but then this contradicts the point (b) about having a constant correction, now if I've misunderstood (b) then please explain further,

and if your saying (b) is not constant as you yourself have highlighted in situation (c) then we have something that is not constant (in terms of correction).

Anhar

 

[beckspace]

wow, suddenly a lot of talking about this

meanwhile I was trying the Time Displacement approach to see if it was possible

I used, lets say, a low Time Resolution of just 30 frames (slowed down on Twixtor from 24 fps to 720fps, the ideal for 720p would be 17.280fps) and used a gradient ramp at 16 bpc and Sapphire's Time Displacement that handles maps of 16bpc and smooths between the time slices (not perfect but helps)

I used a handheld shot to test this

as you can see a Low 'time resolution' creates horizontal waves in the displacement process. i believe that a higher temporal resolution would solve it

Of course, this process is not practical, impossible to create 17K fps even from rocks, not to say to blend it. but there is a short cut to it, must be...


EDIT: the render (48mb) comparing the Original and Time Remapped at a decent resolution >>>>>>>>>>> http://www.megaupload.com/?d=40MP7BTR

the values:

slomo to 720fps

the time displacement and gradient ramp:

Retimed to 24fps (with Motion Blur )

 

[mattsand]

you need them all to make the whole frame. 719 lines have to be time displaced.

Altogether. Not for every line. There's absolutely no need to time shift the other lines when considering the current line. /matt