realtime 3d motion tracking data

R

Ruairi Robinson

Member
Hello,

Say one was (hypothetically) to make a feature film in the near future, and it was pretty effects heavy. And say one wanted to have a way of getting camera motion data in 3d space in realtime from the camera, transmitted wirelessly to a nearby computer, (or have the information storted within the redcode raw files) what would it take to develop such a system? is it plausible?

It would pretty much revolutionise a LOT of visual effects work if this were possible. If you've seen any of the making of stuff for Spielbergs A.I. they had a system that did this, but it looked like the most unwieldy, awkward, clunky setup in the history of motion picture history. But they were able to key realtime and place the characters in virtual sets. Realtime keying aside, for complex VFX this would have AMAZING advantages. There are automated trackers in 3d now, but they tend to fail a fair percentage of the time, and the other times, you can easily spend a week on a shot to get it tracked in 3D - it can be a major pain in the ass, and a major waste of time and money (imagine this on a 2500 shot VFX movie!) but if this process was automatically recorded in an inobtrusive way on set, it would be a major time and money saver in post. You could make nearly every shot a VFX shot if tracking was an automated process if you wanted to - things like replacing billboard on a street, for example, would be a cinch. Or adding more graffiti to a wall in the background! Or changing stuff you CANT alter on location... All this stuff is possible already, but an automated live 3D tracking system would make it much less of a big deal to do this kind of stuff...

I guess the way one would IDEALLY like to work is:
Dump a bunch of static sensors out of sight on the set/location. Preferably These sensors would automatically triangulate with each other (so you wouldn't have to place them on a painstackingly measured grid), and with another sensor attached to the camera (and have to be placed on unmoving surfaces) and the sensor on the camera would have to be be fixed in a position measured precisely relative to the camera nodal point. Also would be ideal if lens and zoom (and lens distortion) info were able to be automatically read. Is it possible for any of this information to be recorded on each frame? (even if manually marked)

Would be even better if you could have a mix of static and free sensors - place a couple of sensors on the back of a truck, and static sensors in the vicinity, and you would save a shitload of animation and tracking work if you wanted say, to place a virtual stunt double on the back of the truck. Or whatever. Barrels that roll off. CG smoke. composite fire elements. Whatever.

About the sensor - I'm really just brainstorming here, but is this technology even possible? is there such a sensor, that can triangulate 3d position data and measurements in relation to other sensors? (or the data from each of them be computed relative to a laptop computer system within wireless range. would it be expensive to produce? Am I missing anything? Would enough people be interested? I think they would, I'd shoot every frame of a feature film this way if the option was there. Of course a Z depth channel wouldn't hurt either :)

I want this ****ing technology. What would it take?

Ruairi Robinson
www.ruairirobinson.com
Red Reservation #862
 
In theory that all sounds viable, dunno if such a thing exists. I do recall reading in Cinefex years ago for that Sylvester Stallone movie Daylight http://imdb.com/title/tt0116040/

that for the ending shot they needed to track Stallone and Survivor Girl into the ocean open, filmed from a helicopter, and the issue was how to lock'em down for tracking. The answer - they used a GPS system that they augmented with additional gear on ground to make it even more accurate - and it worked great, locating a helicopter hundreds of feet off the ground and flying around at I dunno, 30-80 mph, and with this radio based system tracked it within INCHES. So for 50 feet away on set, that sounds pretty doable to track within what, 1/4 centimeter or so? What's accurate enough?

In any case, I like that idea. Just would need an open ended enough metadata system to record that data and preserve it through REDCINE. DPX has a genericized header you can dump all kinds of stuff into - RaveHD actually stores audio info in DPX headers - so it isn't as if there is a shortage of space there.

-mike
 
I think DPGS is what they used. It basically takes GPS data and makes it more accurate by taking readings over a longer period of time. If you don't need realtime tracking data then its fairly easy to get sub-centimeter accuracy by post processing the data to take out the error. Generally it is more difficult to track objects that are closer to the sensor than it is to track objects far away from the sensor because they measure the distance by calculating the "time of flight" - how long it takes for signal to get to the sensor from the transmitter. The closer the distance, the shorter the time.. every nano second equates to +/- 1 inch of measurement error so you need really fast processing to get the accuracy.

It's definatley possible to do what you want to do but there's a cost. Most of the systems we've researched start at around $6K and they can generally track an object in 3D space within an inch or two. The downside is that they aren't small...typically you need at least 4 beacons and they need to be seperated a by a certain distance (100ish feet) to triangulate the position of the tracking pendant.
 
Last edited:
MikeCurtis said:
In theory that all sounds viable, dunno if such a thing exists. I do recall reading in Cinefex years ago for that Sylvester Stallone movie Daylight http://imdb.com/title/tt0116040/

that for the ending shot they needed to track Stallone and Survivor Girl into the ocean open, filmed from a helicopter, and the issue was how to lock'em down for tracking. The answer - they used a GPS system that they augmented with additional gear on ground to make it even more accurate - and it worked great, locating a helicopter hundreds of feet off the ground and flying around at I dunno, 30-80 mph, and with this radio based system tracked it within INCHES. So for 50 feet away on set, that sounds pretty doable to track within what, 1/4 centimeter or so? What's accurate enough?

In any case, I like that idea. Just would need an open ended enough metadata system to record that data and preserve it through REDCINE. DPX has a genericized header you can dump all kinds of stuff into - RaveHD actually stores audio info in DPX headers - so it isn't as if there is a shortage of space there.

-mike
Click to expand...


Hey Mike - thanks for the plug on HDforindies :)

Another possibility for shooting at sea - if the boat was anchored you could cable a sensor to the sea floor to get a locked position relative to the moving sensors... (and maybe sink several other sensors nearby)
I though about GPS, but I dunno... how expensive would that be? I assumed too that it would be grossly innacurate. But how about this: GPS with time/date data stamped too, would let you get the EXACT position of the sun too automatically. Wouldn't that be handy for doing 3D comps in exterior scenes.
I'd love to know if there's a technology that can do this. Cheap.

I'm not sure 1/4 centimetre would be accurate enough to cover the gamut of shooting requirements. Not for small scale stuff anyway...

The other thing you would need is a way to translate this date into popular 3D software - XSI, 3DSmax, Maya and Lightwave. Can't imagine it would be too hard though. AfterFX and Flame too, of course!

I love the idea of loading a "shot" into 3dsmax, or AfterFX and it comes in as a background plate, complete with camera move built in. Imagine how ****ing EASY so many shots would be to comp as well...

Other than DGPS (pr perhaps in addition to it...?) what other methods of collecting spatial data are available, and what's cheap, whats FAST, what's accurate, what needs less setup... can this data transmit through water, or glass... is it line of sight only? Would love to know more...

CVB if these systems are... 6k - is that to RENT, or buy the equipment...?

Cheers,
R.
 
Thats to buy.. alot of them are based on wifi. Neat stuff though. We were unimpressed by the amount of infrastructure needed so we are going down another path for our focus system. I bet by next year you will see something like this on the market for alot less than $6K... wink wink.
 
Last edited:
CVB said:
Thats to buy.. alot of them are based on wifi. Neat stuff though. We were unimpressed by the amount of infrastructure needed so we are going down another path for our focus system. I bet by next year you will see something like this on the market for alot less thatn $6K... wink wink.
Click to expand...

got any links...?
 
i know a hardware setup called cinital will do it realtime.but limited to dolly movements, and other movements like in stedycams which will be compatible with this setup is in R&D.it will record the cameras motion 'live' and feed it into softwares like motion builder.i think this setup can be used with red one camera. here's the link to the site www.cinital.com.

(does this answer ur question?)
 
This is being done now. Douglas Trumball talked about it several years ago.

Occidental Studios in Hollywood has their "Smart Stage" It's exactly what you want.

There are other people doing the same thing.
 
Doug is the man! I used to work on post-production for one of his projects. He has a bunch of patents for some really cool technology. The TV show I worked on, "The Book of Pooh", used live-action puppets in a greenscreen studio, controlled by multiple puppeteers wearing reflective green suits. Realtime keying and motion tracking data were automatically processed and the video fed back to the cameraman. The cameraman could actually see the the puppets in the virtual environment in realtime. But this was all done with a hardware primatte keyer, and the camera was DigiBeta. I was just a roto monkey, specializing in supercloning where the puppeteers rods crossed over in front of the puppets.
 
Didn't George Lucas do this with the last three Star Wars films? Actually had a large on set flat panel and a nice couch if I remember the articles. Real time action mixed with real time CG display.
 
There are a few on set visualization systems out there,
But noting that fits to what Ruairi wants as far as I know.

One really cumbersome option would be a milo or simular rig on set that pumps out its camera data. I've worked on some shows that used that method years ago (it wasnt a perfect soultion by any means).

Viz|FX enables in-studio and on-location visualization of 3D images and animations in real-time, resulting in tremendous cost savings for post-production.
Click to expand...

http://www.vizrt.com/db/106/7/103/document332.ehtml
 
Last edited:
sky1walker said:
if u want a realtime keyer to use with a digital camera(like red) u can try dvmatte blast.dvMatte Blast uses the Graphics Processing Unit (GPU) to achieve high quality, realtime keys on the desktop. You can literally key HD footage while it's playing. here's the link.

http://www.dvgarage.com/prod/prod.php?prod=dvmatteb
Click to expand...

It's not so much that I want to do realtime previsualisation on set, (which is fantastic and has it's place - especially on larger bluescreen stuff on stage) as just having the 3d track data encoded with EVERY shot, so further down the line, doing effects shots isn't such a big deal - even for little fixes that might be a BIG deal to track in post, but if the tracking is no longer an issue, you could do these things without breaking a sweat...
 
Ruairi Robinsonas said:
just having the 3d track data encoded with EVERY shot, so further down the line, doing effects shots isn't such a big deal - even for little fixes that might be a BIG deal to track in post, but if the tracking is no longer an issue, you could do these things without breaking a sweat...
Click to expand...

The data would probably still need to be tweaked a little . Even the method I mentioned, using the motion control camera to give you back its location data is not perfect. Various different real world inaccuracies still lead to the CG elements not lining up perfectly or floating.
 
3D visualization

3D visualization

The BBC paper is very interesting. One of the key concepts of RED that may help in this work is using the USB port to capture data provided by external sensors; then storing that as metadata with each frame, so that the position of the camera - plus iris, focus and zoom data - can be absolutely recreated in a 3D world.
 
That's a very interesting idea Stuart. So the USB port will be flexible in its use? Does that mean 3rd party developers can write drivers for the camera to allow communication with other devices via the USB port?

Nick
 
saw a setup that looked like the one on the bbc paper at NAB NY, it seemed to work fine in realtime, the guys from Abel Cine Tech was checking it out and was considering getting one for rentals, don't remember the price, but it was not cheap
 
jamestmather said:
Hi Ruairi,

James here - check out the following articles :

http://interactive.usc.edu/members/534/archives/2005_02.html

(about half way down - search for "spielberg")

and :

http://www.bbc.co.uk/rd/pubs/papers/paper_05/paper_05.shtml

(the BBCs tracking system - used by Spielberg)

I know its not the catch all that you hoped for but it might give you a starting point.

Later (and thanks for all the invaluable help on mechanics - plus congrats on getting signed)

Later

James
Click to expand...



Hey James - My pleasure! And thanks for coming to the screening the other night :)

All these solutions for tracking seem a little unwieldy. I'll betcha in the not too distant future, there will be a much more graceful/mobile solution that requires a lot less setup time... even if it means I have to kickstart the development myself.

Anyway, can't wait to see what you have in the works, and hope it happens for you VERY soon. I'm expecting great things...

Cheers,
Ruairí
 
You must log in or register to reply here.
Back
Top