Face2Face: Real-time Face Capture and Reenactment of RGB Videos

joe1946

joe1946

Veteran
We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam.

Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target.

The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.
Click to expand...
http://www.graphics.stanford.edu/~niessner/thies2016face.html


Welcome to the future !

teaser.jpg
 
Last edited:
Yeah this won't be used for nefarious reasons at all... Get ready for all sorts of interesting political TV ads in the future where the producers of those ads will be able to make it look like a politician said just about anything.
 
The political ads won't get away with sheer manipulation. I look at this as something for the post instead of the actual reshoot.

Edit : as mentioned below, this will/should enable "virtual" actors, from the long deceased stars to some sort of a combo mix of pretty faces combined with experienced voice actors. As bandied about in the old "voice overdub" debates, this could make the international casts that much more feasible. "Here's a new jaw for the French/German/Spanish/Chinese voiceover and no one will know the difference because my jaw actors are the best".
 
Last edited:
This is a bad thing.

This is something that has almost zero legitimate uses, yet a vast appeal for use in illegitimate purposes.

One more way to trash up our world.
So now in a couple of years there will be no way to ever trust ANY video you see of a person ever. But a vast number of Human Beings will still take what they see to be real.

A hundred million losers are going to use this to animate attractive/popular actresses/models saying lewd sexual things, and the Internet will be flooded with it.
Great.

Add this to the column of technology NOT making the world a better place.
 
Couldn't agree more. Maybe I'm not giving it a chance but this just seems to be all kinds of wrong....
 
Well think of this in more creative ways... How far will this advance the use of a person for motion capture to make an animated character perform? Maybe it will no longer be just voice acting for animation, maybe it will be real acting to move the animation?

But yes, I certainly see the nefarious side to it.
 
You must log in or register to reply here.
Back
Top