New Video Manipulation System Can Realistically Reproduce Anyone's Facial Expressions

Researchers from the Max Planck Institute for Informatics, Germany, Technicolor, France, Technical University of Munich, Germany, University of Bath, United Kingdom and Stanford University, United States, have combined forces for a rather alarming new project. The work, published in a study last month, reveals a new method for using deep-learning in video creation that results in extremely realistic recreations of people's complete facial expressions and movements.

"We present a novel approach that enables photo-realistic re-animation of portrait videos using only an input video. In contrast to existing approaches that are restricted to manipulations of facial expressions only, we are the first to transfer the full 3D head position, head rotation, face expression, eye gaze, and eye blinking from a source actor to a portrait video of a target actor," states the paper.

Deep Video Portraits

The paper is entitled Deep Video Portraits and its results are set to be demonstrated this August at SIGGRAPH, the ACM special interest group on Computer Graphics and Interactive Techniques. A video of the technology put to use illustrates how the researchers employ a video of a chosen subject, in this case, former President Barack Obama, to extrapolate and analyze how his/her face moves naturally.

The landmarks that constitute authentic facial expressions are then tracked in order to be recreated using the researchers' own expressions as sources of visual input information. The result is an eerily similar reproduction of nearly every discreet nuance of human expression belonging to the target source.

To ensure their technology was as accurate as they suspected, the team ran their simulations through various tests. "To demonstrate the high quality of our output, we conduct an extensive series of experiments and evaluations, where for instance a user study shows that our video edits are hard to detect," states the paper.

Ethical implications abound

The project, of course, comes with many ethical implications and potential dangers. In fact, the issues surrounding the use of recreation AIs for criminal and nefarious purposes has been a concern for a while now.

The US military's research arm, the Defense Advanced Research Projects Agency (DARPA), has recently funded the media forensics (MediFor) platform in order to "attempt to level the digital imagery playing field, which currently favors the manipulator." The project seeks to develop technologies for the "automated assessment of the integrity of an image or video" in order to determine the validity of questionable sources and avoid their use in propaganda and misinformation campaign.

[see-also]

The paper does not currently address any ethical concerns surrounding the field. However, its authors do explain that their work is a door to increasing the technology's capabilities. 

"We see our approach as a step towards highly realistic synthesis of full-frame
video content under control of meaningful parameters. We hope that it will inspire future research in this very challenging field," concludes the paper.