Disney's innovative AI can quickly make actors appear younger or older
Researchers at Disney have built an artificial intelligence tool that can make it easier for an actor to appear a different age on screen. Although digital artists can still make necessary modifications to make the effects in a scene look as realistic as possible, the artificial intelligence system can handle most of the aging effects.
Re-aging characters in films using AI
In movies and advertisements, it is costly to create photorealistic digital re-aging and requires artists to go through each scene arduously, frame-by-frame to manually change the character’s appearance and likeness. In the past, there have been attempts to automate the strenuous, time-consuming, and expensive task, using AI, both machine learning and neural networks.
The Disney researchers said that other systems currently available “typically suffer from facial identity loss, poor resolution and unstable results across subsequent video frames.” The research team mentioned that their method offers “the first practical, fully-automatic and production-ready method for re-aging faces in video images.”
The face re-aging network, or FRAN
The team calls their AI system FRAN, an acronym for face re-aging network.
FRAN is a neural network that incorporates neural face models from being pre-trained on thousands of images of different faces without any age pairing. Neural face models are created using deep neural networks and are used in digital work and facial animation.
Researchers mentioned that it would be impossible to train FRAN on datasets using real people because that would require numerous pairs of images showing people with the same facial expressions, on the same backgrounds at two known ages. FRAN was trained using datasets from thousands of randomly generated faces to collect the necessary information for re-aging actors.
The new research conducted by Disney Research Studios shortens the aging process in video creation. Disney Research Studios focuses on innovations in filmmaking by using deep learning, artificial neural networks, and other subsets of artificial intelligence to use in videos.
The neural network can analyze an actor’s headshot and predict what part of the face would potentially be affected by aging. It then applies the aging properties, such as wrinkles for older effect or skin smoothing for a younger aesthetic. The process layers the AI- generated effect on top of the original face. The ability to use AI in the images speeds up the process and efficiently creates realistic images.
Re-aging actors in 2D images
The novel AI model FRAN is able to generate images by improving them on a 2D re-aging workflow, according to researchers at Disney. “FRAN currently only improves upon the predominant 2D re-aging workflow, whereas in particular cases a 3D re-aging solution may be preferable for more elaborate levels of control including, for instance, relighting and other physically based manipulations,” the researchers said in the study.
Any images that require enhancements in a 3D format are still being explored. The team hopes that they will one day be able to also use artificial intelligence on 3D re-aging as well. “We believe that these limitations also represent exciting opportunities for additional improvements in future work.”
Challenges using a U-NET architecture
Researchers mentioned a few limitations with the current AI tool. There are a few challenges, such as generating large changes on images. The team notes the complexities are often difficult in typical U-Net architectures, a form of deep learning. A U-Net architecture is a symmetric architecture used in image segmentation, created in 2015, specifically at the time to solve issues within biomedical image segmentation. It is now used in various artificial intelligence systems. It is named for the shape of its network, forming a U-shape and consisting of the usual AI elements of training the network using input datasets to produce output.
FRAN creates the images using a U-NET architecture incorporating a large number of datasets. The AI system can create realistic and re-aged images within a range of 18 and 85 years old.
Due to the limitation of creating large changes, stated by the researchers, there is difficulty in aging characters from extremely young ages to the present age of the actor. They recommend using other methods, such as LATS or DLFS, known as deep learning from scratch in AI, in these cases.
The research team also noted that another limitation when aging characters using AI, is the difficulty of graying scalp hair on characters, an effect that the AI system currently is unable to capture from the datasets. An additional limitation is re-aging characters while including possible changes in Body Mass Index (BMI) as they age. “Similarly, re-aging can also introduce variation in body mass index (BMI) whose effects on the face we currently cannot control,” the team stated.
The future of FRAN
Despite the challenges, the team is optimistic about using the AI to improve the re-aging process in films and create a simpler, quicker process they can continue to build upon through future advancements. “FRAN has the potential to improve existing re-aging workflows,” the researchers said, “reducing the time it takes to re-age complete shots from a matter of days to just a few hours or even minutes, facilitating the creation of high-quality visual effects at scale.”
Two researchers become the first to map all the glaciers that end in the ocean and estimate their pace of change over the previous 20 years.