Nvidia AI Makes Artists Out of Everyone

“It’s much easier to brainstorm designs with simple sketches, and this technology is able to convert sketches into highly realistic images,” said Bryan Catanzaro, vice president of applied deep learning research at NVIDIA. GuaGAN, a pun on the name of the famous impressionist painter, Gauguin, gives its users three tools; a paint bucket, pen, and pencil.

At the bottom of the screen, there is a collection of possible objects related to landscape images. Select the cloud object, draw a line on the screen and the software will convert that into a cloud.

System ensures 'art' is always unique

But this is not just a stamp, every unique line will produce a unique cloud image. The same process applies for other landscape objects, select the tree object, draw a line and a tree will form, make it bulbous at the top to produce leaves.

The system ensures that even the same sketches with the same settings are different by having random numbers built in that ensure the software creates different results. The deep learning model is trained on millions of images and is hyper flexible, change a pond object to snow and watch your image switch seasons instantly.

Networks team up to constantly improve

“It’s like a coloring book picture that describes where a tree is, where the sun is, where the sky is,” Catanzaro said. 

“And then the neural network is able to fill in all of the detail and texture, and the reflections, shadows, and colors, based on what it has learned about real images.”

The GANs can produce such convincing results because they use cooperating pairs of networks known as a generator and a discriminator. The discriminator is trained on real images and then gives feedback down to the pixel to the generator on the success of its created images.

Nvidia makes software free

The system also allows its users to add filters to provide a more personalized style to the resulting images. “This technology is not just stitching together pieces of other images, or cutting and pasting textures,” Catanzaro said. 

“It’s actually synthesizing new images, very similar to how an artist would draw something.”

The demo is very impressive, creating pretty realistic images in just seconds. But on close inspection, there are gaps between objects and a slight line where they touch. 

Nvidia says they are working to improve this. Right now there doesn’t seem to be planning to launch the software commercially, but Nvidia hopes to make a version available for free for everyone to use.

The research related to the building of GuaGAN has been accepted as an oral presentation at the CVPR conference in June.