Algorithmically generated content can sometimes turn out to be hilariously nonsensical or remarkably dumb. But researchers at the University of Washington (one of whom sold her startup to Facebook) developed a system that uses machine vision to create indisputably cool 3D animations.

The system, called Photo Wake-Up, creates a 3D animation from a single photo. In the paper, the researchers compare it to the moving portraits at Hogwarts, a fictitious part of the Harry Potter world that a number of tech companies have tried to recreate. Previous attempts have been mildly successful, but this system is impressive in its ability to isolate and create a pretty realistic 3D animation from a single image.

The researchers tested the system on 70 different photos they downloaded online, which included pictures of Stephen Curry, the anime character Goku, a Banksy artwork, and a Picasso painting. The team used a program called SMPL and deep learning, starting with a 2D cutout of the subject and then superimposing a 3D skeleton onto it. “Our key technical contribution, then, is a method for constructing an animatable 3D model that matches the silhouette in a single photo,” the team told MIT Technology Review.

The team reportedly used a warping algorithm to ensure the cutout and the skeleton were aligned. The team’s algorithm is also reportedly able to detect the direction a subject is looking and the way their head is angled. What’s more, in order to make sure the final animation is realistic and precise, the team used a proprietary user interface to correct for any errors and help with the animation’s texturing. An algorithm then isolates the subject from the 2D image, fills in the remaining space, and animates the subject.

The researchers note that the final animation can be viewed as a video (as seen aboveo), as well as interacted with on a monitor, in augmented reality, and in virtual reality.

The system, while pretty damn neat, does have its shortcomings. The researchers pointed out in the paper that their method still can’t animate shadows and reflections, and while it can handle some self-occlusions like crossed arms, it can’t handle other ones, such as a subject sitting down with their legs crossed. The researchers also stated that the SMPL model may yield wrong animated poses “due to ambiguities.”

Still, this is definitely one of the cooler models for 3D animation, and it hints at a less static online space. “We believe the method not only enables new ways for people to enjoy and interact with photos, but also suggests a pathway to reconstructing a virtual avatar from a single image while providing insight into the state of the art of human modeling from a single photo,” the researchers wrote in the paper.

Sure, Facebook currently lets you upload a looping video as your profile photo. But a system like this—isolating a subject from its surroundings—feels a lot closer to bridging the gap between fantasy worlds and real life.

Source: gizmodo.com