YouTube has already perfected the art of mixed reality videos that show the VR world and the real world at the same time. A great example is thevideo of Conan O'Brien's virtual trip to outer space taken when he visited YouTube's VR Lab. The video platform's technique still can't capture the whole picture, however, since VR headsets get in the way. Google Research and Daydream Labs had to team up to solve the problem, and they did just that with the help of machine learning, 3D computer vision and advanced rendering techniques.
In order to "remove" the VR headset in a video, the user's face has to be scanned with a camera and reconstructed as a dynamic 3D model that blinks and looks in different directions. Next, the VR headset -- Google used an HTC Vive for its tests -- has to be outfitted with eye-tracking tech in order to detect where the user is looking. Google's and Daydream's technique superimposes the 3D model on the face of the real person. It doesn't completely remove the VR headset, though: it just replaces it with a transparent version to prevent the uncanny valley effect.
As you can guess, the technique will benefit YouTube creators the most at this point in time. In fact, Google is exploring ways to make it accessible to YouTubers. It could become a lot more useful when VR becomes more widespread, though. For instance, it could be adapted to show people's faces in VR video conferences or the faces of friends playing VR games together.
Google's VR painting app lands on the Oculus Rift
Tilt Brush, Google's virtual reality painting app, is available today on the Oculus Rift, complete with a few tweaks to make your 3D art take shape as naturally as possible. Tilt Brush landed on the HTC Vive in April 2016 and it's generated some awe-inspiring work.
To use Tilt Brush on the Oculus Rift, artists will need the system's $200 Touch controllers -- accessories that we found to be incredibly comfortable, responsive and essential for experiencing all the Rift has to offer.
Google has customized the Oculus Rift version of Tilt Brush in a few ways: One, while in-app, rest your finger on a button of the Touch controller to see what function it serves. Two, artists are now able to rotate and resize their work. And three, the Rift's built-in headphones offer unique audio feedback for each tool of artistry.
"Different brushes create different sound effects, and they become a vivid part of the experience through your headphones," Google writes. "We love using audio reactive mode with Rift headphones and seeing strokes come to life with light and sound."
Google recently unveiled the Tilt Brush Toolkit, an open-source toolset that allows artists to use their 3D sketches in Unity, adding animations, interactive elements and music to their doodles.