Meta Neural Network Patent Showcases AI’s Place in Artificial Reality Bet

Meta wants to give you a deeper perspective.  The company filed a patent application for “neural view synthesis” using “tiled multiplane images.” To put it simply, Meta’s filing details a way to make 2D images more immersive in mixed-reality experiences using neural networks.  “The task of synthesizing novel views from a single image has useful applications in virtual reality and augmented reality,” Meta said in the filing. However, creating what Meta calls “view synthesis” can be difficult to do with larger images, the company said. 

To do so, Meta’s tech first separates a single image into two: a “color image” and a “depth image.” The original image is also fed to a neural network to extract a “feature map,” or a representation of all of the important features within it.  Then, the feature map, color image, and depth image are broken up into a bunch of tiny corresponding “tiles,” and fed to a second neural network to create “masks.” The masks separate different images into distinct planes, such as foreground, midground, or background. All of this information is used to render the original “single plane” source image into an immersive, 3D scene. 

This isn’t the first time we’ve seen Meta put AI to work in its dreams of creating a metaverse. The company has filed patent applications for AI-embedded smart glasses for people with sensory impairments, and a system similar to this patent that uses AI for artificial reality image reconstructionIt makes perfect sense why Meta seeks to use AI for these processes, said Jake Maymar,  AI strategist at The Glimpse Group. Generally, 3D-rendering of 2D data can be expensive and time-consuming, especially in the context of artificial reality. Using AI makes this process far more efficient, he said. 

And if that AI rendering isn’t processed locally on the device, but via a cloud network, that opens the door for lower latency and lighter form factor — potentially signaling how the next generation of smart glasses may work, Maymar added. “I always wonder what’s next with patents,” he said. “With this one, I actually think it’s AR glasses.” 

Plus, like the rest of tech, Meta continues forging ahead on the AI front. And in its quest for dominance, it is using one of its biggest assets: an astronomical stockpile of data. The company acknowledged last week that every public Instagram and Facebook post made by adults since 2007 has been leveraged to train its AI models (the only exception being EU-based users).  

And while this data stands to strengthen its foundational models as it seeks to compete with Google and OpenAI, Maymar noted it can also prove incredibly fruitful in artificial reality development. Training on the massive collection of text, image, and video data can lead to AR innovations in rendering, sentiment analysis, content generation, and more, he said.  It’s akin to how Stable Diffusion by Stability AI was “basically trained on all of the internet,” and is now one of the most high-powered generative models on the market, he said. “There’s a wealth of knowledge there that they can train a model on,” said Maymar. “It’s going to be pretty sophisticated.”