Gaming, creating CGI movies, building shared worlds, and creating digital twins are exciting in principle, but the complexity of building 3D models usually serves to limit the ambition of even the most dedicated auteur. However, recent innovations by nVidia, announced earlier this year for their RTX 3090 line of GPUs, are very likely to change that through the use of Neural Radiance Fields, otherwise knowns as NERFs.
NERFs work by capturing the light from several photographs or a video from a scene, then using successive layers of voxel-based approximation techniques to determine the radiance of light around an object from different perspectives. This is then fed into a specialized imaging neural network that can be used to both determine the likely three-dimensional shapes involved as well as the distinction between different objects within that space. With video, this can be translated into animation as well. The resulting models are then mapped into polygonal meshes with associated surface materials.
Moving the Image AI to the Hardware Is A Game Changer
This kind of technology has existed for a few years now, but what differentiates what nVidia has done is to encode this into the RTX 3090 chip directly, meaning that what would often take hours of processing time could instead be done in a matter of milliseconds. Moreover, such processing can also be applied to analyzing the associated kinematics, meaning that other activities, such as rigging models, could also be accomplished at or near real-time speeds.
Beyond this process, the GPU also serves to “fill in the blanks”, inferring based upon both reflected and ambient light and shadows and previous training the likely shape of areas that are not directly visible from the camera’s field of view. Recording the position of the camera relative to the scene (both via gyroscopic data in the camera and GPS) also means that the same scene can be photographed multiple times to fill out any gaps, something that has traditionally been very difficult to accomplish with older techniques.
3D Modeling As Easy as Taking a Video
The most immediate applications for NERFs will end up being in the modeling that GPUs emplaced within autonomous vehicles will do to provide for immediate and accurate feedback of where the vehicle and everything around it is in space. However, in broader terms, this has huge implications for media and gaming production, as videographers could capture scenery, objects, and people then use the corresponding generated models either directly or for use in motion capture (mocap) applications. This could also be used in conjunction with similar tools such as x-rays or sonograms, to capture biological processes (such as a heart beating) and from that generate models of organs that show irregularities that are often impossible to discern in other ways.
Finally, combining NERFs with 3D printing opens up the possibility of replicating anything from bones and teeth to mechanical parts and food within minutes. Again, because this process is fast enough to effectively capture mechanical processes, NERFs applications within Industrial IoT will make it possible to model stresses and motion that can better predict points of (and time to) failure.
A detailed analysis of the process is available from nVidia, along with a github code repository.
Conclusion
It’s hard to understate the significance of this technology. Three-dimensional modeling can be a laborious process, and the ability to generate such models in real-time is likely to have a dramatic impact on business, entertainment, medicine, retail, manufacturing (especially in the industrial space), and other areas. While it doesn’t eliminate the need for 3D modelers on the conceptual side, NERFs not only makes such models available for a much broader audience but will likely prove a critical component in developing augmented reality and industrial digital twin systems. For more information, check out the nVidia Research Blog on Nerfs.
