NVIDIA researchers are teaming up with academic institutions worldwide to drive progress in generative AI, robotics, and the natural sciences. More than a dozen of these projects will be showcased at NeurIPS, one of the most esteemed AI conferences globally, scheduled for December 10-16 in New Orleans. NeurIPS serves as a gathering ground for experts in generative AI, machine learning, computer vision, and related fields. NVIDIA Research will unveil a range of innovations, including techniques to transform text into images, photos into 3D avatars, and specialized robots into versatile machines.
Jan Kautz, Vice President of Learning and Perception Research at NVIDIA, emphasized that NVIDIA Research is at the forefront of advancements in the field. Their collaborative projects with leading academic minds aim to accelerate the development of virtual worlds, simulations, and autonomous systems.
NVIDIA researchers have partnered with universities on multiple projects to enhance diffusion models, the most popular form of generative AI models that transform text into realistic imagery. One of these projects, presented as an oral presentation, focuses on improving the understanding of the relationship between modifier words and main entities in text prompts, resulting in more accurate visual depictions. Another project, called SceneScape, employs diffusion models to generate long videos of 3D scenes from text prompts, maintaining consistency and plausibility across frames, creating various scenes from art museums to ice castles.
A third project shows how text-to-image diffusion models can generate missing parts of 3D objects based on incomplete point cloud data, facilitating applications in robotics and autonomous vehicles.
AI avatars are designed using multiple generative AI models to create and animate virtual characters and convert text into speech. NVIDIA presents new methods for making these processes more efficient. One project converts a single portrait image into a 3D head avatar with high fidelity, including hairstyles and accessories, without requiring multiple images or a lengthy optimization process. Another project introduces P-Flow, a generative AI model that can swiftly synthesize high-quality personalized speech based on a short reference prompt, surpassing existing models in pronunciation, human likeness, and speaker similarity. NVIDIA researchers will present two posters at NeurIPS showcasing innovations in reinforcement learning and robotics. One introduces a framework for developing adaptable reinforcement learning algorithms that avoid common pitfalls and perform well on benchmark tasks.
The second addresses object manipulation in robotics, enabling AI models to generalize to new shapes and categories of objects, improving their ability to interact with previously unseen items.
NVIDIA’s research contributions span the natural sciences, including physics simulations, climate modeling, and healthcare. These projects aim to enhance accuracy, computational efficiency, and predictive capabilities. A neural operator architecture accelerates computational fluid dynamics, achieving significant speed improvements for large-scale 3D simulations.
Collaborating with climate scientists and machine learning researchers, NVIDIA contributes to ClimSim, a vast dataset for climate research that enhances prediction capabilities for storms and extreme events. Additionally, NVIDIA Research interns introduce an AI algorithm capable of delivering personalized predictions for medication dosage effects on patients, improving accuracy in blood coagulation and antibiotic level predictions based on real-world data.
