Frontier AI Agents Power Real-World Simulations in Omniverse

Picture a virtual warehouse: metal shelving towers overhead, crates stacked in geometric patterns, and a humanoid robot stands ready at the starting line. Frank DeLise, product manager at NVIDIA, used the GPT-6 Astra AI agent to orchestrate this entire digital scene inside Omniverse. Through natural-language instructions, Astra connected physics, scene updates, rendering, and user interface components—merging them into an interactive first- and third-person simulator. This made it possible to probe warehouse workflows and assess robot behavior before a single real crate moves. For businesses automating warehouse operations, such simulators promise earlier detection of flaws and faster iteration cycles, directly reducing the time and cost to deploy new robotics.

Autonomous driving teams face a challenge: even minor tweaks to sensors or traffic models can ripple through simulations, changing results in unpredictable ways. Doyub Kim of NVIDIA solved this by instructing Astra to create a simulation called Zero to Alpamayo, set on San Francisco’s Market Street. The AI mapped out every step—from asset generation to dynamic traffic and sensor emulation—serving up a modular, reusable testbed. Kim then experimented with varying weather and lighting, using recorded videos to analyze how models performed under shifting conditions. For automotive companies, this means safer, more reliable validation of self-driving algorithms before they hit real streets, cutting both risk and R&D spend.

Ashley Reid’s work brings precision to digital twins—a critical need for industries like logistics, infrastructure, and manufacturing. Using the combined talents of Astra and Claude Fable 5, she compared outputs from simulated cameras and lidars against real-world recordings. Over three days, this iterative process helped create and refine four digital twins. The agents responded to measured discrepancies, automatically updating geometry and materials based on real sensor feedback. This grounded, data-driven approach ensures simulated environments match reality more closely, raising the bar for robotics testing, predictive maintenance, and operational planning.

What happens when you teach a simulated robot to clear hurdles or disassemble a car suspension? Tae Kim’s Robo Olympics project used Astra to train Unitree G1 humanoids on athletic tasks, with dozens of physics trials informing every improvement. Meanwhile, Jens Jebens tapped Astra for a different challenge: could a simulated robot wield a custom-designed wrench to remove suspension bolts? The answer, tested in Omniverse using real CAD models, was yes. These projects highlight how AI not only accelerates prototyping but also links digital design directly to performance outcomes—critical for companies designing robotic workflows or seeking to optimize maintenance operations.

Nic Johns, NVIDIA’s engineering director, offered a glimpse of remote collaboration’s future by assembling a live, telemetry-fed International Space Station scene in the browser with a single prompt to Astra. He then adjusted the planetary view with another. This seamless integration of assets and live data transforms simulation from a niche engineering tool to a widely accessible platform—potentially lowering the barrier for enterprises to build custom digital experiences or conduct virtual training at scale.