Skycatch, a San Francisco-based startup, has been helping companies mine both data and minerals for nearly a decade.
The software maker is now looking at creating digital twins, with an initial focus on the mining and construction industry, using the NVIDIA Omniverse platform to connect and create custom 3D pipelines.
SkyVerse, part of Skycatch’s vision AI platform, is a combination of computer vision software and custom Omniverse extensions that allow users to enrich and animate virtual worlds of mines and other sites with near real-time geospatial data.
“With Omniverse, we can transform massive amounts of non-visual data into dynamic visual information that is easy to contextualize and consume,” said Christian Sanz, Founder and CEO of Skycatch. “We can really recreate the physical world.”
SkyVerse can help industrial sites simulate variables such as weather, broken machinery and more up to five years in the future — while learning from events up to five years in the past, Sanz said.
The platform automates the entire visualization pipeline for mining and construction environments.
First, it processes data from drones, lidars, and other sensors in the environment, whether at the edge using the NVIDIA Jetson platform or in the cloud.
It then creates 3D meshes from 2D images, using neural networks built from NVIDIA’s pre-trained models to remove unnecessary objects such as dump trucks and other equipment from visualizations.
Then SkyVerse assembles this into a single 3D model which is converted into the Universal Scene Description (USD) framework. The master model is then fed into Omniverse Enterprise for creation of a digital twin synchronized live with real-world telemetry data.
“The simulation of machines in the environment, different weather conditions, traffic jams – no other platform has allowed this, but all of this is possible in Omniverse with hyper-realistic physics and a mass of objects, which is really revolutionary,” Sanz said.
Skycatch is a Premier Partner of NVIDIA Inception, a free global program that supports startups that are disrupting industries with breakthrough technologies. Premier Partners get additional go-to-market support, exposure to venture capital firms and technical expertise to help them scale faster.
Data processing and visualization
The companies have deployed Skycatch’s fully automated technologies to gather insights from aerial data at tens of thousands of sites at several major mining companies.
The Skycatch team first determines the optimal placement of data collection sensors on mining vehicles using the NVIDIA Isaac Sim platform, a robotic simulation and synthetic data generation (SDG) tool to develop , test and train AI-based robots.
“Isaac Sim saved us a year of testing time – going out in the field, placing a sensor, testing it to work, and repeating the process,” Sanz said.
The team also plans to integrate the Omniverse Replicator SDK into SkyVerse to generate physically accurate 3D synthetic data and create SDG tools to accelerate the training of perceptual networks beyond the robotic domain.
After a site’s data is collected, SkyVerse uses edge devices powered by NVIDIA Jetson Nano and Jetson AGX Xavier modules to automatically process up to terabytes per day and turn them into kilobyte-sized analytics that can be easily transferred to front-line users.
This data processing was accelerated 3x by the NVIDIA CUDA parallel computing platform, according to Sanz. The team is also looking to deploy the new Jetson Orin modules for next-level performance.
“It’s not humanly possible to sift through tens of thousands of images a day and extract critical analysis from them,” Sanz said. “So we’re helping to expand the human view with neural networks.”
Using pre-trained models from the NVIDIA TAO toolkit, Skycatch also built neural networks capable of removing foreign objects and vehicles from visualizations and texturing those points into the 3D mesh.
The digital terrain model, which is accurate to less than five centimeters, can then be imported into Omniverse for creation of a digital twin using the easily expandable USD framework, custom SkyVerse Omniverse extensions, and NVIDIA RTX GPUs.
“It only took about three months to create the Omniverse extensions, despite the complexity of our extensions’ capabilities, thanks to access to technical experts through NVIDIA Inception,” Sanz said.
Skycatch is working with one of Canada’s leading mining companies, Teck Resources, to implement the use of Omniverse-based digital twins for its project sites.
“Teck Resources uses Skycatch’s compute engine at all of our mine sites globally and is now extending visualization and simulation capabilities with SkyVerse and our own digital twin strategy,” said Preston Miller, Chief Technology and innovation at Teck Resources. “Delivering near real-time visual data will allow Teck teams to quickly contextualize mine sites and make faster operational decisions on critical and time-sensitive projects.”
Omniverse expansions built by Skycatch will be available soon – learn more.
Safety and durability
AI-powered data analytics and digital twins can make operational processes for mining and construction companies safer, more sustainable and more efficient.
For example, according to Sanz, mining companies need to be able to quickly locate the toe and crest (or bottom and top) of “benches,” narrow strips of land next to an open pit mine. When a machine is automated to enter and exit a mine, it must be programmed to stay 10 meters from the crest at all times to avoid the risk of slippage, Sanz said.
Previously, surveying and analyzing landforms to accurately determine toes and ridges typically took up to five days. With the help of NVIDIA AI, SkyVerse can now generate this information in minutes, Sanz said.
Additionally, SkyVerse eliminates 10,000 over-the-top interactions for customers per year per location, Sanz said. These are situations where humans and vehicles can pass each other in a mine, posing a security threat.
“At its core, Skycatch’s goal is to provide complete context and awareness of what’s happening at a mine or construction site in near real-time – and better environmental context leads to increased safety for operators. workers,” Sanz said.
Skycatch also aims to drive sustainability efforts for the mining industry.
“In addition to mining companies, government organizations want visibility into how mines are operating – whether their surrounding environments are properly supported – and our platform delivers that insight,” Sanz said.
Additionally, minerals like cobalt, nickel, and lithium are needed for electrification and energy transition. These all come from mining sites, Sanz said, which can become safer and more efficient with the help of SkyVerse’s digital twins and AI vision.
Dive deeper into technology for a sustainable future with Skycatch and other Inception partners in the on-demand webinar, Powering Energy Startup Success with NVIDIA Inception.
Creators and developers around the world can download NVIDIA Omniverse for freeand corporate teams can use the platform for their 3D projects.
Learn more about and apply to join NVIDIA Inception.