AI weather model can produce forecasts 5,000 times faster than gold-standard weather model, Microsoft says
A team of researchers set out to revolutionize weather forecasting using artificial intelligence.
So far, forecasts have come out 5,000 times faster and outperformed the current gold standard of weather forecasting models on 74% of data, such as wind speed, wind direction and humidity, according to the Microsoft Aurora model developers.
"Traditional weather and climate models aim to predict how the atmosphere evolves by formulating a set of governing equations using first physical principles (e.g. conservation of mass, momentum, energy, etc.)," Principal Research Manager of the AI Aurora model Paris Perdikaris said. "Then the solutions to these equations are approximated in space and integrated in time using sophisticated numerical algorithms that run on computer clusters."
AIRLINE HARNESSES ‘NEXT GENERATION OF FORECASTING’ TO CUT DOWN ON WEATHER DELAYS AND CANCELATIONS
According to Perdikaris, the European Center for Medium-Range Weather Forecast's (ECMWF) model called the HRES-IFS, which stands for High Resolution Integrated Forecast System, is the gold standard of traditional forecasts. However, producing a 10-day forecast at a 6.8-mile resolution takes about 65 minutes on large sets of computers with hundreds of processors. That's expensive and slow, he said.
"Traditional numerical weather predictions would require hours to produce the same result using a supercomputer with thousands of CPU cores," Perdikaris said. "The primary advantage here is Aurora circumvents the cost of running a physics-based simulation, and its evaluation is highly parallelized across compute cores and fully leverages the capabilities of modern GPU (graphics processing unit) hardware."
Perdikaris said his team looked at AI models like PanguWeather and Google's GraphCast, which are only trained on a single data set and designed to only create a single specific prediction task like a 10-day global weather forecast with a 6.8-mile resolution. The developer said that these were the first AI models that competed with and sometimes outperformed the gold standard.
However, the team wanted the model to tackle a wide variety of Earth system and science data prediction tasks. Aurora was born.
FORECASTING WITH AI: COMPANY CREATES FIRST WEATHER AND CLIMATE GENERATIVE ARTIFICIAL INTELLIGENCE
"With Aurora, we asked the question of whether we can build a much more general AI system that can tap into the vast expand of Earth system data and excel at a range of prediction tasks," Perdikaris said.
So they developed a "foundation model." It's a machine learning program, "that makes decisions in a manner similar to the human brain, by using processes that mimic the way biological neurons work together to identify phenomena, weigh options and arrive at conclusions," according to IBM.
Aurora was trained in two phases: pretraining and fine-tuning. Pretraining is the most expensive and time-consuming part, Perdikaris said. It took about 32 NVIDIA graphics processing units about 2-and-a-half weeks to ingest and learn from a collection of weather and climate simulation data which included analysis and reanalysis. AI models only need pretraining and fine-tuning once.
HOW AI CAN HELP (AND HURT) CLIMATE CHANGE
"Aurora’s effectiveness lies in its training on more than a million hours of diverse weather and climate simulations, which enables it to develop a comprehensive understanding of atmospheric dynamics," the developers wrote for a Microsoft blog. "One of the key findings of this study is that pretraining on diverse datasets significantly improves Aurora’s performance compared to training on a single dataset. By incorporating data from climate simulations, reanalysis products, and operational forecasts, Aurora learns a more robust and generalizable representation of atmospheric dynamics."
Then comes fine-tuning, which improves Aurora's accuracy at specific tasks like 10-day global weather forecasting or 5-day global air pollution prediction.
"During ‘finetuning’ Aurora learns from a much smaller corpus (collection) of high-quality simulation data that correspond to the target prediction task," Perdikaris said. "This phase is less computationally demanding, and can be completed in about 5 days using 8 NVIDIA A100 GPUs (graphics processing units)."
To test Aurora out, the team forecast northwest Europe's Storm Ciaran. The ferocious bomb cyclone killed 14 and spun up winds equivalent to a strong Category 3 hurricane, along with flooding.
STORM CIARAN SPAWNED RECORD-STRONG TORNADO AS DEADLY STORM TORE THOUGH UK'S JERSEY ISLAND
"The low-pressure system associated with Storm Ciarán set new records for England, marking it as an exceptionally rare meteorological event," stated the blog. "The storm’s intensity caught many off guard, exposing the limitations of current weather-prediction models and highlighting the need for more accurate forecasting in the face of climate change."
Other AI weather-prediction models failed to forecast the extreme wind speeds and rapid intensification. Aurora produced results similar to the gold standard traditional weather model at a fraction of the time and cost.
"Aurora’s improved performance on extreme events stems from the fact that it has ‘seen’ a much larger and diverse corpus (body) of data during its pretraining and finetuning phases – this volume of data naturally brings better coverage of extremes," Perdikaris said.
REAL OR FAKE? HOW TO SPOT WEATHER IMAGES GENERATED BY AI
"Aurora’s effectiveness to adapt to new scenarios for which training data may be sparse can be attributed to the fact that the pretraining phase yields a model that has already extracted good features from data, and this allows us to quickly adapt the base pretrained model to new prediction scenarios, especially in case where data scarcity would have otherwise prevented us from training a specialized model from scratch," he continued.
Aurora takes just seconds to form a 10-day global weather forecast, 5-day pollution forecast or concentrations of greenhouse gases using a single GPU.
WE ASKED THE ARTIFICIAL INTELLIGENCE-BASED CHATGPT TO EXPLAIN THE WEATHER. HERE ARE THE RESULTS:
The team is currently working on open sourcing Aurora to make it available to the general public. Perdikaris is also in talks with NOAA, ECMWF and the UK Met Office to evaluate the AI model to "asses it's potential in becoming the backbone of future operational systems."
"I think we are still at least a few years from that, but the field is moving really fast," he added.
"The ability of foundation models to excel at downstream tasks with scarce data could democratize access to accurate weather and climate information in data-sparse regions, such as the developing world and polar regions," stated the blog. "This could have far-reaching impacts on sectors like agriculture, transportation, energy harvesting, and disaster preparedness, enabling communities to better adapt to the challenges posed by climate change."
Original article source: AI weather model can produce forecasts 5,000 times faster than gold-standard weather model, Microsoft says
