OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Advanced Computing and Software;Sustainment;Trusted AI and Autonomy OBJECTIVE: Develop a capability to generate skillful, near real-time environmental forecasts (of the atmosphere, ocean, sea ice, and/or ionosphere) at a much higher spatial horizontal resolution (less than 1 km) and vertical resolution (on the order of 10 m, particularly in the atmospheric boundary layer and/or upper ocean) than current machine learning weather prediction (MLWP) techniques using downscaling or similar methodologies for tactical/local scale applications. DESCRIPTION: While rapid and impressive progress in the development of skillful MLWP and related environmental models over the past several years has been promising for improving near real-time forecasting, missing technological capability requirements must be solved before such tools can be used operationally. Among the biggest gaps in MLWP - as well as in operational numerical weather prediction (NWP) and even climate simulations - is the disconnect between the improved predictability from large/global scale physics and observations and the lack of forecast fidelity at the local/tactical scale (less than 1-km resolution in the horizontal and high-fidelity in the vertical). Some progress can be made rapidly through the use of ML methods ameliorating the intensive computing requirements, but other progress is needed in scientific methodology to attain higher resolution forecast from lower resolution models informed by and/or consistent with physics-based principles and/or models (see references). This SBIR topic seeks to leverage advances in ML methods, environmental prediction, and downscaling/super resolution techniques to develop new capabilities for targeted local and tactical scale short-range to medium-range forecasts. Many uses of environmental predictions (e.g., tropical cyclone wind and surge effects, electro-optical or acoustic propagation in the boundary layer, coastal winds, terrain-induced phenomena, aviation visibility, ice-edge circulations, ocean eddies, navigation through sea ice) require knowledge of small-scale features to properly calibrate the effects of a forecast. Efforts will synthesize various methodologies to take a coarse set of environmental prediction information and utilize additional data, ML methods, and modeling techniques to better inform predictions of tactical/local scale effects. A particular emphasis on validating realistic environmental structures, particularly given the lack of observational data at these scales, will be necessary. PHASE I: Develop and compare innovative approaches for performing high-resolution (less than 1-km horizontal grid spacing and high-fidelity in the vertical) short-range to medium-range MLWP forecasts from a lower resolution set of historical data, observations, and forecast fields. Perform a technical feasibility survey of state-of-the-science methods for developing high-resolution environmental fields, such as downscaling, nesting, mesh refinement, and super resolution sharpening and upscaling. Inform the final selection(s) of methodology by the theoretical background review as well as targeted cases studies to demonstrate the strengths and weaknesses of these methods. Prepare a final report that must include a summary of strengths and weaknesses of viable approaches, a comparative analysis, and viability for operational use (such as forecast stability, bias/accuracy, and limitations). Additionally, a Phase II plan should be included. PHASE II: Expand demonstration into an end-to-end prototype high-resolution (horizontal and vertical) MLWP forecasting capability. While initialization, coarse forecast, and post-processing components are necessary, focus will be on improving the skill and capability of the downscaling phase of the production in particular, addressing the fidelity of components that support resolving smaller scale processes. While explicit use of high-resolution features such as topography, bathymetry, land and sea ice character, vegetation, land surface type, urban character, etc. may not be necessary, ensure that the model should properly account for the effect of these and other local scale influences. Ensure that the downscaling capability is portable and globally relocatable to arbitrary grid sizes and resolutions at any worldwide location, with the ability to scale available data and quantify potential skill as needed. Develop a workflow that includes robust methods of validation and verification and identifies strengths and weaknesses of the product compared to traditional NWP downscaling and nesting. Perform multiple demonstrations in coordination with field testing (may be required), particularly to validate local scale effects that are not easily verified given current environmental observing network coverage. Submit required Phase II deliverables to include regular reporting, participation in program reviews, technical documentation, and the end-to-end prototype software at the conclusion of the effort. PHASE III DUAL USE APPLICATIONS: Perform operational hardening and establish utility and trust for real-time application. Craft and demonstrate dynamic analysis software tools that quickly and accurately convey software system health, error logging and debugging, and processing metadata. Develop additional evaluation metrics and diagnostics to facilitate expert forecaster guidance on using the product (and comparing to other forecast tools in workflow). Ensure that the system has a formalized methodology and data/compute needs for model training and a separate, leaner set of requirements for operational runs, and fully documented. (Note: It is essential that a version of this system is able to run skillfully in a forward/limited compute environment.) Ensure that the techniques are generalizable to apply to a variety of environmental modeling use cases such that follow-on work and commercial applications can be addressed. Dual-use applications will include partnering with other intergovernmental meteorological agencies such as U.S. Air Force, National Oceanic and Atmospheric Administration (NOAA), and National Aeronautics and Space Administration (NASA) as well as commercialization in multiple potential markets with high-resolution decision making requirements based on forecast skill, such as transportation (short time scales) and agriculture (longer time scales). REFERENCES: 1. Rodrigues, Eduardo Rocha, et al. "DeepDownscale: A deep learning strategy for high-resolution weather forecast." 2018 IEEE 14th International Conference on e-Science (e-Science). https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8588749 2. Partee, Sam, et al. "Using machine learning at scale in numerical simulations with SmartSim: An application to ocean climate modeling." Journal of Computational Science 62, 2022, 101707. https://www.sciencedirect.com/science/article/pii/S1877750322001065 3. Yeganeh-Bakhtiary, Abbas, et al. "Machine learning as a downscaling approach for prediction of wind characteristics under future climate change scenarios." Complexity 2022.1, 2022. 8451812. https://onlinelibrary.wiley.com/doi/epdf/10.1155/2022/8451812 4. Bodnar, Cristian, et al. "A Foundation Model of the Earth System." arXiv preprint arXiv:2405.13063 (2024). https://arxiv.org/pdf/2405.13063 5. Salcedo-Sanz, Sancho, et al. "Analysis, characterization, prediction, and attribution of extreme atmospheric events with machine learning and deep learning techniques: a review." Theoretical and Applied Climatology 155.1, 2024. pp. 1-44. https://link.springer.com/content/pdf/10.1007/s00704-023-04571-5.pdf 6. Ben Bouall gue, Zied, et al. "The Rise of Data-Driven Weather Forecasting: A First Statistical Assessment of Machine Learning Based Weather Forecasts in an Operational-Like Context." Bulletin of the American Meteorological Society 105.6, 2024, E864-E883. https://doi.org/10.1175/BAMS-D-23-0162.1 7. Sun, Yongjian, et al. "Deep learning in statistical downscaling for deriving high spatial resolution gridded meteorological data: A systematic review." ISPRS Journal of Photogrammetry and Remote Sensing 208, 2024, pp. 14-38. https://ui.adsabs.harvard.edu/abs/2024JPRS..208...14S/abstract 8. Sha, Yingkai, Ryan A. Sobash, and David John Gagne. "Generative ensemble deep learning severe weather prediction from a deterministic convection-allowing model." Artificial Intelligence for the Earth Systems 3.2, 2024, e230094. https://journals.ametsoc.org/configurable/content/journals$002faies$002f3$002f2$002fAIES-D-23-0094.1.xml?t:ac=journals%24002faies%24002f3%24002f2%24002fAIES-D-23-0094.1.xml 9. Buster, Grant, et al. "High-resolution meteorology with climate change impacts from global climate model data using generative machine learning." Nature Energy, April 2024, pp. 1-13. https://www.nature.com/articles/s41560-024-01507-9 KEYWORDS: machine learning; artificial intelligence; ai/ml; meteorology; oceanography; forecasts; high resolution; machine learning weather prediction (MLWP); downscaling; training datasets; transfer learning; data assimilation; parameterization; high resolution
