DESC0024750

High brightness X-ray/synchrotron light sources are employed in many experiments performed at Department of Energy (DOE) research facilities and in laboratories around the globe. Detectors and other telemetry produce Petabyte-scale datasets from these experiments stored as high-definition (HD) image files in Hierarchical Data Format Version 5 (HDF5). The complexity and production of experiment data is rapidly increasing at DOEĺs modern X-ray facilities, presenting enormous challenges in data acquisition, processing, analysis, storage, and management. These challenges, if left unresolved, will ultimately impact scientific productivity. AirMettle Inc. is developing a real-time smart data lake solution that simplifies big data analytics and accelerates processing by an order of magnitude, or more. AirMettleĺs key innovation involves massively parallel distributed data processing within an easily deployable software-defined storage framework ideal for modern cloud environments. The solution performs basic analytics tasks at the storage layer that reduce network traffic, improve data freshness, and enable real-time operation. This project will enable the storage service to efficiently ingest and then partition HDF5-formatted light source (HD image) data so it can be processed in a massively parallel manner in the storage layer. The teams will also invent the APIs necessary to enable AI-based analytics functions and models to be applied during the in-storage processing of this data. The resulting commercial software/service from this project will be a data storage solution that enables orders of magnitude faster ingestion and analysis of petabyte scale light source data sets, as well as image and video processing for public and private organizations. This should accelerate the pace of scientific research while enhancing industrial processes such as semiconductor manufacturing, leading to lower costs and higher quality consumer goods.