RT&L FOCUS AREA(S): Artificial Intelligence/Machine Learning TECHNOLOGY AREA(S): Chemical/Biological Defense; Information Systems Technology; Battlespace Environments; Sensors OBJECTIVE: Develop a tool capable of automatically generating 3D models by the fusion of images from various sources, such as but not limited to LIDAR, x-ray, photos, satellites, and blueprints. These 3D models will be used for projection via augmented reality (AR), inserted into virtual reality (VR) platforms, and serve as terrain for transport and dispersion (T&D) modeling. DESCRIPTION: Visual representation of chem/bio hazards is one of the various types of information of interest to allow Warfighters to gain situational awareness prior to operations and responses. Recent efforts managed by the Digital Battlespace Management Division at the Joint Science and Technology Office for Chemical and Biological Defense (JSTO-CBD) have looked into leveraging extended reality (XR) technologies to provide modern, advanced, and realistic representations of chem/bio hazards. Active projects are also looking into merging modeling and simulation (M&S) capabilities with XR tools. Nevertheless, there is a gap in the ability to rapidly, automatically, and accurately generate 3D models. The ability to generate 3D models of items (e.g. devices, threat-filled weapons), buildings, and terrain based on multiple image sources to support visualization, mission rehearsal, and Chemical-Biological (CB) hazard T&D modeling is desired. In this development, the Chemical and Biological Defense Program and the JSTO-CBD look to work with small business firms to develop a software capability that can merge and fuse imagery data from sources such as LIDAR, x-ray, night-vision thermal/IR images, visible light camera photos, satellite images, open source maps, mobile phone images/sensors, and/or blueprints (in format of plan PDF or computer assisted drawing, CAD, electronic files). This capability should rapidly and automatically output photorealistic 3D models that can be georeferenced with topographical accuracy allowing varying degrees of fidelity to support different needs on hardware systems with different computational power and/or rendering capacity. The software capability should be able to generate 3D models based on one to all of the image sources mentioned above; it is understood that model fidelity can vary when data sources are limited. In addition to exterior representations, the ability to generate the interior layouts of buildings based on exterior images is of interest. The interior layouts may be generated based on inference model(s) to be developed under this effort, yet should be true-to-drawings when building blueprints are available. If image data for building interiors are available, the ability to merge these data and/or correct inference model(s) for 3D reconstruction is also desired. Considerations should be taken when HVAC information is available as the building airflow plays a key role in hazard T&D in buildings. The ability to output parameters necessary for CONTAM multi-zone models based on blueprints is desired. The tool should also be capable of extracting necessary information to be utilized for generating JSTO-CBD developed box models. The application of artificial intelligence (AI)/machine learning (ML) algorithms may be necessary at any one or various points of the 3D model generation workflow; proposals should identify if and how AI/ML will be utilized. The capability to be developed under this topic should allow flexible outputs in commercial standard formats, which can further be utilized in other commercial platforms, computational fluid dynamics (CFD) modeling, and/or support DoD tools. This software solution must be able to operate in both connected and disconnected environments. Proposals must provide innovative solutions that are forward compatible as well as demonstrate knowledge and expertise working with state-of-art technologies relating to 3D reconstruction/3D model generation/rendering and understanding of intricacy of T&D modeling. Successful developments should adapt modular designs and agile software development processes. PHASE I: Design and develop a process for automatic 3D reconstruction and 3D model generation using the fusion of image data types listed above. Identify methods and approaches to develop an interior inference model and any AI/ML algorithms necessary that are to be developed in Phase II and Phase III. Develop an early prototype to demonstrate the ability to automatically generate 3D models compatible with all modern game engines based on the fusion of two of the above mentioned image sources for items and terrains as well as that based on fusion of two of the above mentioned image sources to include blueprints/CAD files for buildings. The ability to generate parameter files necessary for CONTAM models based on CAD files should also be demonstrated. PHASE II: Refine the design and the prototype to allow automatic 3D reconstruction and 3D model generation that can be georeferenced with topographical accuracy based on one to all image data sources listed above. Develop interior inference model and AI/ML algorithms as needed. Develop the capability to parameter files necessary for CONTAM models based on blueprints (in format of plan PDF or computer assisted drawing, CAD, electronic files) or based on interior inference model that can be associated with the 3D models. Assumptions on airflow when using interior inference model should be scientifically supported. The 3D models generated should also be flexible to accept association with other CONTAM or CFD models of choice of the user. PHASE III: Refine the software capability to allow for options on multiple commercial standard output formats of 3D models with options for varying degrees of fidelity to support various applications and needs since the 3D models generated may be utilized in other commercial platforms, support CFD modeling, and support DoD tools. Demonstrate the ability of the tool to generate 3D models that can be utilized in programs on hardware with wide range of computation power and/or rendering efficiency. Refine interior inference model and AI/ML algorithms. Refine the process to generate parameter files necessary for CONTAM models and/or extracting information necessary to generate box models to be associated with the 3D model. PHASE III DUAL USE APPLICATIONS: This technology can support civilian and military operations, planning, and situational awareness. The ability to generate 3D models based on fusion of image sources can support industries focusing on graphics or XR technologies. Applications in civil engineering, forensic site reconstruction, digital twin generation, medical image fusion, and the construction industry are also realized. REFERENCES: 1. W.S. Dols, et al., 2009, Development and Demonstration of a Method to Evaluate Bio-Sampling Strategies using Building Simulation and Sample Planning Software. NIST Technical Note 1636. 2. Ham, H. et al., 2019, Computer vision based 3D reconstruction: a review. International Journal of Electrical and Computer Engineering (IJECE), 9(4), pp. 2394-2402. DOI: 10.11591/ijece.v9i4.pp2394-2402 3. Remondino, F. et al., 2006, Image-based 3D Modeling: a review. The Photogrammetric Record 21(115): pp. 269 291. DOI:10.1111/j.1477-9730.2006.00383.x 4. Suveg, I. et al., 2002, Automatic 3D Building Reconstruction. Proc. SPIE 4661, Three-Dimensional Image Capture and Applications V, (8 March 2002); DOI: 10.1117/12.460181 5. Xue, J. et al., 2021, Review of Image-Based 3D Reconstruction of Building for Automated Construction Progress Monitoring. Applied Sciences, 11(17), pp. 7840. DOI: 10.3390/app11177840 6.Wang, Q. et al., 2019, Applications of 3D point cloud data in the construction 1 industry: A fifteen-year review from 2004 to 2018. Advanced Engineering Informatics, 39, pp.306-2 319. DOI:10.1016/j.aei.2019.02.007 7. Yang M-D. et al., 2018, Fusion of Infrared Thermal Image and Visible Image for 3D Thermal Model Reconstruction Using Smartphone Sensors. Sensors, 18(7). DOI:10.3390/s18072003 8. Sentenac T. et al., 2018, Automated thermal 3D reconstruction based on a robot equipped with uncalibrated infrared stereovision cameras. Advanced Engineering Informatics, 38, pp: 203 215. DOI: 10.1016/j.aei.2018.06.008 9. https://www.nist.gov/services-resources/software/contam KEYWORDS: 3D reconstruction, 3D model, augmented reality (AR), virtual reality (VR), extended reality (XR), photogrammetry, terrain, building, transport and dispersion (T&D), modeling and simulation (M&S)
