FRAMEWORK FOR APPLICATION LIFECYCLE MANAGEMENT AND CONTINUOUS INTEGRATION FOR PRE-EXASCALE HPC ARCHITECTURES

RT&L FOCUS AREA(S): 5G, General Warfighting Requirements (GWR); Nuclear TECHNOLOGY AREA(S): Information Systems; Materials; Weapons OBJECTIVE: The objective of this project is to develop a secure Application Lifecycle Management (ALM) and Continuous Integration / Continuous Delivery (CI/CD) framework for legacy codes. Such a capability would integrate existing tools into a cohesive framework to automate a series of steps such as test suites, ensuring code coverage of testing, version control e.g. Git & GitLab, streamlining of build process and bookkeeping of these steps/tests/versions. Once built and tested, deploy the application code using a Singularity software container on multiple physical systems, and eventually in a cloud . DESCRIPTION: The Defense Threat Reduction Agency (DTRA) uses High-Fidelity computer codes to investigate weapon effects phenomenology and techniques for countering WMD. The High-Fidelity codes have in some cases evolved over many decades. This topic will begin to bridge the gap from legacy practices to modern practices that combine security, software development and information technology operations (SecDevOps) that test the security of the software as part of a continuous integration/continuous delivery (CI/CD) pipeline. The government-owned computational fluid dynamics code, Second-order Hydrodynamic Automatic Mesh Refinement Code (SHAMRC), is heavily used by many of the DTRA programmatic areas utilizing High-Fidelity computer codes. It is a particularly challenging application for containerized deployment because a pre-processor reads user input and generates unique source code for each problem. The code is then compiled prior to execution. SHAMRC capabilities include non-responding and responding structures, interactive particles, several atmosphere models, multi-materials, many high explosive detonation models, a K-Epsilon turbulence model, a particle surface heating model, non-equilibrium radiation diffusion, water vaporization, and dust non-equilibrium chemistry. SHAMRC is second-order accurate in both space and time, is fully conservative of mass, momentum, and energy, and runs in parallel using Message Passing Interface (MPI). Because SHAMRC is export-controlled software, another large legacy FORTRAN High Fidelity code may be used as a SHAMRC proxy in Phase I. Typically, ALM tools might include capabilities related to Requirements, Software & Hardware Development, Quality Assurance & Test Management, Release & SecDevOps. Because this effort is targeted towards existing legacy High Fidelity application codes, the emphasis should be on Quality Assurance & Test Management, Release & SecDevOps. Component integrations should be modular so that one tool such as GitHub for example, could be replaced with use of GitLab, to meet the needs of a different commercial customer in Phase III. The entire framework must run in user space without root access, although it may utilize existing underlying tools that are installed by the system administrators such as Git, GitLab and Singularity. PHASE I: Develop an approach for design of a secure ALM, CI/CD framework. The Framework design should encompass innovative use of existing open-source tools, such as Git, GitLab and the Singularity container using the fake root capability, with targets of Red Hat Enterprise Linux and SUSE Linux Enterprise Server. At the end of Phase I the performer should have a completed architecture and an early prototype with limited capabilities as proof of concept to demonstrate feasibility of the technical approach. For each component of the framework that is not an encapsulation of an existing tool, a stub may be used as a placeholder for capabilities to be added in Phase II. PHASE II: Develop a production ready framework based on the Phase I approach and integrate with selected existing tools. Implement additional capabilities not available in existing tools to address other proposed features. Demonstrate the use of the framework on Department of Defense (DoD) High Performance Computing Program (HPCMP) systems on several High-Fidelity application codes, to include the SHAMRC code. The ability to migrate the ALM workflow and application code in a Singularity container from a DoD HPCMP physical system to a Cloud Service Provider is desirable. Cloud migration is subject to availability of cloud resources at Impact Level 5. An open research proxy may be used for development on Impact Level 2 resources. PHASE III DUAL USE APPLICATIONS: The secure ALM, CI/CD framework and associated workflows developed for use on very demanding application codes will be well suited, once refined, for use on more general HPC workloads on pre-Exascale architectures. Improvements in this phase are expected to involve ease of use enhancements and hardening of the framework for use on a wide range of application software used in Government research and industry. REFERENCES: 1. Githttps://git-scm.com/; 2. GitLabhttps://centers.hpc.mil/users/gitlabUserGuide.html; 3. JENKINShttps://www.jenkins.io/; 4. Singularityhttps://sylabs.io/docs/#singularity; 5. Singularity fake roothttps://sylabs.io/guides/3.5/user-guide/fakeroot.html; 6. DoD HPCMPhttps://centers.hpc.mil/; 7. SHAMRChttp://www.dtra.mil/Portals/61/Documents/dispatch_v3_i2_web.pdf?ver=2014-09-26-104733-797; 8. HPCMP Documentation:https://centers.hpc.mil/users/documentation.html;