TECH FOCUS AREAS: Autonomy; Artificial Intelligence/Machine Learning TECHNOLOGY AREAS: Information Systems OBJECTIVE: The objective of this topic is to explore the development of explainable decision-making algorithms that address the challenges of deriving explanations of autonomous behavior in decision-making systems. In particular, there is (i) the challenge of handling the fact that autonomous decision-making agents can change future observations of data based on the actions they take and (ii) the challenge of reasoning over long-term objectives of the underlying agent mission. The results of this work may be applied to the development of military recommender systems as part of a Phase III effort by enabling human-interpretable explanations of behavior in automated or autonomous planning solutions. This may also find applications in the commercial autonomous driving sector, where high-performing solutions still lead to unfortunate accidents and fatalities for which the derivation of explanations is difficult. In such settings, explainability not only eases understanding of learning outcomes, but can also be used to develop more effective machine learning algorithms. This topic addresses challenges in the DoD technology area of Artificial Intelligence and Machine Learning as outlined in the National Defense Strategy and, more specifically, the focus area of Autonomy as listed in the USD R&E modernization priorities. This topic will reach companies that can complete a feasibility study and prototype validated concepts in accelerated Phase II schedules. This topic is specifically aimed at later stage development rather than earlier stage basic science and research. This topic addresses challenges in the DoD technology area of Artificial Intelligence and Machine Learning as outlined in the National Defense Strategy and, more specifically, the focus area of Autonomy as listed in the USD R&E modernization priorities. DESCRIPTION: The field of explainable AI has gained significant traction in recent years as evidenced by the DARPA XAI program and other investments aimed at explaining why and how autonomous agents yield a given outcome for some given datum. While there has been much success in this area, it has been mostly limited to the classification problem where libraries such as LIME, SHAP, and Captum have been developed and proven useful in facilitating developer and user understanding of machine learning and AI. Naturally, the capacity to explain autonomous behavior is a conduit to trust in such systems. However, the problem of autonomous decision-making, wherein an agent must interact within an environment and make decisions which change the state of the same, has not been privileged to the same success when it comes to the explain ability of autonomous decision-making policies. Given the importance and growth of this area as evidenced by great advancements in sub-fields like reinforcement learning and planning, including superhuman performance in various games of perception and precision, it is important to have such powerful decision aids explain their behavior in a human-interpretable form for their wider adoption and use, particularly in safety-critical and defense systems. This is particularly so given the interest from the defense sector in human-machine teaming and human-in-the-loop learning. While the explainable AI approaches developed for the classification problem may similarly be leveraged for autonomous decision-making, two key challenges arise in the latter setting. First, the actions enacted by the agent(s) have the capacity to change the data observed in the future via changes in the environment. Second, the agent must reason about a long-term objective accomplished as the result of multiple actions. As such, we seek the derivation of explainable decision-making methods that address these issues. Furthermore, in order to establish trust in the explanations provided by autonomous agents, it is imperative that the uncertainty inherent in a given explanation be quantified or characterized so as to establish a degree of confidence in the explanation. Indeed, this encapsulates two of the four principles of explainable AI developed by the National Institute of Standards and Technology (NIST): explanation accuracy and knowledge limits. These principles focus on the accuracy of the explanation from a learning process and the assurance that the learning system operates only when it has reached a sufficient level of confidence in its output. Intuitively, the fact that learning is closed-loop with decision-making requires an understanding and quantification of uncertainty to understand how an amount of uncertainty produces a change in decision. This informs the human operators' trust in the output of the explainable decision-making algorithms developed. PHASE I: Phase I should completely document 1) the AI-driven explainability requirements the proposed solution addresses; 2) the approach to model, quantify and analyze the representation, effectiveness, and efficiency of the explainable decision-making solution; and 3) the feasibility of developing or simulating a prototype architecture. PHASE II: Develop, install, integrate and demonstrate a prototype system determined to be the most feasible solution during the Phase I feasibility study. This demonstration should focus specifically on: 1. Evaluating the proposed solution against the objectives and measurable key results as defined in the Phase I feasibility study. 2. Describing in detail how the solution can be scaled to be adopted widely (i.e. how can it be modified for scale). 3. A clear transition path for the proposed solution that takes into account input from all affected stakeholders including but not limited to: end users, engineering, sustainment, contracting, finance, legal, and cyber security. 4. Specific details about how the solution can integrate with other current and potential future solutions. 5. How the solution can be sustainable (i.e. supportability). 6. Clearly identify other specific DoD or governmental customers who want to use the solution. PHASE III DUAL USE APPLICATIONS: The contractor will pursue commercialization of the various technologies developed in Phase II for transitioning expanded mission capability to a broad range of potential government and civilian users and alternate mission applications. Direct access with end users and government customers will be provided with opportunities to receive Phase III awards for providing the government additional research & development, or direct procurement of products and services developed in coordination with the program. PPROPOSAL PREPARATION AND EVALUATION: Please follow the Air Force-specific Direct to Phase II instructions under the Department of Defense 21.2 SBIR Broad Agency Announcement when preparing proposals. Proposals under this topic will have a maximum value of $1,500,000 SBIR funding and a maximum performance period of 18 months, including 15 months technical performance and three months for reporting. Phase II proposals will be evaluated using a two-step process. After proposal receipt, an initial evaluation will be conducted IAW the criteria DoD 21.2 SBIR BAA, Sections 6.0 and 7.4. Based on the results of that evaluation, Selectable companies will be provided an opportunity to participate in the Air Force Trusted AI Pitch Day, tentatively scheduled for 26-30 July 2021 (possibly virtual). Companies' pitches will be evaluated using the initial proposal evaluation criteria. Selectees will be notified after the event via email. Companies must participate in the pitch event to be considered for award. REFERENCES: 1. Phillips, P. Jonathon, et al. "Four Principles of Explainable Artificial." NIST. (2020). 2. Arrieta, Alejandro Barredo, et al. "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI." Information Fusion 58 (2020): 82-115. 3. Gunning, David, and David Aha. "DARPA's explainable artificial intelligence (XAI) program." AI Magazine 40.2 (2019): 44-58.
