In the Moment (ITM)   6

Background
The Defense Sciences Office (DSO) at the Defense Advanced Research Projects Agency (DARPA) is soliciting innovative research proposals for research and technology development that supports the building, evaluating, and fielding of algorithmic decision-makers that can assume human-off-the-loop decision-making responsibilities in difficult domains, such as combat medical triage. The Department of Defense (DoD) continues to expand its usage of Artificial Intelligence (AI) and computational decision-making systems. DoD missions involve making many decisions rapidly in challenging circumstances and algorithmic decision-making systems could address and lighten this load on operators. In order to employ such systems, the DoD needs rigorous, quantifiable, and scalable approaches for building and evaluating these systems. Current AI evaluation approaches often rely on datasets such as ImageNet for visual object recognition or the General Language Understanding Evaluation (GLUE) for Natural Language Processing (NLP) that have well defined ground-truth, because human consensus exists for the right answer. However, establishing conventional ground truth in difficult domains is not possible because humans will often disagree significantly about the right answer. Rigorous assessment techniques remain critical for difficult domains; without them, the development and fielding of algorithmic systems in such domains is untenable.

Work Details
The focus of Technical Area 1 (TA1) is developing technologies that identify and quantitatively model key decision-making attributes of trusted humans in order to produce a quantitative decision-maker alignment score.
TA1 proposals should identify the theory (or theories) of decision-making that will form the basis for identifying and quantitatively characterizing key decision-maker attributes.
TA2 focuses on implementing the algorithmic decision-making system, including alignment process.
TA3 is responsible for evaluation design document including design for decision-makers with known misaligned attributes.
TA4 is responsible for outreach event plans provided at least 45 days in advance of an outreach event.

Period of Performance
The anticipated individual awards are expected to be made by DARPA with a target start date of October 2022.