OUSD (R&E) MODERNIZATION PRIORITY: Network Command, Control and Communications; Autonomy TECHNOLOGY AREA(S): Ground Sea; Space Platform; Information Systems; Air Platform OBJECTIVE: The objective of this topic is to develop a mid-fidelity stand-alone testbed for critical research on human-autonomy teaming concepts that uses command and control (C2) tasks relevant to Joint All Domain Command and Control (JADC2) missions (air, land, sea, cyber, and space). Ultimately, use of a mid-fidelity research testbed that enables quick turn research will inform design guidelines and interface components, minimizing long lead time and costly modifications to high-fidelity systems currently used to investigate and validate human-autonomy teaming for military operations. DESCRIPTION: 711 HPW/RHWC 6.2 human-autonomy research has relied on a mix of low-fidelity software, mid-fidelity testbeds, and high-fidelity systems (example of the latter is the Intelligent Multi-UxV Planner with Adaptive Collaborative/Control Technologies [IMPACT]). [1] Although low-fidelity testing yields quick answers to interface concepts agnostic of mission requirements, testbeds provide a quick way to examine concepts within a representative mission environment without the lead time and software costs involved to modify a fully operational system such as IMPACT. A mid-fidelity testbed will be an essential research tool for evaluating new human-autonomy teaming solutions, given that more and more systems will include some form of intelligent aiding due to advancements in artificial intelligence that change the role and tasking of human operators. The Collaboration of Humans and Autonomy Research Teaming Testbed (CHART2) product of this effort will be a critical component for evaluating the effectiveness of candidate control and display technologies in providing support to human and autonomy JADC2 team members, in that current testbeds are inadequate. For example, a mid-fidelity testbed, Adaptive Levels of Autonomy (ALOA; SBIR product from the early 2000s [2]) has supported over a decade of experiments investigating system performance related to levels of automation that are, however, limited in their ability to support teaming with the human operator. There are newer teaming test environments that demonstrate modularity in human-autonomy teaming structures, generate scenarios for observing human interaction, and facilitate the measurement and analysis of responses. [3] However, these mid-fidelity testbeds, as well as the high-fidelity IMPACT system, are inadequate to enable quick-turn research investigating human-autonomy teaming that involves asset allocation/transfer, task allocation/sharing, and varying communication structures. Nor do they support tasks that are representative of those envisioned for future complex JADC2 missions. They are also not designed to simulate or support a range of robust and highly capable types of autonomy (e.g., that support vehicle operations, C2, and/or decision support tools for mission related tasks). Lastly, their autonomy components are less able to work as a teammate to the human operator in flexible communication and interaction structures, how and when humans and autonomous teammates update and share information or collaborate in a task, involving a variety of data streams in support of completing JADC2 relevant mission tasks. Thus, a stand-alone mid-fidelity testbed featuring simulated and/or real autonomy components is needed that supports research examining a variety of human-autonomy teaming/communication structures. This testbed should support representative JADC2 tasks, meaning missions that include task completion with UxV (Unmanned Vehicles), as well as at least satellite and cyber effects to collaboratively address tasks within the simulated environment. A variety of test protocols should also be supported by which the experimenter can specify which domain(s) are available for task completion, as well as which teaming protocols are in effect, how communication structures are configured, and the available candidate display/control interfaces for any given trial/mission. Moreover, the testbed needs to be modular to enable the experimenter to configure a variety of multi-domain scenarios such that tasks/their order/mission events/difficulty level, as well as the autonomy's capability/reliability/transparency can be specified across multiple experimental trials. Ideally, the tasks, although representative of envisioned JADC2 tasks, should be easily trained to enable a wide range of test participants (college students to DoD subject matter experts). With respect to completion of specific JADC2 related tasks/missions, the testbed should be designed such that the human-autonomy teaming can be dynamic and context dependent. The testbed should support assessment of team collaboration on task completion, in order to determine what teaming structures and station interfaces best support mutual visibility and directability across human-autonomy members, as well as enhance task performance, completion of mission objectives, and the human's situation awareness and appropriate trust in the autonomy. [4,5] Specifically, the CHART2 testbed should enable timely evaluation of the effectiveness of candidate controls and displays in establishing and updating working agreements that define each human-autonomy team member's responsibilities for completing JADC2 task related functions, as well as coordinate courses of action, communicate pertinent information, track task completion/system status, and support shared situation awareness. [6] The results from research using a human-autonomy teaming focused mid-fidelity CHART2 testbed in DoD laboratories will mature solutions and accelerate follow-on validation research in high-fidelity systems (e.g., IMPACT), as well as inform C2 interface requirements and decision support aids needed for eventual JADC2 military applications. For example, multiple quick turn experiments using the CHART2 testbed can narrow down specific symbology and needed level of detail on the autonomy's processing to effectively apply multiple domains for task completion. The product's mission context and tasks also have the potential to be reconfigured for human-autonomy teaming applications appropriate for multiple civilian and commercial domains (e.g. Air Traffic Control, Emergency Response coordination), as well as basic research to examine factors influencing human-autonomy teaming on task performance (e.g., human's personality/experience level/workload and autonomy support's timeliness, transparency, etc.). PHASE I: Phase I will primarily focus on a) exploring the human autonomy teaming research space to determine what teaming concepts and interfaces will be supported and configurable in the testbed, b) identifying JADC2 tasks to represent within the testbed and relevant team performance metrics with easily exportable formatted data, and c) describing the proposed hardware/software developmental approach for implementing the testbed to best support a variety of teaming related experimental designs. For the latter, relevant features for experimenter control include, but are the limited to, selection of: available domains, number and types of tasks to be completed and resulting workload level, teaming structure (number of teammates, responsibility of teammates, etc.), features of controls and displays by which the human-autonomy team members interact, capability/transparency/reliability of the autonomy, and objective and subjective participant data to record and analyze. PHASE II: Develop and demonstrate a prototype testbed with the human-autonomy teaming concepts and JADC2 representative tasks identified in Phase I. This demonstrative testing should focus specifically on: 1. The choices for manipulated teaming structures. 2. Validated measures collectable using the testbed and their exportability 3. How the solution can be sustainable and scalable to the needs of researchers. Other specific DoD or governmental customers who express interest in using the product should also be identified. PHASE III DUAL USE APPLICATIONS: The software development practices employed against this topic will inform research testbed designs for a variety of human and autonomy teaming applications, both commercial and government. Additionally, with modifications, the testbed could be reconfigured to support research examining human-autonomy teaming in support of related civilian applications. REFERENCES: Draper, M., Rowe, A., Douglass, S., Calhoun, G., Spriggs, S., Kingston, D., ... & Reeder, J. (2018). Realizing autonomy via intelligent hybrid control: Adaptable autonomy for achieving UxV RSTA team decision superiority (also known as Intelligent Multi-UxV Planner with Adaptive Collaborative/Control Technologies (IMPACT)) (AFRL-RH-WP-TR-2018-0005). Wright-Patterson Air Force Base United States: Air Force Research Laboratory. Johnson, R., Leen, M., & Goldberg, D. (2007). Testing adaptive levels of automation (ALOA) for UAV supervisory control (Technical Report AFRL-HE-WP-TR-2007-0068), Air Force Research Laboratory. O'Neill, T., McNeese, N., Barron, A., & Schelble, B. (2020). Human autonomy teaming: A review and analysis of the empirical literature. Human Factors, 0018720820960865. Calhoun, G., Bartik, J., Ruff, H., Behymer, K., & Frost, E. (2021). Enabling human-autonomy teaming with multi-unmanned vehicle control interfaces. Human-Intelligent Systems Integration (Special Issue on Human-Autonomy Teaming in Military Contexts ), 3, 155-174. 1-20. https://doi.org/10.1007/s42454-020-00020-0 O'neill, O. (2002). Autonomy and trust in bioethics. Cambridge University Press. Calhoun, G., Bartik, J., Ruff, H., Behymer, K., & Frost, E. (2021). Enabling human-autonomy teaming with multi-unmanned vehicle control interfaces. Human-Intelligent Systems Integration (Special Issue on Human-Autonomy Teaming in Military Contexts ), 3, 155-174. 1-20. https://doi.org/10.1007/s42454-020-00020-0 KEYWORDS: Human Autonomy Teaming; Teaming; Testbed; Test console; Tasks; Automation; Human AI teaming; Human Agent Teaming
