OUSD (R&E) MODERNIZATION PRIORITY: Artificial Intelligence/Machine Learning TECHNOLOGY AREA(S): Human Systems, Information Systems OBJECTIVE: Create software-hardware systems to sense and process a user's physiological states, providing just-in-time triggering of adaptive automation to improve computer-supported task performance. DESCRIPTION: Artificial Intelligence (AI) enabled systems are used by human users to reduce mental workload and ensure greater accuracy and speed at performing task or set of tasks through a robust division of responsibility between AI and human user. Technological capabilities to sense and process a user's cognitive state and intervene with adaptive automation aids when needed (and not when the user performs well) would improve human-AI collaborative performance. This SBIR topic is intended to introduce new technology to improve aid triggering that is not presently known to exist. Proposals will posit physio measures that can be sensed, processed, and used as triggers to aid user only when the user needs help. The research question is: How can physiological indicators from a user be used to improve AI-user collaborative performance? The contractor will need to provide a proposal outlining a plan for how technological innovations can be used to: (1) sense pertinent physiological data from the user; (2) transform these data into meaningful digital signatures; (3) detect and set cutoffs to determine whether the physiological states indicate a need for cognitive intervention to aid goal completion in the form of adaptive automation; and (4) provide solutions to how AI can aid a militarily relevant task (e.g., convoy route-planning, threat assessment from military intelligence, command and control, etc.) when the user is struggling as indicated by poor performance or physiological indicators. Physiological measures are often associated with different cognitive processes and therefore may act as triggers for adaptive automation meant to aid the user. Physiological measures are objective (i.e., not involving subjective opinions of the user), and therefore plausible options for AI triggers. The goal of this SBIR initiative is to develop a system that can: (1) sense output from the user for two of five identified physiological variables; (2) assess two of five distinct cognitive variable states; (3) set quantitative thresholds for each of the variables (via physiological assessment) at which the system engages or disengages adaptive automation, (4) implement at least one adaptive automation application using any two of the five physiological variables, and (5) demonstrate task performance increases through empirical testing. PHASE I: The end of Phase I should produce several outcomes in the range of TRL2-3. (1) Identification of five target physiological measures that relate to specific cognitive variables with at least two citations corroborating these conclusions. (2) Proposed ranges for the two most promising physiological variables that would indicate a need for triggering adaptive automation based on literature search or pilot testing and consultation with ARL researchers. (3) One citation and description of technology that senses and records two of the five chosen physiological measures. (4) An explanation and justification for a plan to leverage each of the existing technologies to process measures in real time. (5) A plan to form new technology which can record and process in real-time minimally two of the five physiological variables that have not previously been recorded simultaneously in the same device. The integration of physiological variables is a particular challenge as physiological variables often differ on the time scale in which they may be collected and analyzed from tens of milliseconds to seconds. These five points need to be incorporated into a report at the end of Phase I that also: (1) Discusses the project's problem space and current limitations to demonstrate full understanding of what needs to be solved; (2) Explains a methodology to overcome challenges and limitations; (3) Provides a conceptual design of the problem solution with anticipated performance at the end of Phase II, and (4) Outlines what will be done in Phase II. A successful report will demonstrate a path forward for using physiological variables from the user to give technology critical inputs to understanding when and how the user can be helped when challenged. Therefore, a successful Phase I will make a strong argument that the two chosen physiological variables are measurable and are predictive of the associated cognitive states. PHASE II: The end of Phase II should produce several outcomes in the range of TRL 4-5. (1) Methods of measuring and processing two of the five physiological variables that relate to each cognitive state isolated in Phase 1. (2) One working demonstration for each measure with a display representing a near-real-time assessment of the measure. (3) One working demonstration of at least three of the physiological measures in an Army-oriented task and scenario in which the user's physiological state triggers adaptive automation. The task must involve decision-making regarding uncertainty as this a major focus area for Artificial Intelligence in general and particularly in Multi-Domain Operations. (4) A demonstrated ability to transfer adaptive automation trigger data to a third-party software. (5) Recommendations and paths forward for implementing adaptive automation based on the remaining physiological measures. The contractors should work toward the following benchmarks to enhance the odds of Phase III investment: (1) Flexibility of approach to account for numerous tasks (e.g., air traffic control, Army training programs, surveillance and sentry duties, security screening); (2) Resiliency of equipment to continue working in rugged conditions; (3) Ability to detach and stay powered when in environments without ready power sources; (4) Capability to interface with a range of secondary systems; (5) Capability to transition technology for commercialization to industry and possible Army applications; (6) Robust statistical procedures to account for large variability in physiological recordings; and (7) Plan for data sharing and use of experimental data, particularly for use by government personnel. A successful conclusion to Phase II will demonstrate technology that can predict through two physiological-variable inputs when a user needs help and experimental results showing the efficacy of the equipment with at least a 0.6 Area Under Curve (AUC) improvement in performance. PHASE III DUAL USE APPLICATIONS: At the conclusion of the SBIR, the contractor will be well positioned to offer numerous technological applications for end users in both the commercial and military domains. In particular, the contractor may design adaptive automation for mental workload intensive jobs such as Intensive Care Unit monitoring and coordination, air traffic control, sports psychology, tutoring system development, pandemic responses, and military intelligence analysis. When physiological measures indicate difficulties with cognitive processing in any of these domains, adaptive automation may be triggered to ease the cognitive burden associated with performance of duty and thus improve outcomes. In the end, the contractor should be positioned to produce one or more potential commercial technologies that could be inserted into defense systems. The market contains many examples of work processes that involve users engaging with smart technology and computers. Following a successful Phase II, award winner can use the knowledge gained and technology created to optimize any number of these processes with adaptive automation using physiological sensors attached to the end user and deliver improved AI-user collaborators performance across a host of tasks and jobs. REFERENCES: Byrne, E.A., Parasuraman, R.: Psychophysiology and adaptive automation. Biol. Psy. 42, 249--268 (1996) Kaber, D.B., Wright, M.C., Prinzel, L.J., Clamann, M.P.: Adaptive Automation of Human-Machine System Information-Processing Functions. Hum. Fact. 47, 730-741 (2005) Parasuraman, R., Barnes, M., Cosenzo, K., & Mulgund, S.: Adaptive automation for human-robot teaming in future command and control systems. Technical Report. Army Research Laboratory (2007) Recarte, M.A., Nunes, L.M.: Effects of verbal and spatial-imagery tasks on eye fixations while driving. J. Exp. Psy.: App. 6, 31--43 (2000); Segerstrom, S.C., Nes, L.S.: Heart rate variability reflects self-regulatory strength, effort, and fatigue. Psych. Sci. 18, 275--281 (2007) Batmaz, I., & Ozturk, M.: Using pupil diameter changes for measuring mental workload under mental processing. J. App. Sci., 8, 68-76 (2008) Cassenti, D.N., Gamble, K.R., & Bakdash, J.Z.: Multi-level cognitive cybernetics in human factors. In K. Hale and K. Stanney (Eds.), Advances in Neuroergonomics and Cognitive Computing (pp. 315-326). New York: Springer (2016) Cassenti, D.N., Kerick, S.E., & McDowell, K.: Observing and modeling cognitive events through event related potentials and ACT-R. Cog. Sys. Res., 12, 56--65 (2011) Critchley, H.D.: Book review: Electrodermal responses: What happens in the brain. Neuroscientist, 8, 132--142 (2002); Feigh, K. M., Dorneich, M. C., & Hayes, C. C.: Toward a characterization of adaptive systems a framework for researchers and system designers. Hum. Fac., 54, 1008--1024 (2002) Goldstein, D. S., Bentho, O., Park, M. Y., & Sharabi, Y.: Low-frequency power of heart rate variability is not a measure of cardiac sympathetic tone but may be a measure of modulation of cardiac autonomic outflows by baroreflexes. Exp. Physio., 96, 1255-1261 (2011) Kaber, D. B., & Endsley, M. R.: The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theo. Iss. Ergo. Sci., 5, 113-153 (2004) Kaber, D. B., & Riley, J. M.: Adaptive automation of a dynamic control task based on secondary task workload measurement. Int. J. Cog. Ergo., 3, 169-187 (1999) Marinescu, A. C., Sharples, S., Ritchie, A. C., Sanchez Lopez, T., McDowell, M., & Morvan, H. P.: Physiological parameter response to variation of mental workload. Hum. Fac., 60, 31-56 (2018) Minotra, D., & McNeese, M. D.: Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring. Cog., Tech. & Work, 19, 161-177 (2017) Naicker, P., Anoopkumar-Dukie, S., Grant, G. D., Neumann, D. L., & Kavanagh, J. J.: Central cholinergic pathway involvement in the regulation of pupil diameter, blink rate and cognitive function. Neurosci., 334, 180--190 (2016) Parasuraman, R., & Riley, V.: Humans and automation: Use, misuse, disuse, abuse. Hum. Fac., 39, 230--253 (1997) Picard, R. W., Fedor, S., & Ayzenberg, Y.: Multiple arousal theory and daily-life electro-dermal activity asymmetry. Emotion Rev., 8, 62--75 (2016); Steinhauser, N.B., Pavlas, D., & Hancock, P.A.: Design principles for adaptive automation and aiding. Ergo. Des., 17, 6--10 (2008) Thayer, J.F. & Lane R.D.: Claude Bernard and the heart-brain connection: further elaboration of a model of neurovisceral integration. Neurosci, & Biobeh. Rev., 33, 81--88 (2009) KEYWORDS: Artificial intelligence; Cybernetics; Information Science; Behavior; Intelligence; Physiological psychology; Military psychology
