Universal Neural Information Acquisition Architecture for Cognitive Augmentation

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Human-Machine Interfaces; Advanced Computing and Software; Space Technology; Integrated Network Systems-of-Systems OBJECTIVE: Develop a composable and extensible software architecture targeting a low/no code user interface for the configuration, logging, and viewing of real-time neural and peripheral sensors combined with human cognitive performance tasks. Additionally, the architecture should be extensible to support the addition of new novel tasks and emerging sensors. DESCRIPTION: Brain-machine interface (BMI) technology may provide a decisive decision advantage to Airmen by providing dynamic decision support or performance augmentation in response to changes in neural patterns. Coupling BMI with peripheral sensors that monitor heart rate, skin conductance, pupil dilation, or gaze position could enhance the ability to detect issues such as fatigue, workload, or stress. However, challenges emerge when attempting to combine the diverse sensor data meaningfully while maintaining the temporal characteristics. The problem is made more difficult given the heterogeneous nature of data streams delivered by different device manufacturers. As a result, there are many bespoke solutions for sensor fusion with a specified set of input devices. Often these solutions are brittle and tied to a particular manufacturer for one or more devices. More generalized solutions are available (e.g., lab streaming layer [LSL; 1]). These solutions provide a good start for tackling parts of the integration challenge, but do not provide a holistic commercial off-the-self (COTS) solution. In the case of LSL, barriers include limited device support, disjointed user-interface implementations, and the difficulty of integrating new devices. Adding new devices, sensors, or tasks to a project is particularly problematic given that it can a require months of specialized development, discouraging innovation and adoption. Meanwhile, the burgeoning commercial market for wearable sensors and neurotechnology will require additional integration efforts. To facilitate the development and adoption of real-time BMI, a low/no code software architecture is needed for the fusion of neural, physiological, and behavioral data streams that is agnostic to the sensor systems providing inputs. Solutions should be flexible such that input streams can be manipulated (e.g., preprocessed), grouped, hidden, or automated (e.g., monitored) based on the user's selected preferences. Solutions should also have bidirectional integration with established software packages (e.g., MATLAB, Python) and languages (e.g., C#, C++) for relaying user inputs, and facilitating data processing, analysis, and machine learning. Solutions should also have the capability to send/receive data from external APIs for proprietary data processing. Solutions should have clear, well-documented API requirements for input/outputs from new devices and software and, ideally, model existing and emerging (e.g., [2]) industry norms and open-source solutions where feasible. Initially, solutions should be able to function on a secure isolated network, although compelling cloud-based solutions will be considered, especially with documentation describing on-premises setup and configuration. PHASE I: Phase 1 should focus on concept development. The resulting proposal should completely document: 1. The proposed approach to implementing a low/no code software architecture for multimodal sensor fusion of neural and peripheral sensors and human cognitive performance software 2. The composability of the proposed system, including a description of the level of technical expertise necessary to add multiple sensors to the system and begin data collection 3. The interoperability of the proposed system, including a description of the amount of software development necessary to add a novel, previously unsupported, sensor device to the system 4. The extensibility of the proposed system, including a description of the capabilities for signal processing and machine learning to be applied to individual input streams 5. The bidirectional communication capabilities, including a description of the needs for a another system to receive data from the proposed system PHASE II: Performers will develop and demonstrate a prototype system. The demonstration should focus specifically on: 1. Evaluating the expertise and technical skills necessary to setup a new system from scratch. Documentation of an independent external evaluation of the implementation and use of the system is highly encouraged, although not required. 2. A description of the interoperability of the system to accommodate an array of existing neural and peripheral sensors and a justification of the selected sensors. 3. A description of the specific details and requirements about how the solution can integrate with current and potential future sensors. Performers will should specifically address what about their solution makes it sustainable. 4. An evaluation of the time necessary for integrating a to-be-determined set of novel neural and/or peripheral sensors into the system 5. An evaluation of the extensibility of the system for integrating real-time or near real-time data processing and analysis signal with local or remote resources. PHASE III DUAL USE APPLICATIONS: The performer will pursue further generalization of the technology developed, with the aim of transition to a working commercial or warfighter solution as a self-contained BMI middleware application that can be operated by non-experts. REFERENCES: 1. Swartz Center for Computational Neuroscience (2018). Lab Streaming Layer. https://github.com/sccn/labstreaminglayer; 2. Easttom, C., Bianchi, L., Valeriani, D., Nam, C. S., Hossaini, A., Zapa a, D., ... & Balachandran, P. (2021). A functional model for unifying brain computer interface terminology. IEEE Open Journal of Engineering in Medicine and Biology, 2, 91-96. KEYWORDS: Brain Machine Interface; Brain Computer Interface; Training; Learning; Cognitive Enhancement; Closed loop systems; Extended Reality; Neuromodulation; Cognitive Interventions; Cognitive State