Leveraging Multimodal and Generative Artificial Intelligence to Advance the Application of Social Robotics in Caregiving

Number of anticipated awards: 2 Budget (total costs, per award): Phase I: [$ 500,000] for [ 12] months; Phase II: [$ 2,500,000 ] for [ 2] years Fast Track proposals will be accepted: Yes Direct to Phase II proposals will be accepted: No It is strongly suggested that proposals adhere to the above budget amounts and project periods. Proposals with budgets exceeding the above amounts and project periods may not be funded. Summary: The burden of caregiving for a PwD arising from psychological (e.g., responding to repetitive questioning) and non-psychological (e.g., aiding daily function) stressors is overwhelming. Assistive technologies can attenuate caregiving burden. However, most solutions are mobile or web-based that (i) are limited to specific domains of need/tasks, (ii) cause cognitive overload in PwD (e.g., complex interface), and (iii) have dependencies such as device accessibility/literacy, and an abstract two-dimensionality, constrained within a digital space, which is devoid of sensory-richness; this fails to replicate physical reality. These limitations reduce meaningful PwD engagement. Social robotics can yield multifunctional caregiving aids that capture the three-dimensional and multi-sensory (e.g., touch, gaze) characteristics of the real world. Achieving such functional versatility requires overcoming challenges that hinder intuitive, nuanced, and adaptive verbal and physical human-robot-interaction, and multi-environment robot mobility. Training approaches to achieve these desired outcomes have relied on brute-force approaches where scripted interactions and environmental scenarios integrated with natural language processing and machine learning were used to incrementally improve performance. However, such training datasets are laborious to build, which has slowed progress in social robotics. Multimodal and Generative AI can address the need for: (i) Large training datasets for various physical and social behaviors, (ii) Efficient and improved multisensory integration and interpretation, and (iii) Enhanced Human-Robot-Interaction. The broad objective of this contract proposal is to comprehensively leverage Multimodal and Generative AI to provide a new impetus to revolutionize the application of social robotics in caregiving for PwD.