80NSSC24CA087

Small Business Innovation Research Program (SBIR) Phase II

Person Aware Liaison (PAL)

Collaborative humanndash;robot teaming behaviors use the complementary strengths of humans and robots and are at their most successful when the human trusts in their robot companion. However, despite contemporary approaches to make robots more transparent in their behaviors and safer in their operation, humans still do not trust robots the same way they do a human partner. Humans may feel prohibited from placing full trust in their robot collaborators due to current robotsrsquo; inability to observe all communications from their partner, draw usable inference from the contextual clues of the interaction, and respond in a courteous manner. To that end, we propose Person Aware Liaison (PAL) as an intermediary interface between the human and the robot as they complete a collaborative task. PAL generates multimodal observations (speech, gestures, where the humanrsquo;s gaze lingers, body posture, and robot registered events) during the execution of the task and fuses the information into a singular natural language transcription of events. This transcript is queried by a large language model (LLM), a state-of-the-art text-based inference model to understand the humanrsquo;s intent. Finally, PAL acts upon the extrapolated intent by developing safe, dexterous motion plans for an articulated robot arm and executing the necessary actions in a courteous manner that avoids interrupting the human user while they are focused on completing task subgoals. PAL is task agnostic and can use modular task knowledge to quickly adapt to new applications. We will demonstrate the component capabilities of PAL in simulation and the complete system in a physical, representative manufacturing setting.

The goal of phase II is to continue the R&D efforts initiated in Phase I. Funding is based on the results achieved in Phase I and the scientific and technical merit and commercial potential of the project proposed in Phase II.

Z5

232056

SBIR_23_P2

Nasa Sbir Support Office