Baylee • Autonomous Care Robot

Built 36-hour at the University of Toronto Robotics Association hackathon (UTRA Hacks 2025), Baylee is a mobile companion robot inspired by Baymax. The project combines real-time facial emotion recognition with conversational AI and physical actuation to provide both emotional comfort and tangible health supplies.

What It Does

• Emotion Detection & Support: Uses an Intel RealSense camera and OpenVINO/Open Model Zoo models to classify user expressions (sadness, distress/anger, happiness, surprise).
• Physical Dispensing: Actuates three motorized linear slider compartments to dispense tissues when you're sad, medication when needed, or bandages and sanitizer for first aid.
• Conversational Care: Routes detected emotional state and voice input through a local LLM (Llama via Ollama) with tool-calling to trigger physical robot actions, guide breathing exercises during stress, or celebrate when the user is happy.
• Autonomous Navigation: Reads Stereo Vision data on a Raspberry Pi to scan room obstacles and drive toward users.

How We Built It

We split our work across hardware and software sub-teams during the 36-hour hackathon:

• Chassis & Mechanics: Rapidly fabricated the enclosure using 3D-printed gears, standoffs, linear sliding racks, foam board, and acrylic.
• Electronics: Connected a Raspberry Pi 4 to multiple Arduino Uno and Nano boards driving the gearmotors, RealSense camera, OLED mood faces, and status LCDs.
• Software Pipeline: A Python vision service continuously streams facial emotion telemetry into a TypeScript backend. The LLM evaluates sentiment and dispatches hex command packets over UART to the Arduino motor controllers.

Real Challenges & What We Learned

• Pre-voice Model Era: This hackathon took place before the release of voice capable LLMs, so we had to implement a custom speech-to-text and text-to-speech pipeline while filtering the llms own output.
• Micro-Service Management: Instead of running a full instance of ROS we took inspiration and used lightweight microservice architecture routing everything though redis pub/sub. This allowed us to iterate far faster splitting the work between multiple team members and languages.

Results

• Won Best Use of Generative AI at UTRA Hacks 2025 among dozens of university robotics entries.