List All Projects List Graduate Projects List Senior Design Projects List REU Projects Other Undergraduate Projects
Project # 92
A Hierarchical Multi-Agent Architecture for Vision-Based Interaction in Mixed Reality
| Team Lead: | Song Tu Linh Do |
| Other Team Members: | |
| Faculty Advisor(s): | Jiayi Meng |
| Department: | CSE |
| Project Type: | REU |
| Title: | A Hierarchical Multi-Agent Architecture for Vision-Based Interaction in Mixed Reality |
| Abstract: | Enabling intelligent vision-based interaction in Mixed Reality (MR) requires coordination between language reasoning systems and real-time spatial processing. This work introduces a hierarchical multi-agent system that connects an LLM-powered backend with image analysis in a Unity MR environment to process vision-related user queries. When a query is received, a supervisor agent determines the task type and routes it to a specialized vision agent. The vision agent invokes a tool that communicates with Unity to perform image detection. Unity returns structured JSON data containing detected objects and spatial metadata, which is passed back to the agent to generate a grounded natural-language response. The system highlights coordinated agent routing, data flow, and separation between query interpretation, external processing, and response generation. The architecture focuses on organizing communication between agents and external tools to support vision-aware interaction in spatial computing environments. |
| Poster: | View Poster #92 (PDF) |
| Judges Scoring: | Submit Evaluation (judges only) |