List All Projects List Graduate Projects List Senior Design Projects List REU Projects Other Undergraduate Projects
Project # 166
Piano Hero
| Team Lead: | Leonardo Sampayo |
| Other Team Members: | Brian Villegas | Gerhard Bruck | Tianyao Su | Jatin Edla | Satvik Parvathareddy |
| Faculty Advisor(s): | Dr. Chenxi Wang |
| Department: | CSE |
| Project Type: | Senior Design |
| Title: | Piano Hero |
| Abstract: | Piano Hero is an augmented reality piano-learning system built in Unity and designed to run on an Android smartphone, bringing a Guitar Hero-style experience to any real piano. The goal is for falling notes to guide the player through songs while the system tracks their hands and checks whether they press the right keys at the right time, using only the phone's camera mounted above the keyboard with no special sensors required. Two open-source tools power the core functionality. MediaPipe, a real-time AI library developed by Google, tracks the player's hand and finger positions through the camera. OpenCV reads small printed markers placed at the piano's corners to calculate exactly where the keyboard surface sits in 3D space, giving the system a reliable frame of reference that works on any piano without additional hardware. Together, these tools allow a single Android camera to understand both where the piano is and where the player's fingers are, in real time. The current prototype demonstrates real-time hand tracking, ArUco marker-based surface detection, and zone-based key press recognition on Android, with a secondary camera view designed for AR glasses output. This lays the groundwork for a complete song-based learning experience. |
| Poster: | View Poster #166 (PDF) |
| Judges Scoring: | Submit Evaluation (judges only) |