1
Air Rhythm — Play music with your hands
Happy to share one of the projects I have been working on: Air Rhythm.
It’s a computer vision rhythm game where you play music by moving your hands in front of a webcam. In this demo, I’m playing a short version of Für Elise by catching falling notes with my fingertips.
How it works:
• Your fingertips control virtual drumsticks.
• Catching a falling circle plays its musical note.
• Following the note sequence lets you build the melody yourself.
• A live camera inset shows your hands and the tracked skeleton behind the interaction.
The computer vision behind it:
I used MediaPipe’s pretrained Hand Landmarker to detect 21 landmarks on each hand, then built the interaction around those predictions:
• Adaptive smoothing to reduce tracking jitter.
• Collision checks along fingertip movement between camera frames.
• Time-based note movement and sequence-aware scoring.
• Synthesized audio triggered by successful hits.
Tech stack:
• Desktop: Python, OpenCV, MediaPipe Hand Landmarker, NumPy, sounddevice, and pywebview.
• Browser: TypeScript, OpenCV.js, MediaPipe Hand Landmarker, and Web Audio.
• Shared interface: HTML, CSS, and Canvas.
• Hosting: GitHub Pages.
Most of the testing went into making quick movements register and reducing jitter. Partially visible hands still challenge the tracker, so the interface prompts users to keep their hands and wrists in view.
Both versions process camera footage locally on your device without uploading it.
Try the live web version: https://adolph1999v.github.io/air-rhythm/
GitHub repo, code, and documentation: https://github.com/Adolph1999v/air-rhythm
Give it a play and let me know what you think 😄
Add a comment
0/2000