Project Nirvana

Using the power of Computer Vision to shed a light on your air drumming skills!

Comment

Inspiration

Although my habit of mindlessly drumming in the air while listening to music leaves people shooting judging looks my way, it does often leave me wondering what I would actually sound like if I had a real drum kit in front of me.

What it does

The actual code isn't fully complete but it is functional as a 'proof of concept'.

The webpage starts a webcam stream and the script detects hand gestures in specific regions and plays a corresponding drum sound accordingly. The video is split into quadrants:

Upper Left Quadrant -> A 'point' gesture in the upper left quadrant results in a Hi-Hat sound. Upper Right Quadrant -> A 'point' or 'closed fist' gesture in the upper right quadrant sounds a Snare Drum. Lower Right Quadrant -> A 'closed fist' gesture in the lower right quadrant gives a Bass Drum sound.

The webpage also has a live model detection feed that illustrates what features the model has detected.

How I built it

I used the Handtrack.js module (built on Tensorflow.js) to detect objects via the webcam. The script filters the detections so as to only return the gestures involving hands (point, pinch, closed fist, open palm etc) and then extracts the position of the hand on the screen in (x,y) format.

I split the stream into quadrants. The script constantly checks for the gestures and their positions on the screen and based on the position and gesture, a corresponding drum sound is played.

Challenges I ran into

The initial plan was to use the mediapipe hands module rather than handtrack.js, but the implementation in a web format proved too challenging for me and I was unable to get it working.

Accomplishments that I'm proud of

This was my first ever hackathon, so honestly I'm just proud I managed to get a proof of concept result done.

What's next for Project Nirvana

I'm going to iron out the bugs and optimise my script to detect regions more accurately and also incorporate more sounds.

Built With

Share this project:

Updates