Deaflingo

Duolingo meets sign language--Learning sign language should be more than a Google search, and we've made it possible for just your webcam to recognize your hands and make your technique better.

Comment 1

Inspiration

We saw an American Sign Language book and thought to ourselves how egregious and mundane it would be to learn in such a pedantic way. So, we came up with a new way to learn sign language: have a computer teach you. Not to mention, the four of us had the littlest clue about any sign language knowledge, and were even more eager to cleverly force our way into it.

What it does

Think Duolingo or any typing game, but with sign language. The web app uses a device's built-in camera and pulls data in real-time to detect one's hands. We then use a model to detect which "sign" is being produced, and the user has the option to also spell out words in an effort of accuracy and speed (much like type racing). However, if they don't, all good. We themed the whole website around monkeys who will give you helpful tips on how to improve!

How we built it

We first began by trained our own model for recognizing hand signs based on a dataset publicly available that has over 200,000 images of ASL letters. We decided to stick with the simple A-Z alphabet for simplicity and a higher-level approach to the idea. This was done using Python libraries and technologies such as TensorFlow, Keras, MediaPipe, and OpenCV. After successfully training a model, we moved onto using a web app to retrieve webcam data from the browser and process in the backened. We used the Python Flask framework along as AJAX techniques to communicate via HTTP requests with the webpage created in a simple stack of HTML/CSS/JS. This proved helpful in eliminating reloads and processing the camera data.

Challenges we ran into

1) Training the dataset. Three out of four of us have Silicon Macs, which proved to be a nightmare for developing a solution to making a model of any sorts. However, we were lucky to stumble upon silicon-compatible versions of some of the technologies that we used. We also naively decided to make our own dataset initially but realized it would be much more practical to find one publicly available.

2) We decided to make all of our own vector graphics, which took a good amount of time.

Accomplishments that we're proud of

Training an entire model to recognize hand gestures and symbols within the timeframe that we had (having also had virtually zero experience with TensorFlow or related tech), we were incredibly pleased with how it turned out. We were also proud of how we used the browser's webcam and interfaced it with Python.

What we learned

Markus: How to dynamically communicate between the front end and back end without refreshing the page and rendering static templates with Jinja, instead using AJAX and saving files to the web app which Flask can then give to the front end via HTTP requests.

Lane: I had a lot of fun working with TensorFlow and the MediaPipe library to recognize hand features. At first, I approached it with Haar Cascades, however, I quickly realized there were too many false positives and it was not practical for our use case. So, creating our own model completely from scratch resulted in a good deal of knowledge learned.

Syoma: I dabbled in UI before, but this project really pushed me to create dynamic elements, like those cartoon monkey eyes! It's been a challenging but rewarding learning experience in web development, especially for captivating interfaces and vector graphics. I'm excited to use these new skills and keep growing as a developer!

Andrei: During the hackathon, I enhanced my front-end skills in CSS, HTML, and JavaScript, while learning teamwork and time management. This hands-on experience improved my problem-solving in a fast-paced environment.

We learned sign language! Seriously, we are somewhat fluent in basic ASL strictly because of our sick website <3

What's next for Deaflingo

While the alphabet is cool, we look to expand to the entire scope of ASL (complete phrases, etc.). We also would like to add more functionality such as a competitive aspect (1v1 sign language) that includes leaderboards, as well as phrase integration for quotes. In doing so, our model would also become drastically more capable. Ultimately, our goal is to make learning sign language genuinely a fun experience.

Built With

+ 2 more
Share this project:

Updates