Mise | Devpost

Our full size logo
Our smaller logo
Landing page
Webcam recording component
Output component
Conversion page from ASL to text

Inspiration

After witnessing the daily challenges faced by our disabled friend, including communication barriers, we felt compelled to embark on a project aimed at alleviating these struggles. Our goal is to develop a solution that empowers individuals with disabilities, like our friend, by providing them with tools to overcome communication obstacles and navigate the world more independently. By leveraging technology, we aim to create a platform that facilitates smoother interactions and fosters inclusivity in society.

What it does

Creating a system that dynamically detects American Sign Language (ASL) and converts it to text involves several steps and can be a complex task.

How we built it

Using opencv and mediapipe were able to detect hands and landmark them. Using the data from the detections we built an LSTM machine learning model to convert the signs into text. We then passed this text through flask to our react.js front end which then got displayed with tailwind.css.

Challenges we ran into

In our project, putting all the pieces together was tough. Two parts gave us the most trouble: first, getting the OpenCV camera to show up on the website was difficult. It took a lot of trying different things. Second, making sure all the data was sent and received in real-time was tricky. We had to figure out a lot of stuff and make it work smoothly. But, we kept going and worked together, and in the end, we got it all to work! It's cool because now our project can help people with disabilities talk easier.

Accomplishments that we're proud of

We are most proud that we came out of this hackathon with happy faces and a working product. Often times working in a team with a tight deadline is stressful, however our team came together as one unit to create our final working product.

What we learned

Aside from all we learned about web applications and fullstack development we learned what it meant to be a team. Our conditions were far from perfect but we trudged on knowing we had each others backs.