Hologlass (Code Green)

Imagine having the ability to transcribe, and recall your conversations, hands-free, in real-time. That's Hologlass – your DIY Google Glass alternative, made to enhance your communication, and memory

Comment 2

🌟 Inspiration

At the heart of HoloGlass lies a profound inspiration: the belief that technology should not only serve us but elevate our human experience. Our journey is driven by the desire to empower individuals, enhance connections, and enrich lives in ways previously unimaginable.

🌐 What it does

HoloGlass swfitly transcribes conversations in real-time, displaying them directly into your field of vision. Utilizing some of the best modern language models, we partition and summarize all your previous conversations.

🛠️ How we built it

The foundation of HoloGlass rests on a robust tech stack. We harnessed both AWS Amazon Transcribe for audio-to-text conversion and Google Cloud Transcription to be able to follow a live conversation word by word with great accuracy. We choose to work with AWS and Google models so that we may compare the two and see which is quicker and more accurate.

Cohere's dialogue summarization technology condensed conversations into key insights. Google Cloud Speech-to-Text provided real-time transcription capabilities. Streamlit served as our UI framework, offering a user-friendly interface. Python, alongside libraries like sounddevice and soundfile, managed audio recording and processing.

Supabase ensured secure data management. This intricate integration and user-centric design formed the backbone of HoloGlass, delivering an experience that marries technology with human interaction, all while upholding rigorous ethical and privacy standards.

🧠 Challenges we ran into

This was a first-time endeavor for many of us, both in terms of developing hardware and ensuring its seamless integration with our software program. The complexities of hardware-software synergy posed a learning curve, demanding a deeper understanding of the intricacies involved in creating a holistic solution.

🦾 The Hardware

The hardware side provided its own share of troubles. Starting with the brain of the system, the ardunio, we wanted to make sure that the connection the ardunio had with the rest of the system was wireless in order to make the system more portable and light weight. However, connecting the ardunio via bluetooth was not simple as it may sound. Moving on to the reflection instruments (glass and mirrors), the exact order in which each glass is placed greatly effects the visibility of any text or images displayed on the users field of vision. This is why after experimenting with many different compositions we settled on using a magnifying glass to first magnify the text then reflect that into the rest of the system.

🤖 The Software

The software side has gone through many different phases of experimentation. We first started with using two different transcribe models (AWS, and google cloud) in order to have options later down the line of which to choose. The biggest struggle of all was live speaker diarisation. Usually speaker diarazation involves having enough context in the conversation to be able to identify the number of people speaking in one conversation. However, when we are transcribing real time, we run into the problem of there not being enough context or time to tell who is speaking. To try and combat this we had the program continuously record our program to try and run a successful partitioning of the speakers. Although this wasn't able to be fully implemented it was fun and noteworthy challenge in our development process. Another layer of the software side was the bridge between the hardware and the software. This was originally a very big concern of ours however by utilizing supabase and different python libraries we were able to sucessfully establish communication and store that data on the web.

🚀 Accomplishments that we're proud of

By far the biggest accomplishment was not only completing what we had set our minds on but also being able to add on to our idea in a way we have not thought about before. At first, our idea was to simply show the words in the users field of vision as they are speaking, however that later brought upon the idea of having a central application to be able to view your conversation later on as well as an accompanying AI generated summary.

One big accomplishment that may be overlooked at first glance is the DIY part of the project. The parts evolved were very much affordable. This could give us the chance to potentially standardize the kit and make it available to the public.

✍️ What we learned

From the tinkering around with the hardware as well as the debugging of the software it is safe to say that we are going to leave this hackathon better equipped and more experienced then when we came in. For starters we did not expect the configuration of the glass to be too hard to figure out but as it turns out there are many things we did not take into consideration at first. For the software we have grown to like Google API services vs AWS because of its simplicity in the setup process as well as its performance. We have also learned about the useful capabilities of Streamlit that allowed us to display our data in a professional user interface. Supabase is a definite life savor and will most certainly be used by us in our future projects.

🤝 What's next for Hologlass

It would be amazing to be able to standardize the materials for this project and be able to provide it to people at a cheap cost. This would allow so many people to get the opportunity to not only learn the components involved in making the Hologlass but also have a useful tool to be used in their every day life. It would also be amazing to be able to refine the application part of this project so that the user can go through their past conversations on any system of their choosing.

Built With

Share this project:

Updates