Eduardo

Golf Putt

What we made

We developed a unique, educational learning experience to teach users with animations and videos. This platform leverages augmented reality to guide users through learning new activities with visual and auditory feedback for an immersive learning journey.

Inspiration

Learning is challenging, even for tasks now considered basic. Humans often learn by observing others and repeating what we see. Augmented reality, with its ability to overlay animations onto the real world, simplifies the learning process significantly. This approach removes the need to constantly shift attention and head movements between watching a video and applying feedback, allowing learners to focus more effortlessly. Furthermore, augmented reality allows users to see how their actions are supposed to look from every angle. This allows users to get a better understanding of each action.

Recently, when I attempted to put on snowboard boots for the first time, I experienced this firsthand. Despite their resemblance to regular shoes, the process was unexpectedly challenging. The YouTube videos I watched failed to provide clear guidance, as they didn't effectively demonstrate the necessary angles for inserting and securing my feet. An animated overlay would have offered better clarity, enabling me to adjust orientation and focus on specific steps. Additionally, juggling focus between my shoe and phone screen proved confusing, prompting me to place my phone on the snow to minimize head movements.

Looking ahead, our product aims to leverage mixed reality platforms, providing users with real-world integration alongside instructional animations. Imagine encountering a choking person with no one else around – you could access the Heimlich maneuver animation in real-time, learning and assisting simultaneously. This unique approach not only enhances learning but also empowers users to help others in critical situations.

Impact

This is a revolutionary educational learning experience. This has the potential of being the premier learning system in the future. Because recent studies have found that the average human attention span is decreasing, we are targeting learning in small bursts. We believe that this key change will drive usership up significantly.

Furthermore, this product has nearly unlimited learning based use cases ranging from healthcare to sports. Whether it is learning how to shoot a basketball or how to do a situp, seeing how an action is meant to be done alongside yourself can make it much easier to learn a new skill. We imagine this being used for medical purposes as well such as doing CPR or the Heimlich maneuver.

How we built it

First, we processed the speech input using a speech-to-text API. Then, we generated a text embedding for this text using Gemini and compared it to our Pinecone database of text embeddings for every animation. We found the most similar embedding and processed this embedding to ensure it was accurate. If this embedding passed our checks, we loaded the corresponding animation overlayed onto the camera feed. Each animation was downloaded using the Mixamo database. As soon as the animation properly loads into the frame, we play corresponding audio explaining how to perform the action. This audio is generated by first using Gemini to generate a description of the action. Then we played the audio using an API enabled through Unity.

Challenges we ran into

We ran into many challenges during the development process, the most significant of which was successfully deploying our APIs on a public URL and then integrating it with Unity. We built two APIs using the FastAPI framework: one for creating the text and a second for outputting the text as audio. Despite both working locally and being deployed on the same file, the API to generate audio did not properly interact with Unity. We ended changing everything and not using the API we created. Instead, we used AWS with Polly for text to speech.

Accomplishments that we're proud of

We successfully developed the first working prototype for Eduardo, demonstrating both its current capabilities and future potential. By seamlessly integrating augmented reality to overlay animations onto the real world, we've showcased the power and versatility of the product.

Witnessing the foundation of Eduardo take shape is truly exciting, and we're filled with enthusiasm for Eduardo's journey ahead. We anticipate its growth and evolution, eagerly envisioning the expanded capabilities and impact it will have on users' learning experiences.

What's next for Eduardo

Our primary objective for the future of Eduardo is to achieve a state of limitless animations. Currently, the application is constrained to pre-existing animations, which are stored in Unity and called upon as needed. However, this approach imposes limitations on the variety and number of animations available, thereby restricting the application's capabilities.

To address this limitation, we plan to transition away from relying solely on predetermined animations sourced from platforms like Mixamo. Instead, we aim to adopt alternative methods for animation creation that offer greater flexibility and customization. Our vision is to empower users to prompt their own unique 3D animations, thereby enhancing the personalization and adaptability of Eduardo.

To accomplish this goal, we believe that leveraging the Move AI API represents the most promising solution. This shift towards user-driven animation creation will propel Eduardo towards its goal of offering an unparalleled and truly limitless learning experience.