What it does
Visually impaired or blind people have difficulty crossing busy roads. The app we have created empowers them by detecting road crossings and incoming cars and communicating the information audibly to the user.
How we built it
Our app is written with next.js and either sends live camera frames or frames of a video to a Flask server. The Flask server handles the processing of the data in three parts. First, using OpenCV, it detects the crosswalk lines and extrapolates to find the centerline. Then, using a Keras classification model we trained, it uses the traffic signals to figure out whether it is time to cross the road. It additionally uses a YoloV8 model to detect any cars that are near the crosswalk. The resulting data is sent back to the app, which communicates it to the user using text-to-speech.
Challenges we ran into and solutions
The phone couldn't process the data fast enough --> Data was sent to and processed on a separate server Sending the image over the network was too slow --> Using better compression and lowering the resolution (and compensating with better processing code) Inference with the YOLO model was slow CV wasn't nearly accurate enough to get traffic lights, especially at a difference --> Trained a Keras model for classification
What's next for CrossSight
Testing and training models for various types of crosswalks Faster detection Haptic motors and a more integrated experience for the user Ready-to-release mobile app
Log in or sign up for Devpost to join the conversation.