PhotoGraph

the landing page to upload your media
the knowledge graph, summary on left
the slideshow of your memories

Inspiration

We were really interested in the idea of combining the power of multimodality and knowledge graphs, and after brainstorming different ideas we came up with the idea of creating a new, fun, interactive way to view your memories.

What it does

You upload your photos and videos from your photo gallery, which we encode and create an interconnected knowledge graph that you can interact with. On hovering over a vertex, and get a summary of all your memories that surround it. If you click on the vertex, we show you a presentation of all the related memories.

How we built it

For each batch of your photos and videos, we use Gemini's multimodality to extract relations in a JSON format. After some post processing to remove duplication, we update the knowledge graph with the information. We used an SQL database to store the knowledge graph using an Edge List structure. We used threejs webgl to render the graph to the screen making the object 3 dimensional and interactive.

Challenges we ran into

We had trouble trying to display the knowledge graph as well as have interactive features in the frontend. For the backend, it was difficult understanding how we can add multimodality to a knowledge graph and trying to extract meaningful relations. One thing we failed to do is to detect different faces and entities to store into the vertices and hosting, which would have significantly enhanced the quality of the graph.

Accomplishments that we're proud of

We are proud to have a project that contains most of what we initially ideated on and having a fully working project and trying out many new technologies that we had to dive deep into documentation for.

What we learned

We also learned a lot of new technologies such as Google file storage, file handling, Supabase, multithreading/async processing and most importantly the capabilities of Gemini 1.5 Pro

What's next for PhotoGraph

Be able to extract more types of relations such as entity, temporal, audio (apart from speech), geospatial relations and more.
Host the website, or migrate to an app that connects to google photos to easily add memories into the graph
More interactivity such as asking questions about your memories which then we provide images, videos and a graph path to answer it.
Create presentations similar to google photos with more than one vertex.