Exploring Video Search with OpenOrigins: Frame Search Versus Multi-Modal Embeddings

OpenOrigins is developing a platform to help archivists quickly and efficiently find relevant videos in digital media archives by providing advanced search capabilities. The company is considering two technological approaches: frame-by-frame analysis of videos using image embeddings, and multimodal embeddings. While the former offers high accuracy in multimodal semantic search but may miss temporal context or changes between frames, the latter leverages Google's multimodal embedding model to enable users to search videos using images, text, or videos, converting all inputs into a common embedding space. This approach efficiently manages large datasets with temporal context and supports multiple input types for search queries, making it an excellent choice for complex search scenarios.