Video Understanding with FiftyOne and Steamboat Willie

Video understanding is an important yet complex area of study in computer vision due to its multiple modalities, time-series elements, open-ended questions, and answers. With the increasing amount of video data available, AI models that can parse through millions of videos to understand if it falls into a set search criteria are crucial for efficient data processing, training, or storage. Two main approaches to understanding videos are Vision Question and Answering (VQA) and Action Recognition. VQA LLMs have vast knowledge of human language and context in images but can only work on frames or single image-based prompts. On the other hand, action recognition models can take in video and understand the time component of data but lack depth in responses and are limited to a set number of classified actions. For general understanding of video datasets, VQA is currently considered more effective than action recognition models. However, the evolution of Video Understanding remains an unsolved problem with many potential approaches being explored.