CoTracker3: Simplified Point Tracking with Pseudo-Labeling by Meta AI

CoTracker3 is a point tracking model developed by Meta AI to accurately track multiple points in videos, even when those points are temporarily obscured or occluded. It simplifies previous models like TAPIR and CoTracker while improving data efficiency through pseudo-labeling, allowing it to train on real videos without annotations. This makes CoTracker3 more scalable and effective for real-world use compared to traditional models that rely on complex architectures and large synthetic datasets. The model's innovations include a simplified architecture with multi-layer perceptron (MLP) handling 4D correlation features, iterative update mechanism, cross-track attention for handling occlusions, and two operating modes: online and offline. CoTracker3 has been tested on several point tracking benchmarks and consistently outperforms previous models in terms of accuracy, efficiency, and data usage. Its applications span across 3D reconstruction, robotics, video editing, and special effects.