Florence-2: How it works and how to use it

Microsoft's new large vision model (LVM), Florence-2, is a significant step towards the goal of a unified vision model. It demonstrates impressive results with a compact, parameter-efficient model and can perform a wide variety of image-language tasks such as captioning, optical character recognition, object detection, region detection, region segmentation, vocabulary segmentation, and more. Florence-2 follows the "playbook" of large language models (LLMs) research by building on top of other recent vision research to learn general representations that are useful for many tasks. It is designed in a simple way - to take in textual prompts (in addition to the image being processed), and generate textual results. The architecture unifies the way diverse types of information, such as masked contours, locations, etc., are input to the model, permitting a unified training procedure and easy extension to other tasks without the need for architectural modifications.