BLIP-2 Q-Former

A lightweight query transformer that bridges the modality gap between frozen image encoders and large language models.

BLIP-2 leverages the Q-Former to extract fixed-length visual features from a frozen ViT-g/14 encoder, feeding them directly into LLMs like Vicuna or Flan-T5. This architecture uses a two-stage pre-training strategy: first, it forces the Q-Former to learn visual representations most relevant to text; second, it trains the module to act as an informative bottleneck for language generation. By keeping the heavy-duty backbones frozen, BLIP-2 achieves state-of-the-art zero-shot VQA performance with 54x fewer trainable parameters than Flamingo, making high-tier multimodal reasoning computationally accessible.

https://arxiv.org/abs/2301.12597