- https://arxiv.org/pdf/2305.11175.pdf
- Source: arxiv
TL;DR
VisionLLM unifies detection, segmentation, captioning, and VQA under a single LLM autoregressive decoder by expressing all tasks as natural language instructions, eliminating task-specific heads entirely.
Summary
Vision models traditionally require separate task-specific heads for detection, segmentation, captioning, and VQA. VisionLLM eliminates this by routing all vision tasks through a single LLM autoregressive decoder, where task instructions are natural language commands and outputs — bounding box coordinates, class tokens, and text — are generated in the same decoding pass. This enables open-ended instruction following for vision tasks without architectural switches. On COCO detection, VisionLLM surpasses specialist models while remaining competitive on captioning, all from one unified architecture.
Key contributions
- Proposes using a single LLM decoder for heterogeneous vision tasks without any task-specific heads.
- Introduces natural language task instructions as the universal interface replacing task-specific prompts and switches.
- Demonstrates autoregressive coordinate prediction as a viable alternative to regression-based detection heads.
- Shows that a unified architecture surpasses specialist models on COCO detection while remaining competitive on captioning.
When to cite
- When arguing that LLM decoders can generalize across heterogeneous vision output types (coordinates, classes, text).
- When comparing unified generalist architectures against task-specific vision heads for detection and segmentation.
- When discussing open-ended instruction following as a paradigm for multi-task vision models.
- When looking for evidence that autoregressive bounding box prediction matches regression-based detection at scale.