Open-world object counting leverages the robust text-image alignment of
pre-trained vision-language models (VLMs) to enable counting of arbitrary
categories in images specified by textual queries. However, widely adopted
naive fine-tuning strategies concentrate exclusively on text-image consistency
for categories contained in training, which leads to limited generalizability
for unseen categories. In this work, we propose a plug-and-play Semantic-Driven
Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the
training set to unseen categories with minimal overhead in parameters and
inference time. First, we introduce a two-stage visual prompt learning strategy
composed of Category-Specific Prompt Initialization (CSPI) and Topology-Guided
Prompt Refinement (TGPR). The CSPI generates category-specific visual prompts,
and then TGPR distills latent structural patterns from the VLM’s text encoder
to refine these prompts. During inference, we dynamically synthesize the visual
prompts for unseen categories based on the semantic correlation between unseen
and training categories, facilitating robust text-image alignment for unseen
categories. Extensive experiments integrating SDVPT with all available
open-world object counting models demonstrate its effectiveness and
adaptability across three widely used datasets: FSC-147, CARPK, and PUCPR+.

Este artículo explora los viajes en el tiempo y sus implicaciones.

Descargar PDF: