UMassAmherst_VanHorn

Fine-grained visual classification has blossomed under the advances in large-scale image datasets like iNaturalist as well as algorithmic contributions like ResNet, ViT, Vision-Language Models (VLMs) like CLIP, and Multimodal Large Language Models (MLLMs). However, VLMs and MLLMs have garnered increased interest in FGVC due to the surprising fact that underperform more classical, simpler, and smaller approaches. We currently are investigating the root causes and solutions for this underperformance, focusing on making sure that VLM/MLLM responses are visually-grounded (eg. fine-tuningt), figuring out more faithful ways to evaluate their responses (evaluation procedures and benchmarks), and being able to steer predictions with expert knowledge (Visipedia).