PCoreSet：通过视觉-语言模型知识蒸馏实现高效主动学习

PCoreSet: Effective Active Learning through Knowledge Distillation from Vision-Language Models

June 1, 2025

作者: Seongjae Kang, Dong Bok Lee, Hyungjoon Jang, Dongseop Kim, Sung Ju Hwang

cs.AI

摘要

知识蒸馏（KD）是一种广泛应用的框架，通过利用教师模型的知识来训练紧凑、任务特定的模型。然而，其在主动学习（AL）中的应用，即通过迭代样本选择以最小化标注成本，仍未被充分探索。这一空白源于KD通常假设有充足的标注数据，而AL则运作于数据稀缺的场景中，其中任务特定的教师模型往往不可得。本文提出ActiveKD框架，通过利用大规模视觉-语言模型（VLMs）的零样本和少样本能力，将AL与KD相结合。ActiveKD的一个关键方面是VLMs的结构化预测偏差——即其预测在概率空间中形成聚类。我们将此结构视为教师模型的归纳偏差，捕捉对学生学习有益的可泛化输出模式。为利用这一偏差，我们提出了概率核心集（PCoreSet），一种在概率空间而非特征空间中最大化覆盖的选择策略。PCoreSet策略性地选择类别多样的未标注样本，在有限标注预算下促进教师知识更高效的传递。在11个数据集上的评估表明，PCoreSet在ActiveKD框架内持续超越现有选择方法，推动了AL与KD交叉领域的研究进展。

English

Knowledge distillation (KD) is a widely used framework for training compact, task-specific models by leveraging the knowledge of teacher models. However, its application to active learning (AL), which aims to minimize annotation costs through iterative sample selection, remains underexplored. This gap stems from the fact that KD typically assumes access to sufficient labeled data, whereas AL operates in data-scarce scenarios where task-specific teacher models are often unavailable. In this paper, we introduce ActiveKD, a framework that integrates AL with KD by leveraging the zero- and few-shot capabilities of large vision-language models (VLMs). A key aspect of ActiveKD is the structured prediction bias of VLMs -- i.e., their predictions form clusters in the probability space. We regard this structure as an inductive bias of the teacher model, capturing generalizable output patterns beneficial to student learning. To exploit this bias, we propose Probabilistic CoreSet (PCoreSet), a selection strategy that maximizes coverage in the probability space rather than the feature space. PCoreSet strategically selects categorically diverse unlabeled samples, facilitating more efficient transfer of teacher knowledge under limited annotation budgets. Evaluations on 11 datasets show that PCoreSet consistently outperforms existing selection methods within the ActiveKD framework, advancing research at the intersection of AL and KD.