Informative Object Annotations (IOTA) - CVPR 2019
Lior Bracha and Gal Chechik
Abstract
Capturing the interesting components of an image is a key aspect of image understanding. When a speaker annotates an image, selecting labels that are informative greatly depends on the prior knowledge of a prospective listener.
Motivated by cognitive theories of categorization and communication, we present a new unsupervised approach to model this prior knowledge and quantify the informativeness of a description. Specifically, we compute how knowledge of a label reduces uncertainty over the space of labels and utilize this to rank candidate labels for describing an image. While the full estimation problem is intractable, we describe an efficient algorithm to approximate entropy reduction using a tree-structured graphical model. We evaluate our approach on the open-images dataset using a new evaluation set of 10K ground-truth ratings and find that it achieves 65% agreement with human raters, largely outperforming other unsupervised baseline approaches.
Data
Ground-truth data - IOTA-10K (raw)
Image-Level Labels from the Open Images Dataset
Method
Our poster from CVPR 2019
Chow-Liu Tree (vocabulary of 765 words)