UNC Referring Expressions

Humans refer to objects in their environments all the time, especially in dialogue with other people. For example, one often uses referring expressions in everyday speech to indicate a particular person or object to a co-observer, e.g., “the man in the red hat” or “the book on the table”. In this project, we explore generating and comprehending natural language referring expressions for objects in images.

Use the Refer API to load 4 datasets: RefCLEF, RefCOCO, RefCOCO+, RefCOCOg.

Based on the above datasets, there are two tasks proposed:

Referring Expression Generation: Generate a referring expression describing the target region.
Referring Expression Comprehension: Localize the object mentioned by the given referring expression.

	MAttNet: Modular Attention Network for Referring Expression Comprehension CVPR 2018 Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg [Paper] [Demo] [Code]
	A Joint Speaker-Listener-Reinforcer Model for Referring Expressions CVPR 2017 Licheng Yu, Hao Tan, Mohit Bansal, Tamara L. Berg [Paper] [Code] [Project Page] [Talk] (Spotlight presentation 8%)
	Modeling Context in Referring Expressions ECCV 2016 Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, Tamara L. Berg [Paper] [Dataset] [Talk] (Spotlight presentation 4.7%)
	ReferItGame: Referring to Objects in Photographs of Natural Scenes EMNLP 2014 Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, Tamara L. Berg [Paper] [Dataset] [Project Page] (Oral)