PatchGame - Learning to Signal in Referential Games

NeurIPS 2021

University of Maryland, College Park

HTML5 Icon


We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of “important” image patches, while the task for the listener is to match the speaker’s message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification).


  title={PatchGame: Learning to Signal Mid-level Patches in Referential Games},
  author={Gupta, Kamal and Somepalli, Gowthami and Gupta, Anubhav and Jayasundara, Vinoj and Zwicker, Matthias and Shrivastava, Abhinav},
  journal={Advances in Neural Information Processing Systems},
Last updated on September 28, 2021