Abstract

We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of “important” image patches, while the task for the listener is to match the speaker’s message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification).

Cite

@inproceedings{gupta2021patchgame,
  title={PatchGame: Learning to Signal Mid-level Patches in Referential Games},
  author={Gupta, Kamal and Somepalli, Gowthami and Gupta, Anubhav and Jayasundara, Vinoj and Zwicker, Matthias and Shrivastava, Abhinav},
  journal={Advances in Neural Information Processing Systems},
  year={2021}
}