Mode-Seeking Divergences: Theory and Applications to GANs
Refereed conference paper presented and published in conference proceedings
CUHK Authors
Other information
AbstractGenerative adversarial networks (GANs) represent a game between two neural network machines designed to learn the distribution of data. It is commonly observed that different GAN formulations and divergence/distance measures used could lead to considerably different performance results, especially when the data distribution is multi-modal. In this work, we give a theoretical characterization of the mode-seeking behavior of general f-divergences and Wasserstein distances, and prove a performance guarantee for the setting where the underlying model is a mixture of multiple symmetric quasiconcave distributions. This can help us understand the trade-off between the quality and diversity of the trained GANs’ output samples. Our theoretical results show the mode-seeking nature of the Jensen-Shannon (JS) divergence over standard KL-divergence and Wasserstein distance measures. We subsequently demonstrate that a hybrid of JS-divergence and Wasserstein distance measures minimized by Lipschitz GANs mimics the mode-seeking behavior of the JS-divergence. We present numerical results showing the mode-seeking nature of the JS-divergence and its hybrid with the Wasserstein distance while highlighting the mode-covering properties of KL-divergence and Wasserstein distance measures. Our numerical experiments indicate the different behavior of several standard GAN formulations in application to benchmark Gaussian mixture and image datasets.
Acceptance Date11/02/2023
All Author(s) ListCheuk Ting Li, Farzan Farnia
Name of ConferenceInternational Conference on Artificial Intelligence and Statistics
Start Date of Conference25/04/2023
End Date of Conference27/04/2023
Place of ConferenceValencia
Country/Region of ConferenceSpain
Proceedings TitleProceedings of The 26th International Conference on Artificial Intelligence and Statistics
Year2023
Month4
Volume Number206
Pages8321 - 8350
LanguagesEnglish-United States
