Face model compression by distilling knowledge from neurons
Refereed conference paper presented and published in conference proceedings


Full Text

Times Cited

Other information
AbstractThe recent advanced face recognition systems were built on large Deep Neural Networks (DNNs) or their ensembles, which have millions of parameters. However, the expensive computation of DNNs make their deployment difficult on mobile and embedded devices. This work addresses model compression for face recognition, where the learned knowledge of a large teacher network or its ensemble is utilized as supervision to train a compact student network. Unlike previous works that represent the knowledge by the soften label probabilities, which are difficult to fit, we represent the knowledge by using the neurons at the higher hidden layer, which preserve as much information as the label probabilities, but are more compact. By leveraging the essential characteristics (domain knowledge) of the learned face representation, a neuron selection method is proposed to choose neurons that are most relevant to face recognition. Using the selected neurons as supervision to mimic the single networks of DeepID2+ and DeepID3, which are the state-of-The-Art face recognition systems, a compact student with simple network structure achieves better verification accuracy on LFW than its teachers, respectively. When using an ensemble of DeepID2+ as teacher, a mimicked student is able to outperform it and achieves 51.6 × compression ratio and 90 × speed-up in inference, making this cumbersome model applicable on portable devices.
All Author(s) ListLuo P., Zhu Z., Liu Z., Wang X., Tang X.
Name of Conference30th AAAI Conference on Artificial Intelligence, AAAI 2016
Start Date of Conference12/02/2016
End Date of Conference17/02/2016
Place of ConferencePhoenix
Country/Region of ConferenceUnited States of America
Detailed descriptionorganized by AAAI,
Year2016
Month1
Day1
Pages3560 - 3566
ISBN9781577357605
LanguagesEnglish-United Kingdom

Last updated on 2021-05-12 at 23:48