short-paper

Teacher, trainee, and student based knowledge distillation technique for monitoring indoor activities: poster abstract

Authors:

Rahul Mishra,

Hari Prabhat Gupta,

Tanima DuttaAuthors Info & Claims

SenSys '20: Proceedings of the 18th Conference on Embedded Networked Sensor Systems

Pages 729 - 730

https://doi.org/10.1145/3384419.3430450

Published: 16 November 2020 Publication History

Get Access

Abstract

Recent years have witnessed unprecedented growth in sensors-based indoor activity recognition. Further, a significant improvement in recognition performance of indoor activities is observed by incorporating Deep Neural Network (DNN) model. In this paper, we propose knowledge distillation based economic and efficient indoor activity recognition approach for low-cost resource constraint devices. Here, we adopt knowledge from teacher and trainee (cumbersome DNN models) for training student (compressed DNN model). Initially, student and trainee both are beginner and trainee helps the student in learning from the teacher. The student, after certain steps, is mature enough for directly learning from the teacher. We introduce an early halting mechanism for simultaneously reducing floating-point operations and training time of the student model.

References

[1]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.

Google Scholar

[2]

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proc. CVPR, pages 4133--4141, 2017.

Crossref

Google Scholar

[3]

Chungkuk, Bumsoo Yoo, Minsik Kang, and Cho. Snow: Subscribing to knowledge via channel pooling for transfer & lifelong learning of convolutional neural networks. In Proc. ICLR, pages 1--12, 2019.

Google Scholar

[4]

A. Mishra and D. Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In Proc. ICLR, pages 1--15, 2017.

Google Scholar

[5]

H. Zhao, X. Sun, J. Dong, C. Chen, and Z. Dong. Highlight every step: Knowledge distillation via collaborative teaching. IEEE Transactions on Cybernetics, pages 1--12, 2020.

Google Scholar

[6]

J. Yang, H. Zou, S. Cao, Z. Chen, and L. Xie. Mobileda: Toward edge-domain adaptation. IEEE Internet of Things Journal, 7(8):6909--6918, 2020.

Crossref

Google Scholar

Cited By

View all

Hohman FKery MRen DMoritz D(2024)Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning ExperiencesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642109(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642109
Mishra RGupta H(2024)Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge DistillationIEEE Transactions on Mobile Computing10.1109/TMC.2023.3297026(1-12)Online publication date: 2024
https://doi.org/10.1109/TMC.2023.3297026
Mishra RGupta H(2023)A Model Personalization-based Federated Learning Approach for Heterogeneous Participants with Variability in the DatasetACM Transactions on Sensor Networks10.1145/3629978Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1145/3629978

Index Terms

Teacher, trainee, and student based knowledge distillation technique for monitoring indoor activities: poster abstract
1. Computing methodologies
  1. Machine learning

Recommendations

Enhanced Accuracy and Robustness via Multi-teacher Adversarial Distillation
Computer Vision – ECCV 2022
Abstract
Adversarial training is an effective approach for improving the robustness of deep neural networks against adversarial attacks. Although bringing reliable robustness, adversarial training (AT) will reduce the performance of identifying clean ...
Knowledge Distillation based Online Learning Methodology using Unlabeled Data Stream
MLMI '18: Proceedings of the 2018 International Conference on Machine Learning and Machine Intelligence

In supervised learning, the performance of the learning model decreases with the change of time step due to concept drift caused by overfitting of the training data. As a methodology to mitigate such concept drift, an online learning methodology has ...
Robust knowledge distillation based on feature variance against backdoored teacher model
Abstract
Benefiting from large well-trained deep neural networks (DNNs), model compression has captured special attention for computing resource limited equipment, especially edge devices. Knowledge distillation (KD) is one of the widely used compression ...
Highlights
- A robust knowledge distillation method.
- Removing backdoors from models during knowledge distillation.
- Defense against backdoors in teacher models.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SenSys '20: Proceedings of the 18th Conference on Embedded Networked Sensor Systems

November 2020

852 pages

ISBN:9781450375900

DOI:10.1145/3384419

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 November 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

SERB

Conference

SenSys '20

Sponsor:

SenSys '20: The 18th ACM Conference on Embedded Networked Sensor Systems

November 16 - 19, 2020

Virtual Event, Japan

Acceptance Rates

Overall Acceptance Rate 174 of 867 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
86
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Hohman FKery MRen DMoritz D(2024)Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning ExperiencesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642109(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642109
Mishra RGupta H(2024)Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge DistillationIEEE Transactions on Mobile Computing10.1109/TMC.2023.3297026(1-12)Online publication date: 2024
https://doi.org/10.1109/TMC.2023.3297026
Mishra RGupta H(2023)A Model Personalization-based Federated Learning Approach for Heterogeneous Participants with Variability in the DatasetACM Transactions on Sensor Networks10.1145/3629978Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1145/3629978

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Enhanced Accuracy and Robustness via Multi-teacher Adversarial Distillation

Knowledge Distillation based Online Learning Methodology using Unlabeled Data Stream

Robust knowledge distillation based on feature variance against backdoored teacher model