Designing Tiny Machine Learning Models for Keyword Spotting Using Knowledge Distillation – A Comparative Review

Small-footprint Keyword Spotting (KWS) refers to the process of spotting key-words locally within highly optimized, resource-constrained IoT systems. Since these devices generally have low capability in computation, memory and power, the design of lightweight and accurate keyword spotting models is a big challenge of research. Tiny Machine Learning (TinyML) is a solution to this challenge that allows for the deployment of deep learning models on edge devices with limited resources. But size reduction invariably results in a drop in accuracy of recognition. Recently, Knowledge Distillation (KD) has been a promising method to compress knowledge from a large and accurate teacher model to a small student model while retaining competitive performance, and it has attracted significant research interest. In this review, the recent progress of TinyML-based keyword spotting models with knowledge distillation is compared. Different learning strategies, small neural network architectures, model compression methods, and deployment are explored in terms of model size, computational cost, latency for inference, and recognition accuracy. Moreover, existing methods are reviewed with their pros and cons and research directions such as self supervised learning, adapt knowledge distillation, few-shot learning, and hardware-aware model optimization are highlighted. The purpose of this review is not only to give researchers a thorough overview of the latest advances and future directions for efficient TinyML keyword spotting, but also to highlight the challenges encountered by current implementation methods.