Real-time facial and body pose emotion recognition for children with autism based on YOLOv8 and LSTM

Abstract

Children with autism spectrum disorder (ASD) often face challenges in recognizing and expressing emotions, which can affect their behavior and participation in inclusive classroom environments. This study proposes a real-time multimodal emotion recognition system integrating deep learning and Internet of Things (IoT) technologies to support early emotional monitoring in children with ASD. The framework combines YOLOv8 for facial expression detection and YOLOv8-based pose estimation for body movement analysis, along with a long short-term memory (LSTM) network for temporal emotion classification. At the facial level, the system recognizes five emotional states: sad, happy, neutral, boredom, and tantrum. At the temporal level, the LSTM model classifies behavioral sequences into three categories: neutral/bored, happy, and tantrum, enabling hierarchical emotion interpretation from instantaneous expressions to temporal patterns. Experimental results show that the facial expression model achieves 92% precision, while the LSTM-based classifier reaches 95% peak validation accuracy and 93.33% final test accuracy. The system is deployed on a web- based monitoring platform with real-time notifications for educators and parents. The proposed approach demonstrates effectiveness in providing timely emotional insights to support early intervention and improve inclusive education for children with ASD.