Hybrid Dual-Stream Deep Learning Approach for Real-Time Kannada Sign Language Recognition in Assistive Healthcare

Abstract

Background: Recent advances in sign language recognition (SLR) focus on high-resource languages (e.g., ASL), leaving low-resource languages like Kannada Sign Language (KSL) underserved. Edge-compatible, real-time SLR systems for healthcare remain scarce, with most existing methods (CNN-LSTM, 3D ResNet) failing to balance accuracy and latency for dynamic gestures. Objective: This research work aims to develop a real-time, edge-deployable KSL recognition system for assistive healthcare, addressing gaps in low-resource language processing and spatio-temporal modeling of regional gestures. Methods: We propose a hybrid dual-stream deep learning architecture combining EfficientNetB0 for spatial feature extraction from RGB frames. A lightweight Transformer with pose-aware attention to model 3D hand keypoints (MediaPipe-derived roll/pitch/yaw angles). We curated a new KSL medical dataset (1,080 videos of 10 critical healthcare gestures) and trained the model using transfer learning. Performance was evaluated quantitatively (accuracy, latency) against baselines (CNN-LSTM, 3D ResNet) and in real-world tests. Results: The system achieved 97.6% training accuracy and 96.7% validation accuracy, 81% real-world test accuracy (unseen users/lighting conditions). 53ms latency on edge devices (TensorFlow.js, 1.2GB RAM), outperforming baselines by ≥12% accuracy at similar latency. The two-stage output pipeline (Kannada text + synthetic speech) demonstrated 98.2% speech synthesis accuracy (Google TTS API). Conclusion: Our architecture successfully bridges low-resource SLR and edge AI, proving feasible for healthcare deployment. Limitations include sensitivity to rapid hand rotations and dialect variations. Keywords: Assistive Healthcare, Edge AI, Kannada Sign Language, Low-resource Language, Real-time Recognition, Transformer.