A Systematic Review of Multimodal Representation Learning for Indian Sign Language Recognition Using Spatial Temporal and Transformer Based Approaches
Main Article Content
Abstract
In India, Indian Sign Language is the important means of communication for people from the Deaf and hard-of-hearing communities, with automated recognition providing the opportunities for overcoming communication problems and supporting simple human-computer interaction. The studies in the field have moved from the hand-crafted features and traditional machine learning to the use of CNN-based representation learning and temporal modelling, landmark and pose-based representations, multimodal learning and the use of Transformer-based technologies. Herein, we carry out a comparative analysis of 36 research articles dedicated to ISL and relevant sign language recognition techniques being focused on research datasets used, tasks, modalities, representations, architectures applied, strategies of fusion, results obtained and the limitations of the investigation. The findings of our analysis suggest a shift from traditional isolated signing classification techniques towards dynamic and multi-modal approaches capable of taking into account the timing of signs and the variability in the signing. While CNNs and transfer learning techniques seem to be working well for related tasks, new methods involving the use of language models can handle more complex patterns in signing. Nonetheless, problems with continuous recognition, generalization across signers, dataset diversity and benchmarking, robust multimodal fusion and computational efficiency are still a huge concern. Thus, according to our results, the integration of RGB and depth video, hand landmarks and body position along with effective temporal modelling seems to be the way to make ISL recognition systems more robust, generalizable and deployable in practice.