Hand sign language translator
Transforming motion into meaning. An advanced computer vision pipeline utilizing MediaPipe and LSTM neural networks for real-time sign language translation.
Project Type
Graduation Project
Core Technologies
AI & Computer Vision
Industry
Accessibility Technology
Date
2025 - 2026
Challenge
The primary challenge was developing a robust AI model capable of accurately interpreting dynamic hand gestures in real-time, regardless of the signer's position.
Building the American Sign Language recognition system required capturing temporal motion sequences using MediaPipe and processing them through a TensorFlow LSTM network. A major hurdle was "translation invariance"—where changes in a user's standing position or distance altered the spatial data. We solved this by implementing a Relative Normalization algorithm to standardize coordinates based on facial landmarks. Additionally, extracting exactly 30 consistent frames per video and preventing model overfitting required meticulous data preprocessing and a highly modular code architecture.
Solution
To address this, we engineered a modular Python pipeline that standardizes spatial data using Relative Normalization, effectively negating positional shifts by anchoring coordinates to the nose landmark. Raw mobile videos were systematically processed into consistent 30-frame arrays to capture precise temporal motion, while utilizing Dropout layers in our LSTM network to prevent overfitting.
Furthermore, we developed a streamlined real-time inference engine that efficiently processes live camera feeds. By seamlessly bridging complex AI computations with a clear output display, the system instantly translates physical gestures into readable American Sign Language text. This results in a highly responsive, practical application that highlights the power of machine learning in breaking down communication barriers.
Project
Impact
Developing this AI-driven ASL translator demonstrated the profound potential of machine learning in improving accessibility. The modular architecture and our custom Relative Normalization algorithm resulted in a highly accurate, real-time inference model. Moving forward, the vision is to scale the dataset to support over 100 distinct signs and optimize the system for edge devices, further bridging the communication gap for the deaf and hard-of-hearing community.
Future Scope
Scaling the vocabulary to 1000+ ASL Words
