Working On A Low-cost Sign Language Recognition System For Hearing-impaired Students — Need Advice On Collecting Datasets

Hi everyone,

I’m a computer science student currently working on a project called 𝐒𝐢𝐠𝐧𝐁𝐫𝐢𝐝𝐠𝐞, an AI-powered accessible learning platform designed to improve classroom communication for hearing-impaired students.

The main goal of the project is to build a 𝐥𝐢𝐠𝐡𝐭𝐰𝐞𝐢𝐠𝐡𝐭 𝐬𝐢𝐠𝐧 𝐥𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐫𝐞𝐜𝐨𝐠𝐧𝐢𝐭𝐢𝐨𝐧 𝐬𝐲𝐬𝐭𝐞𝐦 𝐭𝐡𝐚𝐭 𝐜𝐚𝐧 𝐫𝐮𝐧 𝐨𝐧 𝐥𝐨𝐰-𝐜𝐨𝐬𝐭 𝐝𝐞𝐯𝐢𝐜𝐞𝐬 (𝐧𝐨𝐫𝐦𝐚𝐥 𝐥𝐚𝐩𝐭𝐨𝐩𝐬 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 𝐆𝐏𝐔𝐬) so that it could realistically be deployed in schools.

Current approach:

– MediaPipe Holistic for hand + pose landmark extraction

– Landmark normalization

– Random Forest classifier for sign prediction

– FastAPI backend + React frontend

– Real-time webcam input

The system currently supports 𝐛𝐚𝐬𝐢𝐜 𝐰𝐨𝐫𝐝-𝐥𝐞𝐯𝐞𝐥 𝐬𝐢𝐠𝐧 𝐝𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧 and includes a 𝐜𝐥𝐚𝐬𝐬𝐫𝐨𝐨𝐦 𝐦𝐨𝐝𝐞 𝐟𝐨𝐫 𝐛𝐢𝐝𝐢𝐫𝐞𝐜𝐭𝐢𝐨𝐧𝐚𝐥 𝐜𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧

– Student signs → converted to text

– Teacher speech → converted to live captions

Right now the biggest limitation is 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐬𝐢𝐳𝐞. I only have a small set of labeled sign images/videos, which makes it difficult to expand vocabulary or experiment with temporal models.

I’m looking for advice on a few things:

1. 𝐃𝐚𝐭𝐚𝐬𝐞𝐭𝐬 𝐟𝐨𝐫 𝐈𝐧𝐝𝐢𝐚𝐧 𝐒𝐢𝐠𝐧 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 (𝐈𝐒𝐋) or similar landmark-based sign datasets.
2. Best ways to 𝐜𝐨𝐥𝐥𝐞𝐜𝐭 𝐚 𝐬𝐦𝐚𝐥𝐥 𝐛𝐮𝐭 𝐮𝐬𝐞𝐟𝐮𝐥 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 for word-level or classroom-related signs.
3. Suggestions for improving the model while keeping it 𝐥𝐢𝐠𝐡𝐭𝐰𝐞𝐢𝐠𝐡𝐭 𝐞𝐧𝐨𝐮𝐠𝐡 𝐭𝐨 𝐫𝐮𝐧 𝐨𝐧 𝐂𝐏𝐔 𝐝𝐞𝐯𝐢𝐜𝐞𝐬.
4. Any feedback on the system design or architecture.

Eventually I’d like to extend it toward 𝐬𝐞𝐪𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐰𝐨𝐫𝐝 𝐝𝐞𝐭𝐞𝐜𝐭𝐢𝐨𝐧 𝐨𝐫 𝐬𝐢𝐦𝐩𝐥𝐞 𝐬𝐞𝐧𝐭𝐞𝐧𝐜𝐞-𝐥𝐞𝐯𝐞𝐥 𝐢𝐧𝐭𝐞𝐫𝐚𝐜𝐭𝐢𝐨𝐧, but still keep it deployable on low-resource hardware. Currently this is done by the react side like when users sign it stores the sequence of words.

If anyone has worked on sign language recognition, accessibility tools, or dataset collection, I’d really appreciate your suggestions.

Thanks