Project
Continuous Real-Time BdSL Recognition
Real-time recognition of continuous Bangla Sign Language, built on a purpose-collected 9,010-video, 530-class dataset.
With M. Ahmed and S. I. Moyeen — paper under review. Started as my undergraduate thesis at RUET (Jan 2023 – May 2024), advised by Sumaya Ishrat Moyeen.
Problem & motivation
Bangla Sign Language has very few public resources, and what exists is mostly isolated signs: one clip, one label, neutral start and end poses. Real signing is continuous — signs blend into each other, and a usable system has to keep up with the signer in real time. We built both halves of the missing infrastructure: a continuous BdSL dataset (9,010 videos, 530 word classes) and a real-time recognition pipeline evaluated with latency as a first-class metric.
The dataset
The dataset has its own page — stats, methodology, benchmark protocol, and citation. It will be released alongside the paper.
Approach
The pipeline extracts holistic landmarks per frame (MediaPipe Holistic — hands, pose, and face), then LSTMs over the landmark streams produce a running word sequence, with an attention mechanism that improves sentence-formation accuracy in continuous signing. Landmarks rather than raw pixels keep per-frame cost low enough for real time on consumer hardware. Full training details and ablations are in the paper.
Results
| Model | Top-1 (%) | Latency (ms/frame) |
|---|---|---|
| MediaPipe Holistic + LSTM-attention (ours) | — | — |
| Frame-level CNN baseline | — | — |
Demo
Meanwhile, the first stage runs live in your browser — try the hand-tracking demo on the dataset page. Nothing is uploaded; it’s the same landmark representation the recognition model consumes.
Limitations & what I’d do next
- Vocabulary is 530 words, not a language. Continuous recognition over this vocabulary is a real step, but full BdSL needs grammar-aware modelling, not word spotting at scale.
- Signer diversity bounds generalisation — performance on unseen signers is the metric that matters; signer-independent splits are part of the benchmark design, and expanding the signer pool is the single highest-value extension.
- Landmark extraction is a single point of failure: motion blur, occlusion, and hand-on-hand contact drop landmarks exactly during the most informative frames. A pixel-landmark hybrid stream is worth testing.
- Segmentation boundaries: continuous decoding still makes most of its errors at sign transitions; CTC-style training and explicit boundary modelling are the next experiments.
- Deployment target: the end goal is on-device mobile inference — quantised temporal models on landmark streams look feasible.