OpenHands
openhands.ai4bharat.org
openhands.ai4bharat.org
Being able to read out finger spelling is kinda useless when most of the relevant population can't spell (why bother with an abitrary phonetic encoding for words in a language you don't know)
Newer systems can read entire signs, which is cooler, but still useless because they can't handle the grammar or context. Just some words in isolation.
Don't waste time on this. This is not an accessability tool for the disabled, it is a "feel proud tool" for undergrads with a machine learning library.
The real technological solution that would be helpful is still a few more major advancements away. It needs to handle grammar (ASL has radically different grammar than most spoken languages fyi). Going from a spoken language to a sign language is also significantly more useful.
Go learn ASL (or your local sign language) and encourage others to learn. That effort will do more good in making the world accessible to those who depend on sign language to understand and be understood.
FWIW I do ML research and these sound very possible with todays tech. I could probably hackathon this. This sounds more like a classic case of "engineer doesn't know what the users need" or "normie doesn't understand the actual life/needs of the disabled."
It sounds like you should set up a hackathon! Or at least publish some information so that future hackathon participants don't fall into the common tropes.
The root problem is that language concepts from spoken/written languages don't map 1-1 onto.
Some examples: - "pronouns" (really "referring to anybody or anything who isn't here to point at) are spacial. Things get assigned a location in the conversation to be gestured towards to reference them. This requires a level of spacial recognition that doesn't exist yet outside of VR with a lot of hardware. (Extra challenge recognizing the location of things that ARE NOT ACTUALLY THERE)
- descriptor/word order are very context sensitive (basically whatever order the signer thinks will refine things down for the watcher)
It doesn't encode well to enable the alignment based translations. Encodings exist for teaching purposes, but they are not easy to use for more relaxed used (that's is what signing is for). Corpuses don't exist for that either.
Modern translation techniques double down on having a large corpus and that basically all written languages have similar syntax structures even if the representations differ. Yes the "technology" exists in the same sense we know how to colonize the asteroids. We don't need a radical improvement of technology, just somebody to bother investing for decades without return. A hackathon isn't going to cross that void.
While I agree that most people naively approaching sign language vastly underestimate the challenge, I wouldn't consider it a waste of time, though. At least you learn a bit about machine learning and maybe with a bit of luck it raises awareness for the real challenges faced by sign language speakers.
I'm totally ok with somebody deciding to translate finger spelling as an educational exercise. Have at! Don't sell it as more than that, because it isn't.
I understand where you're coming from, though. A friend of mine is involved in Sign Language linguistics and everytime we meet she gripes about how much of her time is wasted by having to respond to AI companies who are supposedly inventing "revolutionary" technology to automate Sign Language translation.