The race to teach sign language to computers

That's really cool. Was this funded by the government? I ask because it's on a .gov site.

One thing to keep in mind is that ASL, for example, is the first language of many people and it is a totally different from the English language.

What if you had deaf friends and could wear glasses that would translate sign language for you? What if you were deaf and had a robotic assistant? What if you wanted to do translations for a hearing audience?

Sign language is not a way to speak English using your hands -- it is a distinct language with distinct grammar and distinct vocabulary.

If you are born Deaf, you will not have the opportunity to natively acquire a spoken language. Our brains are not designed to acquire spoken language through writing -- and our spoken languages are designed to be acquired through listening, not reading.

It's easy to think that a Deaf signer living in America is a native speaker of English, but they very well may not be, having never heard the sounds and therefore been unable to internalize all of the many, many necessary grammatical and morphological rules of the language.

Expecting all Deaf people to simply use written English when dealing with a computer is like expecting all Americans to use Spanish to interact with their computers -- technically possible (Spanish is a perfectly good language, and relatively accessible starting even from grade school here), but ignores the human reality that people want (need) to speak their own language.

There are not yet standard ways to write sign languages. Partly this is because of difficulties in translating the medium to text -- unlike speech, sign language production is not entirely linear, in addition to the fact that your hands can be articulated comprehensibly in much more sophisticated ways than your mouth can.

If you wonder why you might not be able to access this, keep in mind that archive.is and Cloudflare DNS do not play nicely with each other [0]. It might also bite you if you are using Firefox because of the DNS over HTTPS service from Cloudflare that Firefox uses [1].

Loading the article directly into Firefox's reader mode by prepending about:reader?url= to the URL works fine though.

[0] https://twitter.com/archiveis/status/1018691421182791680

[1] https://support.mozilla.org/en-US/kb/firefox-dns-over-https

American Sign Language grammar is subtle and complex.

Sign languages don't only use hand shape, but also hand motion, body pose, facial expression, mouthing words, etc. Cropping still images around the hands won't do you much good at all.

Essentially, vocal languages are one-dimensional, thus easy to solve. Filter out disturbances, background noise, and train the machine to adapt to dialects, and you're halfway home.

Sign language is at least three-dimensional, it can embed more information. The ASL signs for "forget" and "because" differ only in how you mouth them - easy enough, but sometimes you even skip mouthing them because there is sufficient context. What's worse is the location of the hands can vary depending on how lazy the signer is feeling, so "forget" is supposed to be signed from the forehead but it can also be signed from some empty point in front of your shoulder, off a "phantom person"'s imaginary forehead. You rely on the listener to understand that you intend for there to be a phantom person there.

Parsing sign language means you have to turn a 2D recording into a 3D model - even humans can have difficulty with this, parsing signs over Zoom - and you need to solve parsing facial expressions, reading lips and more. And you need the machine to be smart enough to detect the assignment of phantom persons to various locations around the signer. That's my impression as a deaf person anyway.

This paragraph from the article highlights some of the problems.

> Research led by a team at Microsoft, a big computing firm, and published in 2019, estimated that a typical publicly available corpus of a spoken language consists of around a billion words from as many as 1,000 different speakers. An equivalent data-set in a sign language might have fewer than 100,000 signs from just ten people. Besides large numbers, a good corpus also needs variety. This means conversations between native signers of diverse backgrounds, dialects and levels of fluency. Because deaf people more often have physical disabilities than do those with unaffected hearing, representing those with restricted fluency of movement is important.

All sign languages use much more than handshape and position. And even outside of facial expression grammar (ie non-manual communication) there is a whole host of other complexities that make parsing sign language difficult.

Take classifiers: there are eight morphological types that allow users to express complex ideas without the use of any type of vocabulary-based utterances. You can certainly sign the equivalent of, “Did you see the car parked close to the tree,” (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the point across, but it’s not efficient, and doesn’t really utilize sign language in a way that feels normal for users.

But that same question can be posed in a more condensed format if you use a classifier, (#CAR 5:CL-“tree”/3:CL-“car parked really close” SEE?). In the second utterance, one use the lexicalized (fingerspelled “car”), then holds up a five hand with bent elbow in their non-dominant hand, use the three hand representing the car placing it reeeeeeally close (with grammatical, NMC emphasis) to the tree, and finally asking the receiver if they saw it.

In that example alone, I’m using two types of classifiers: element and locative. Imagine a complex story using all eight classifier types with just a little vocabulary mixed in? I doubt AI/ML could interpret the context, characters, actions, events, descriptions, etc. accurately. Anyway, I hope this example shows there is a lot more complexity in computerized interpretation of signed languages well beyond gesture/vocabulary recognition.

Note: My glossing may not be perfect. It's been a while, and I’m a bit rusty.

Fragmentation also is a big issue. Not only in the sense that there are separate very different sign languages (American Sign Language has little in common with British Sign Language, they are not mutually comprehensible), but even for each separate sign language you also get different 'dialects' in different areas/major cities, which are mutually comprehensible but have lexical variation depending on the speaker's location/community that needs to be properly represented in your datasets in order to be acknowledged and understood. In some sense it seems the equivalent of what the fragmentation was for spoken languages before the printing press and age of nationalism facilitated language standardization.

That should be way, way simpler. It sounds like a neat computer vision class project.