Speaking in code: hands-free programming

Just as anecdotal evidence, I was surprised to find out that a Radiologist friend of mine spends the majority of his work day 'writing' MR reports by speaking to (from what I understood) some modded 'Dragon' natural speaking software. He also assured me this is pretty much the norm for all his colleagues as well.

Although anecdotal, this is real-world evidence, that you can write something that is, highly technical, with a lot of domain-specific knowledge, to professional standard, via voice dictation.

The question then becomes, is there a market, for example, for a "C++ aware" or "Python aware" modded voice application? How big would the market be? How much would people pay? Could it be covered by health insurance in an RSI case?

I can fully support advice 2) 4) An ergonomic keyboard will work wonders.

1) NEVER take meds that mess with your brain! That's such an American vice. 3) ? 5) yes, but building muscles is better than building fat (can be combined;)

How long ago was this?

It certainly is not impractical now. I wrote my PhD thesis, along with most of the software using VoiceCode.io. These days I’m on Talon. Dictation has basically been my sole key input device for three years. Like anything, it takes practice.

Personally it was something I had to learn. Like I imagine it's harder to type creatively when you need to look at the keyboard for a few seconds to find each letter. I think your brain is doing something similar when you're searching for commands and trying to mentally model the dictation system to get the right words out.

I'm the creator of Talon. You're exactly right about context sensitivity, and you're wrong that I'm not designing for it. One of Talon's secret missions is to make extreme context sensitivity possible. I've done a lot of things at a low technical level to make this possible eventually, such as:

- You can change the entire voice grammar from anything to anything in a couple of milliseconds.

- I'm building a scope system, kinda like syntax scopes in a text editor like Sublime, but for arbitrary systemwide stuff. So you'll be able to say "this command activates on scope 'code.ruby.method' and 'filename matches test_*.rb" (which takes advantage of the fast grammar swap feature).

Getting really specific with commands and action implementations means heavy lifting can be done for you in a lot of really cool cases, and recognition accuracy goes way up. It's also hard to remember all this stuff, one fix is to use floating UI that shows you the most relevant context-specific commands.

As far as "generally accessible", another goal of Talon is to preempt RSI and find ways to convince people who can type fine to use it anyway. Two ways are "make many things way faster/smoother than typing" and "lots of cool eye tracking features".

> However, it should not be difficult to design one for sppech based input.

Personally I feel that this shouldn't even be necessary. In theory all you need to be able to vocalise is what you want the code to achieve, then that could be mapped to something similar to a snippet based on what language you are using. Most programming languages share the same core abilities, it's just the syntax to achieve the task that matters once you know what the task is. So this shouldn't need to be handled by a dedicated programming language for voice coding, it could just be handled by a wrapper (A text editor for your voice with an advanced NLP based snippet system).

Something like: "Check if A is equal to B, if it is, return A, if not, return B" would be much more efficient than saying "if parenthesis capital A equals equals captial B..." and so on.

Professional advice is a good idea.

One suggestion that helped for an acquaintance: if it's RSI from typing on a standard keyboard, but isn't too severe, switch to a fancy split ergonomic keyboard (provided that's different to what you currently use), change the key layout to Dvorak, throw away your existing typing habits and slowly learn to touch type again.

Another relative was diagnosed with https://en.m.wikipedia.org/wiki/Carpal_tunnel_syndrome . Surgery can help.

You may want to keep an eye out for the Xbox Adaptive Controller for games [1]

[1] https://www.xbox.com/en-US/xbox-one/accessories/controllers/...

The thing that helped me the most was realizing that my pain might not be located at the problem site. I had a nerve impingement due to https://en.wikipedia.org/wiki/Thoracic_outlet_syndrome which was causing wrist pain, even though nothing was wrong with my wrists at all.

Physical therapy can help a lot.

You're seeing text input demos because conventional english dictation systems have been horrible at fast and precise technical text input. This is a classic example: https://www.youtube.com/watch?v=MzJ0CytAsec

I believe it's far easier to solve navigation than text input. I will gladly record for you (or anyone here who has an idea) a synthetic (pre-planned) demo video of some more complex task. Want to come up with a specific workflow for to test? Like maybe "find <some project> on github, fork, clone it to my machine, perform <some specific refactor> / code editing task, commit, push" all without my hands, and I upload the first take.

The underlying systems are far more powerful than the article covered, e.g. in Talon you can write an editor plugin that collects filenames and turns them into commands in realtime to switch between files. Someone did a PoC with Talon's eye tracking to specify a symbol for insertion by looking at it. There have been demos of scope-based navigation / selection.

Personally I heavily optimized my alphabet/number input case (there's a specially designed alphabet that can do about 280 letters per minute at 100% accuracy), and use Vim commands directly, but many of the users use more conventional editors.

I plan to get into the habit of live streaming hands-free coding at some point so people can get a better idea of what it looks like to use this sort of thing, and so I can go back and watch and realize that I could be doing some things much better.

You're probably thinking of some other voice command system, or Dragon's built-in commands (which are bad because they have no continuous recognition, requiring constant pauses while talking). Voicecode behaves exactly as described in the article.

I just defined a command "return" in Voicecode which presses enter.

"testing return testing test test return test return test" spoken in one breath types the following:

  testing
  testing test test
  test
  test

Voicecode defines commands in Dragon by just adding words to the English vocabulary, leaving Dragon in dictation mode, and running a parser on the English output. Commands are executed anywhere they land in the phrase. The Voicecode grammar makes this somewhat less painful by using lots of non-English words for commands... but this approach is convoluted and hurts accuracy quite a bit imo.

No, pausing will not prevent the word being recognized as a command in Voicecode. However, you can tell VC to ignore commands and just treat what's spoken as regular text. The command "keeper" will do this.