As someone who's used tiling WM's for years this confuses me. How I use my trackball? I don't outside of having to access something in Firefox that vimium isn't allowed to touch.
Keyboard UI tends to be verb first. E.g in Vimium if you want to open a link in a new tab you press F to start the "Open in new tab" command. Then it lists the links you can open in a new tab for you to select from.
Whereas mouse UI tends to be noun first. E.g you see a link, and right click it to list the commands. You then select the "Open in new tab" command.
I think the reason for the difference is that the mouse can reach an arbitrary location easier than a keyboard, but once it's there it has much less options to continue.
For me, this means that the mouse is more exploratory, and the keyboard is more task oriented. If I'm toying around with ideas the mouse works better, but if I have something specific I want to do you can't beat the keyboard.
Writing this I've realised that I could repurpose the underutilised side buttons on my mouse as modifiers :p Hold top thumb button: press left click to go previous window, right to go next; hold bottom thumb: press left to go to previous workspace, right to go to next. Could have top thumb + mouse wheel for volume (and middle click for mute). I guess bottom middle could open a radial menu of commonly used apps/commands :)
On the other hand, things like Surfingkeys, accidentally, sometimes allow me to circumvent bad design of a Web page, like when there's text you wouldn't be ordinarily able to select by dragging your mouse (because it responds to mouse events and changes etc.) this is still usually possible with Surfingkeys. (Without it, you'd have to "inspect element, find the element with text and extract it from there). Also, often times I can use this extension to scroll things that you wouldn't be able to scroll with mouse (again, due to bad Web page design).
Mouse only wins in situations with NIH GUI elements, esp. s.t. require drag-and-drop (volume adjustment on video clips), or controlling built-in elements (like this text area) to resize them (which is really just a browser defect in accessibility, but hey, most of the time I have to use Web, I have to overcome the incompetence of the people building it, thus the selection of tools).
Not saying it's a bad experience. I find it's quite nice if I'm currently looking at the correct link so I can get the letters from the label easily.
It's just a different paradigm. If I didn't know what a link was, I wouldn't know that I could press f to open the link selection. This means I can't select a link unless I had a "select unknown" button, and then use that to inspect it.
Select unknown can be done with the keyboard: Emacs' embark[^1] is a good example of it. You use search/arrows to go to an interesting piece of text (e.g something underlined), run embark, and it lists all the potential things you can do with it (e.g open in external browser, or download an archived copy). It's just that keyboard UIs do not tend to be geared around it.
That though is the sort of thing the mouse is good at: you see something interesting, you prod it to see what you can do with it.
Say you saw a phrase you didn't understand in the browser. Currently, with a mouse you'd highlight it, right click, and then select the search with search engine option. With keyboard you'd open the search engine, and enter the phrase. Both get you to the same place.
Now if we extend the mouse to have a "I would like to know idioms better" option. You could just select the phrase, right click, and a definition could be there waiting for you next to the context menu. It could even be there on hover.
With the keyboard though, you'd generally run an "I would like to know idioms better" command. This would then ask you which idiom on the page would you like to understand. You then select it, and presto. A faster version could potentially have idiom meanings show up the moment you ran the command by a similar hover effect (or just listing all discovered idioms in a popup etc).
One of the differences here though is that from the mouse you still have a list of actions on the text open, so maybe if you liked the definition you could then copy the phrase.
This certainly could be done from the keyboard UI: just have it list other applicable commands that could be run on the selection generated from the command that searched for phrases. No reason why it could not then spawn something asking if you wanted to "(c)opy, c(u)t" etc like the context menu.
But that is only visible because there was a phrase you didn't know. If you didn't know that there was a command to define phrases, you wouldn't have found out. In theory you could have commands register everything they could act on, and then have another command let the user select an interactive thing, and list the commands that registered for it. I think that would be overwhelming though, as something like a dictionary command would register for every word.
Ultimately the mouse is just different then the keyboard, and I think that there is interesting potential in having a tool that can interrogate arbitrary things on your screen.
Yeah, that's how Surfingkeys works too more or less.
What you are saying is that you need to learn to use this interface in order to be efficient with it -- this is pretty much the story of any human activity, any tool ever. Whatever you learn how to do, you get simplified tasks and more forgiving tools. It's very rare that you'd keep the same tool and the same approach if you keep specializing in something. So, I don't see it as a problem. Yes, you need to learn, yes, because knowing how to do something more efficiently is worthwhile?..
> Currently, with a mouse you'd highlight it, right click, and then select the search with search engine option. With keyboard you'd open the search engine, and enter the phrase. Both get you to the same place.
Nope. That's not even close to how it works. With Surfingkeys, when I look at a phrase / word that I want to look up in a search engine, I can do this:
1. Press "v", and do the Jedi-jump style navigation to select the beginning or end of the text I'm interested in (i.e. it will take another 1-3 key presses). 2. Press "v" again to enter visual selection mode (similar to Vi's). 3. Use Vi's navigation commands to select the text of interest (again, another 2-3 key-presses). 4. Press "o" to search in default search engine (but there are also bindings for other engines, or I could select them by name w/ auto-complete. Or, I could press "y" to copy selected text and paste it elsewhere (also outside of the browser). Or, I could press "t" and paste it into Google Translate text box in a new tab etc.
This also works for all kinds of "unselectable" texts (eg. in GMail, the email title is not normally selectable with the mouse because instead it will try to open it). With Surfingkeys I can select the title of an email w/o opening it in GMail.
Another benefit is precision. I don't need to pay a lot of attention to the mouse pointer's position. If I know what text I want to select, I can do it with my eyes closed, or, more likely, looking at a different place on the screen which may require my attention at that time. Not that useful with selection, but is very useful with search / when I need to fill a bunch of forms, and I need to look at the original when searching for the corresponding text in my copy.
> It could even be there on hover.
With Surfingkeys you hover by pressing C-h. See the steps above for how to focus the element of interest, and then by pressing C-h you'd invoke the mouseover event. This also beats mouse interaction for elements that are hidden or partially hidden by other elements on the page due to bad / broken layout.
> again, another 2-3 key-presses
I guess atm, I would do `/again<enter>nvi(`. That's based off the fact that I don't think "again" shows up much in the text, and I can jump quickly between instances via `n`. Then once I'm there I realise that the rest in in parens, so I go for `vi(`. In retrospect I should have searched for "again," as the comma is even more specific.
It's funny: "Another benefit is precision. I don't need to pay a lot of attention to the mouse pointer's position." is close to an argument I'd make for the mouse. I think that might be because when I'm thinking, I idle with the mouse, higlighting things that look interesting to me etc. So when I want to move forwards, it's pretty close where I want to be. If I had to select something via keyboard I'd have to look for distinct markers to work out if I can do something like `vi(`, whereas the mouse lets me do fuzzy visual matching (acme would. This is only for mouse though, my touchpad/trackpoint is less precise.
My point was not that I think the mouse is better then the keyboard. My point was that I think the mouse tends towards a fuzzier exploratory mode, and the keyboard tends towards more focused execution modes. Even if you like one mode more, there's no reason not to improve the other.
Now after writing all this, I'm not sure that's correct as I can think of other exploratory keyboard UIs that work (e.g Norton Commander style file managers).
I do still think that there is no reason not to improve your mouse experience, a nd that it does have untapped potential. Acme editor has this thing where you can just right click a word, and it'd jump to the next instance of that word. No selection needed. So if you're reading some code and see an interesting variable you could right click to see where it's next used. This of course assumes you have your hand on a mouse already, which in the case of Acme you probably do as it has a heavy mouse focus.
Hover was a terrible example though. I hate having things appear during hover. I was just trying to think through what the least interaction would be, and you're quite right that if you can focus via keyboard easy it'd also be easy to get hover.
----
A weird thought came to me during this: what if the vimperator/vimium/surfingkeys link selector was ported to the mouse? You could hold down the left mouse button, and this would label all the links with numbers, and highlight number 1 (or 0 if you prefer). Then when you spun the mousewheel down it'd move to the next one. Spinning it up would go to the previous link. Finally when you released the left mouse button, it'd open the selected link.
So how it'd play out is you'd hold left, see you want link 11, spin the mousewheel that many times, and then release. I suspect you'd learn how far the numbers feel over time, so it'd get fairly precise (and you can adjust by spinning back).
No idea how practical it'd be.