1,603 karma · joined October 21, 2014
I haven’t used it all that much simply because I enjoy walking and usually do that. But for certain travel scenarios it’s awesome. I upgraded from a 20 minute car ride across town to a 30 minute e-bike ride.
- constrained context
- part of a hands-free workflow
A couple use-cases I have been pondering are driving assistant and cooking assistant.People are already used to using their phone or car's nav system to give them directions to an unfamiliar place. But even with such a system it's useful to have a human navigator in the car with you to answer various questions:
- What's my next turn again?
- How long till we get there?
- Are there any rest stops near here?
- What was that restaurant we just passed?
- Is there another route with less traffic?
These questions are all answerable with context that can be provided by the mapping app: - List of upcoming directions
- Overall route planning
- Surrounding place data
- Traffic data and alternate route information
It's possible to pull over to the side of the road, take off your distance glasses, put on your reading glasses, and zoom/pan the map to try to answer these questions yourself. But if the map application can just expose its API to the language interface layer, then a user can get the answers without taking their eyes off the road.The information is contextual and constrained based on a current task. In some cases it might be more desirable to whip out your phone and interact with the map to look up the answers on a screen, but often it won't be worth stopping the car, and so the conversational interface is better.
Cooking assistant is a similar case: you are busy stirring something and checking on the oven -- you don't want to wipe the flour off your hands to pick up your phone and ask how many teaspoons of sugar you need. Again: contextual and constrained info based on a current task, and your hands and eyes -- the instruments of traditional UIs -- are otherwise occupied.
Today, our software interfaces generally have one of two kinds of entity on the other end: humans, or other software. In the near future there will be another type of entity: language models. We need to start thinking of how our APIs will change when they're interacting with an LLM -- e.g. they'll need to be discoverable and self-describing; error states will need to be standardized or explicit with instructions on how to correct; they'll need to be fast enough to fit in a conversational interface; etc. It's arguable that such traits are part of good API design today, but in the future they may be required for the API to function in a landscape of virtual agents.
I really wish other cars on the highway observed this rule, but it’s common to see people doing ~80 mph with like two car lengths of buffer. When I myself maintain proper following distance, other drivers take this as an invitation to move into the empty space, forcing me further back in the cluster (and annoying the driver behind me).
When I go to my bank's website, it's easy for me to find my tax forms. They're usually one or two clicks away in some prominent top-level navigation component. But I would rather converse: "Download tax forms"/"Which ones: A, B, or C"/"The first two"/"Here ya go".
And in the future I'd rather say "Hey computer, download my tax forms from my bank and attach them in a draft email to <tax preparer>".
And in the later future that I'd rather say "Hey computer, do my taxes", and it will know what sources to gather info from and how to make sense of the numbers and how to file on my behalf. Or better yet, the machine will anticipate I need my taxes done and will kick off that process and solicit approvals and information from me. (Maybe by then the government will just tell me how much I owe ;))
Power users will likely sometimes find utility in dropping down "close to the metal" -- i.e., pointing and clicking and typing. But power users will mostly be composing workflows of models working together to achieve tasks (the term "scripts" will fit nicely). Yes this will be buggy and error-prone, but there will be glue models papering over the errors and trying different things and waiting for outages until the human user's intent is done.
Language models are giving us another layer of abstraction over software. Perhaps the final layer, as this one can interact directly with language (which is reified thought). This layer has the ability to cope with the inherent ambiguity of thought by being conversational -- it can ask you to clarify what you mean and gain confidence that it understands your intent.
It seems like the pieces are there: ability to “reason” that kitchen is a room in the house, that to get to another room the agent has to go through a door, to get through a door it has to turn and pull the handle on the door, etc. Is the limiting factor robotic control?
within the last couple years i've been occassionally trying to cultivate that sense again, but with only blips of success.
[0]: https://www.shawnmatthewcrawford.com/balloons-the-balloon-cl...