Basically it starts computing a response every time a word comes out of the speech recognizer, and if it is able to finish its response before it hears another word then it starts speaking. If more words come in then it stops speaking immediately; in other words, you can interrupt it. It feels so much more natural in conversation than ChatGPT's voice mode due to the low latency and continuous listening with the ability to interrupt.
There are a lot of things that need improvement. Most important is probably that the speech recognition system (Whisper) wasn't designed for real time and is not that reliable or efficient in a real time mode. I think some more tweaking could improve reliability considerably. But also very important is that it doesn't know when not to respond. It will always jump in if you stop speaking for a second, and it will always try to get the last word. A first pass at fixing that would be to fine tune a language model to predict whose turn it is to speak.
There are also a lot of things that this architecture will never be able to do. It will never be able to correct your pronunciation (e.g. for language learning), it will never be able to identify your emotions based on vocal cues or express proper emotions in its voice, it will never be able to hear the tone of a singing voice or produce singing itself. The future is in eliminating the boundaries between speech-to-text and LLM and text-to-speech, with one unified model trained end-to-end. Such a system would be able to do everything I mentioned and more, if trained on enough data. And further integrating vision will help with conversation too, allowing it to see the emotions on your face and take conversational cues from your gaze direction and hand gestures, in addition to all the other obvious things you can do with vision such as chat about something the camera can see or something displayed on your screen.