You can just embed those new words and they get added to vocabulary. It is fairly simple. One risk is - When a new word is very close to a known one, it tends to fall to it.
Would it make sense to rather compare distances to sub words rather instead full words ?
I don't know if this would be possible because now you have to deal with sequence modeling to transform audio to sub words to words to sentences.