Does this mean that Chrome is always listening on my computer's internal microphone?
You only need to per-visit when it's an insecure localhost endpoint or something similar IIRC. (or if you have an extension which clears that setting every page load)