I could, as you suggest, set a default, but if I do that then that browser owns all of my links and the only way to change it is to hunt down the setting to disable the default (which is buried in the settings UI I suppose).
So I live with the (extreme annoyance) of clicking on a link within the browser that then prompts me which browser I'd like to use to open it.
Getting the UX right on this is by far the most important part of the project.