Nice work! I've made a similar demo as a web app [1] and obtained a few orders of magnitude faster performance, in the 50ms for 1.6M images on single laptop core. I've limited experience with coreml but there's a few tricks that could probably help you too? If you feel like hacking on it together shoot me an email :) (in profile)
[1] https://paulw.tokyo/post/real-time-semantic-search-demo/