Checkout the LERF work from the NerfStudio team at UC Berkeley. SMERF is addressing a different problem, but there are definitely ways to incorporate semantics and detection as well.