While I'm somewhat concerned about the privacy implications, it has been very convenient. The biggest problem it has solved is "how to listen to music", and if that isn't a pain point for you, you don't really need it.
My kids have told Alexa to recognize their voices. I have several boys, and Alexa really can't tell them apart very well. (And, I haven't figured out how to edit the profiles; one of them taught Alexa he was John Cena, and I don't know how to correct it; another son uses a deep voice to tell Alexa he is Batman, which is kinda funny) I would not be concerned, at this point, about other people's devices keeping shadow profiles of you. (This statement probably needs to be reassessed every five years).
I did play with an open source voice assistant, but these are so much better, hardware- and software-wise. Would I take a self-hosted voice assistant if it reliably did most of the things I use Alexa for? Absolutely.