We use esp32-s3 at my company (smart speaker) but we don't don anything fancy.

Can you explain this? Why use esp as a peripheral if you already have an ARM chip?

We were considering moving off of esp to something that would make it easier do cpu-bound AI inference on-device or to enable more advanced audio DSP algos.