I was not aware that _any_ Espressif hardware even supported classic bluetooth other than the very first ESP32 (which I am not sure if they're even available). And I was getting around 50ms latency back then (with the original ESP32 and SBC!)
What exactly did you try?