Odd, this is not happening here on macOS 15.0.0. Turning bluetooth off either via the system settings app or the menu bar icon shuts off bluetooth immediately with no prompt for me...