What doesn't seem clear to me, and I'd appreciate if you could tell me why you think differently, is that KYBER-512 isn't as strong as it was targeted to be. I find djb's argument on this narrow point fairly convincing: KYBER-512 isn't as secure as AES-128 (by the methods used to measure "secure" in this competition).
Given that I already generally use AES-256, why shouldn't I treat this the same way as AES-128?
That is, "it's probably fine-ish, but if you have the power, just go one bigger".