One interesting topic of semi-paranoia that hasn't been discussed is timing attacks. I'm not even implying its a realistic concern, but a timing attack could be imagined that works better with longer keys.

So 1024 is run too fast to gather intel based on timing, but an imaginary 8192 bit key would run slowly enough to leak one random bit of key per session via timing analysis, as a physics-style thought experiment.

Not trying to claim out that longer keys are weaker in reality, but am trying to make the point that being impervious to timing analysis attacks might be kinda important. Or another way to phrase it is there's more ways to break a key than pure math or torture.