> non-native speakers must find this phrase so confusing
Even native speakers struggle with it (i.e. me, when I first saw it being used).
'Leave' in its form meaning 'permission' is only generally used in 'I am on leave from work today'. Many other Commonwealth and Anglophone countries (Australia, NZ, Canada, Singapore) just call it a 'visa validity period'; indefinite leave to remain is 'permanent residency'.