I mean we deal with daylight saving time all the time and I know it's not the same because the leap second affects UTC, not just local time zone, it's just that you are either dealing with monotonically increasing time like epoch, or you are dealing with "human" time and I found no distinction in the latter.
Is it "just" that leap seconds or delay seconds caused problems in epoch to utc conversion? Note the just in quotes, but did I just answer my own question? :)