You can definitionally get lower latency with H.264 - somewhere around 5-10ms encode and 2ms decode. That's what makes Parsec and streaming gaming platforms possible even with all the other latencies involved.
H.265 might be slower, but EVC, especially LCEVC, is supposed to be about 30% lower latency encoding than H.264.