From my limited understanding:
JITs can take into account the actual data being processed. And most importantly here, its size.
Knowing the size will help with making chunks of work fit into L1, L2, L3 caches. Creating sensible SIMDs operations. Choosing what goes into a warp.
Also, sometimes it is better to rematerialize a computation than having stored it; and the threshold to do it depends on the space and computing costs.
Languages like Halide [1] let you hand-tune this threshold. I guess this is the kind of work XLA does here.