Hacker News Reader

I think you are on to something, but I don't think it's the final answer. Your explanation sounds true, but it doesn't really explain how greater pressure on P6 speeds things up.

I'm working with a slightly modified loop that counts down rather than up, so that the loop has slightly fewer instructions. I've switched to using an "add to memory" to reduce the instruction count further. None of these seem to directly affect the speed of the loop, but here's the fastest loop I have:

        .p2align 3
  1:
        addq    %rax, counter(%rip)
        dec %rax
        jmp 2f
  2:
        jne     1b
        rep ret

On Skylake, it executes in 0.497 s. If I remove the "jump to the next instruction", it slows to 0.646 s. This agrees with your explanation. But here's the part I don't understand, and that may undercut your theory: if I change the first line to be ".p2align 4" (that is, if I increase the minimum alignment from 8 to 16) the speed is the same whether or not I have the extra jump. Even more confusingly, it's always the slower speed!

Here's what port usage looks like for the two different cases (both with the same instructions, just different alignments):

                                   Fast         Slow
      INSTR_RETIRED_ANY      |  1600095750 | 1600095846
    CPU_CLK_UNHALTED_CORE    |  1643927320 | 2192886305
 UOPS_DISPATCHED_PORT_PORT_0 |   645482071 |  774370373
 UOPS_DISPATCHED_PORT_PORT_1 |   653347597 |  782702101
 UOPS_DISPATCHED_PORT_PORT_2 |   299084760 |  298170998
 UOPS_DISPATCHED_PORT_PORT_3 |   300474456 |  292800108
 UOPS_DISPATCHED_PORT_PORT_4 |  1410210909 | 1994332142
 UOPS_DISPATCHED_PORT_PORT_5 |   406138100 |  791887776
 UOPS_DISPATCHED_PORT_PORT_6 |   989257513 | 1338919000
 UOPS_DISPATCHED_PORT_PORT_7 |   200496146 |  209093087

From the first line we see that both Fast and Slow have the same number of instructions executed (which makes sense, since both are executing the same instructions), but down below we see very different numbers of µops dispatched! I'm not sure what to make of this, but I think this shows that it is not (solely?) a front end issue.

View on HN

Fast Slow Runtime (RDTSC) [s] | 4.912542e-01 | 6.476124e-01 RESOURCE_STALLS_ANY | 135,743,115 |1,298,835,857 IDQ_DSB_UOPS |1,441,709,567 | 97,635 LSD_UOPS | 197,902,567 |1,999,996,841 IDQ_MITE_UOPS | 360,547,507 | 82,354 INSTR_RETIRED_ANY | 1600095725 | 1600095769 CPU_CLK_UNHALTED_CORE | 1662620966 | 2192914751

1,348,491,506 resource_stalls_any 1,694,613,980 uops_issued_stall_cycles 759,763,767 uops_executed_stall_cycles 994,974,421 uops_retired_stall_cycles

109,300,935 resource_stalls_any 120,303,318 uops_issued_stall_cycles 164,322,430 uops_executed_stall_cycles 15,834,456 uops_retired_stall_cycles

756,686,062 uops_dispatched_port_port_0 782,579,771 uops_dispatched_port_port_1 279,387,822 uops_dispatched_port_port_2 314,214,218 uops_dispatched_port_port_3 1,994,313,938 uops_dispatched_port_port_4 807,514,909 uops_dispatched_port_port_5 1,050,884,205 uops_dispatched_port_port_6 209,316,457 uops_dispatched_port_port_7

640,957,968 uops_dispatched_port_port_0 637,621,266 uops_dispatched_port_port_1 399,224,647 uops_dispatched_port_port_2 400,205,411 uops_dispatched_port_port_3 1,610,294,604 uops_dispatched_port_port_4 643,780,748 uops_dispatched_port_port_5 848,142,785 uops_dispatched_port_port_6 470,700 uops_dispatched_port_port_7

The loops with the following attributes qualify for LSD/micro-op queue replay: • Up to eight chunk fetches of 32-instruction-bytes. • Up to 28 micro-ops (~28 instructions). • All micro-ops are also resident in the Decoded ICache. • Can contain no more than eight taken branches and none of them can be a CALL or RET. • Cannot have mismatched stack operations. For example, more PUSH than POP instructions.

The Decoded ICache consists of 32 sets. Each set contains eight Ways. Each Way can hold up to six micro-ops. The Decoded ICache can ideally hold up to 1536 micro-ops. The following are some of the rules how the Decoded ICache is filled with micro-ops: • All micro-ops in a Way represent instructions which are statically contiguous in the code and have their EIPs within the same aligned 32-byte region. • Up to three Ways may be dedicated to the same 32-byte aligned chunk, allowing a total of 18 micro-ops to be cached per 32-byte region of the original IA program. • A multi micro-op instruction cannot be split across Ways. • Up to two branches are allowed per Way. • An instruction which turns on the MSROM consumes an entire Way. • A non-conditional branch is the last micro-op in a Way. • Micro-fused micro-ops (load+op and stores) are kept as one micro-op. • A pair of macro-fused instructions is kept as one micro-op. • Instructions with 64-bit immediate require two slots to hold the immediate.