Hacker News Reader

Comment by m00dy | Hacker News Reader

How did you convert time to the cycles ? Cycles are more about instruction set.

Sure, put a "roughly" in there. It's a simple multiplication by clock speed. This is Justine who designed APE. I think they know.

Edit, and also no, cycles are not "more about instruction set". They're pulses of electricity that make their way through a chip, they are about as far from instruction-set-specific as it gets.

I've rounded the number and added a tilde since you're right. There's actually a lot of interesting depth behind this subject for x86 alone. RDTSC timestamps are guaranteed to be invariant across all models irrespective of clock speed (except k8) so a mapping exists between RDTSC reported clock cycles and nanoseconds; however, it usually isn't accessible on client grade hardware and has to be measured haphazardly by the operating system. In my experience on CPUs the last ten years it's usually in the ballpark of .323018 for both Intel and AMD. Multiply ticks by that and you get a pretty good nanosecond approximation. If you want a fixed point expression for your benchmarks:

    static inline unsigned long ClocksToNanos(unsigned long x, unsigned long y) {
      // approximation of round(x*.323018) which is usually
      // the ratio between inva rdtsc ticks and nanoseconds
      unsigned long difference = y >= x ? y - x : ~x + y + 1;
      return (difference * 338709) >> 20;
    }

That's just one quick and dirty way your approximation could be approximated. Should work great for any benchmark games. Obviously don't use it for like, an X-ray or something. You can get the RDTSC value as follows:

    static inline unsigned long Rdtsc(void) {
      unsigned long Rax, Rdx;
      asm volatile("rdtsc" : "=a"(Rax), "=d"(Rdx) : /* no inputs */ : "memory");
      return Rdx << 32 | Rax;
    }

Intel also has recommendations for things like using CPUID and memory fences if you need to reign in speculative execution, but they can be costly. If you want to go even deeper then RDTSCP has a nice feature that lets you know if the operating system switched you to a different core during your measurement interval because it gives you TSC_AUX.

I'm not sure whether any of that timing precision will do any good on GitHub Actions, where this seems to be running. I don't think the machine is going to be quiet enough to get a good measurement of anything.

Noise and jitter in the VM should cancel out with repeated measurements.

Here's something interesting. If I run my test in a RHEL5 (Linux c. 2010) VirtualBox VM then it actually goes faster than a modern Linux kernel running on bare metal, taking only 14µs. If I run it in a RHEL7 VM then it needs 131µs.

So >1ms hello world execution should be a red flag. If that's actually the latency of GitHub's kernels and can't be explained by something like a shell script doing the time interval measurements, then it'd be interesting to learn what it's doing.

It appears helloworld is the only test with any repeats, and it only has 5 repeats. https://github.com/hanabi1224/Programming-Language-Benchmark... (Also, you can't average out the VM scheduling something else during the test, if it is constantly scheduling other stuff as it is a very busy build machine.)

Here's the measurement code, it appears to be significantly more complicated than a simple fork/exec/wait loop but that could just be all the C# getting in the way: https://github.com/hanabi1224/Programming-Language-Benchmark... Note that we are definitely measuring the C# async runtime to some degree. Nevertheless you are probably right that the bulk of this 1.8ms is in the executable under test, and it truly is just bloat. Running `hyperfine ./empty-main-function` from rustc on my Mac gives 0.8ms.

Looks like it might be async C# thread sleep code launching bash launching the benchmark.

Is that due to SPECTRE & Meltdown mitigations?

I disable Spectre mitigations on my Linux workstation since it doesn't host VMs or browsers. I'm sure if those were enabled it'd go much slower. But either way it's pretty clear that a lot of stuff got added to the kernel since 2.6.18 that slows down process launches dramatically, then some of it probably got optimized, and the rest stuck. Would be interesting to know what it is.

You should be able to disable them for your VM guests, but I don’t know if VirtualBox does.

I'm guessing from a bit of googling that APE stands for the Actually Portable Executable[0] (which once I found I realised I had read about before, but I wasn't aware of the acronym).

[0] https://justine.lol/ape.html

It was mentioned in the top level comment.

Dumb question: why are cycles and instruction set not related?

Because instruction set is the interface and the architecture is the implementation? Or something else?

Very simple: CPUs do not execute a constant number of instructions on each cycle. Some instructions are really slow, and their slowness is measured in cycles. You can usually do one XOR in one cycle, because it's a really simple piece of circuitry. Some instructions have complicated circuitry, which means that it takes multiple pulses of electricity to move the input signals through all the logic gates inside. XOR can do it in one pulse, but a division might take 20 pulses. Cycles are just pulses.

One important component of a program's performance, especially in numeric code, is how many instructions it manages to execute per cycle. If you have written your program's instructions in a 'good' order, then the CPU will e.g. not queue up many divisions in a row that depend on each other. It can do two different things at the same time. You can't do anything in less than one cycle, but you can do two of the same thing in the same cycle if you have two available execution units for that task.

The most direct answer to your question is that different instruction sets have different ideas about how much work a single instruction should do. The basic divide is between CISC (complex instructions, x86 qualifies) and RISC (basic instructions, ARM). It is a dramatic oversimplification but one instruction in a CISC architecture might involve 10 instructions in a RISC. So you can't compare instruction counts between architectures. But you can compare cycles and still make a bit of sense.

If this is interesting to you, you should try this game, where you can build a CPU purely out of the most basic logic gates: https://nandgame.com/

A cache miss stalling the CPU can be worth 100 CPU cycles or more.

So, instructions can have different pulses. therefore, different cycles. How did we end up 1.8ms to 5M cycles ?

2.6 Ghz * 1.8ms = 4,680,000 CPU cycles. To reiterate, this is independent of instructions. That's the number of CPU cycles it took to execute hello world.

thank you.

Cycles are a function of time and clock rate. Most CPUs run at 1-4 GHz, so multiplying seconds by a billion gives you a cycle estimate within a factor of 10.

Reply on news.ycombinator.com