What is a flop?
nhigham.com
nhigham.com
* MIPS was perhaps the integer equivalent to FLOP, still used in modern microcontrollers because the 8051 at 12MHZ would only execute 1MIPS (12 clocks per instruction). Modern 8051 chips obviously have sped up to 1 clock per instruction, but MIPS (and Dhrystone MIPS in particular) are still a common benchmark today.
* FLOPs is very difficult to calculate in theory because modern CPUs have vector units, and multiple pipelines per core. You could have 3x AVX512 instructions in parallel on today's CPUs on a single core.
* FLOPs we're traditionally a 64-bit operation for the supercomputer community. Today, most FLOPs are 32-bit for video games. Finally, the deep learning / neural net guys have popularized 16-bit flops, and even 8-bit iops.
* 'The' flop is a misnomer because it's almost always the multiply-and-accumulate instruction: X = A + B * C. Which... Is two operations per instruction (per shader/SIMD lane). Eeehhh whatever. Who cares about these details?
* As 'Dhrystone' is the benchmark for MIPS, the benchmark for 64-bit flops is Linpack.
If FMA is supported, it can either be counted as one or two operations, depending on the rule of the benchmark involved or the marketing of the processor. The marketing specification of a processor's theoretical peak performance sometimes counts a single-instruction FMA as two operations. On the other hand, for the purpose of code profiling, counting FMA as one operation is more realistic... As you said, who cares about these details?
This idea that additions and multiplications can be combined like this as "equivalent operations" is kinda bullshit. But hey, if its "how its done" (and its done this way because multiply-then-add is how you do matrix-multiplications...) then so be it.
Just remember that this is an arbitrary subdivision of a matrix multiplication operation, that may not have much relevance as a benchmark outside of matrix multiplications.
Unrelated to the thread but I just read that Top Fuel dragster engines only do about 240 revolutions over the 1000 foot race[1].
Never thought of it like that before so blew my mind a little.
A modern processor is typically superscalar, with multiple individually pipelined floating point units that can each be issued one instruction per cycle: regardless of whether that is a `fmadd`, a `fadd or something else.
If you'd issue a `fmadd` to a unit, then you are prevented from issuing a `fadd` to it in that same cycle, and vice versa. That's one op, not two.
However, floating point units are sometimes heterogeneous. I think the question should be, rather: should you count the throughput of any ops, or of arbitrary ops. In other words, if a CPU runs at 1 MHz and has two units but where only one can run `fmul`, should you count the CPU as having one or two MFLOPS?
I don't think there's enough information to answer that question. For starters, even if the CPU was issued a long sequence of nothing but additions, it's possible that it might spend some fraction of that million cycles waiting on memory.
I don't think FLOPS are counted from first principles, I think they're measured empirically using benchmarks. It's possible one benchmark will yield 1.25 MFLOPS and another 1.8 MFLOPS. Split the difference and call it a day.
Actually the origin of this term was VAX MIPS (VAX 780 specifically) because that was a ubiquitous, pretty fast for its time minicomputer. There were faster machines, and slower mainframes still being built, but that was what the late 70s were like.
When the 8051 was released in 1980 it surely didn’t run at 12 MHz! Back then the Z80 sold because it could run 8080 code at a blistering 2 MHz.
BTW the benchmark for FLOPS in those days was Whetstone, hence the otherwise weird name “Dhrystone”
http://bitsavers.informatik.uni-stuttgart.de/components/inte...
It was 12T clocked: even though the clock was 12MHz, it would only operate at 1MHz / 1MIPS, because it took 12-clock-ticks to even perform one addition.
IIRC, there was a standard crystal (11.0592 MHz crystal?? I forget exactly) for the communications at the time. So going just above 11 MHz (or really, just above 11.0592 MHz) was needed for reliable serial comms.
(Love those old Intel books too)
Nevertheless, FWIW, MIPS started out as Vax MIPS, and at first people often used to write "VAX MIPS".
Realistic code on a 8051 runs only about 450k-800k instructions per second @12 MHz.
If you do exclusively square roots over a giant matrix, being told the FLOP is based on a single 64 bit float division might not help you. (I cannot conceive of a problem solved only in square roots. This is a hyperbolic/reductive argument)
Given the distinction of single/parallel a lot of FLOPS stories and MIPS stories were somewhat mythic anyway. Then manufacturers started gaming the test suites.. it was always very notional.
I ask, if IOPS is in fact any better? or SQL transaction models? I might be stuck in a problem which is a LEFT OUTER JOIN do I not need to know that cost distinct from a well optimised single B Tree index cost?
I worked on an 11/780. Many in fact, both as an operator and as a user. It was awesome. But, I reminded myself the Dec-10 alongside had happily been supporting 70-80 users in parallel, running MUD, being a sort-of mainframe, I had no conscious sense of how "big" an IBM 360 was by comparison. A Vax 11/780 was however, really quite fast at processing the same FORTRAN jobs which ran on the Dec-10. The Dec-10 had a floating-point systems "ice cream cabinet" glued on the side as an array processor. It ran tasks compiled to run on it very very fast but that was a narrow problemspace. This was 1982-3.
Division and square root are generally slower than the other arithmetic operations, in both latency and throughput. They are finally partially pipelined in recent CPUs (a result every two or three cycles), but were totally unpipelined in mainstream designs for many years before that. A decade ago, they might take a few tens of cycles, now they’re generally somewhere around ten cycles latency on “real” CPUs, vs 3-5 cycles latency for the other floating point arithmetic instructions.
Anyway the question is. If you implemented fast inverse square root with out the hack. (i don't know, perhaps packing your own bits, or it might be screwball enough you only assembly could do it.) would it be as fast?
It's not actually that fast, either. Every mainstream architecture from the last 15 years (except maybe RISC-V, since it's exactly the sort of thing they would forget to add) has a "approximate square root" instruction with single-cycle throughput, and they're all both more accurate and more efficient than this. It was good for its time, but it's time has passed.
The author of this post seems to have fallen for this error.
FLOPS = FLoating point Operations Per Second, akin to the power of a given machine (which could be measured in Watt = Joule per second)
A task of W FLOP on a machine capable of P FLOPS will take T=W/P seconds. Or, a machine capable of executing a task of W FLOP in T seconds is capable of P=W/T FLOPS.
TL;DR: 1 FLOPS = 1 FLOP/sec