I've been wondering lately if the modern compilers should all be using C as the intermediate language (or some language-specific code optimization opportunities could be lost if they do that).
You can't use C's casts (undef for out of range float -> int conversions, for example), arithmetic (undef for signed overflow), or shift operators (implementation-defined behavior for signed right shifts, undefined behavior for left shifts into the signbit or shift counts not in [0, n)). You can work around these by defining functions with the semantics that your language needs, but they get gross pretty quickly (they are both much more verbose and more error-prone than having an IR with the semantics you really want, and they require optimizer heroics to reassemble them into the instructions you really want to generate). Alternatively, you can use intrinsics or compiler builtins, but then you're effectively locking yourself to a single backed anyway, and might as well use its IR.
The issues around memory models (especially aliasing, but also support for unaligned access, dynamic layouts, etc) are worse.
Even LLVM IR is too tightly coupled to the semantics of C and C++ to be easily usable as a really generic IR for arbitrary languages (Rust, Swift, and the new Fortran front end have all had some struggles with this, and they're more C-like than most languages). C is much worse in this regard.
The behavior of shift operations on signed integers will be fixed in C++20 and C2x, as part of the effort to require twos complement representation. It is a massive potential source of UB in currently standardized C and C++.
All the other problems listed remain.
[1]: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p090...
* It's missing several useful operators, such as classic bit manipulation (count trailing zero, byteswap), or even 8- and 16-bit arithmetic. Checked arithmetic is another useful one that's not present (or even really possible in C's ABI).
* Signed integer overflow is UB.
* Utterly no support for SIMD types.
* Proper IEEE 754 floating-point control is kind of spotty, although it tends to be as bad or worse in most other languages.
* ABI control is poor. You can't come up with any way to return multiple register values, for example.
* Anything that's not a vanilla function isn't supported. No uparg function support (required for Pascal), multiple entry points (required for Fortran), or zero-cost exception handling (required for C++). Hell, even computed goto isn't actually supported.
And all of this is assuming you have strong control over how you expect implementation-defined behavior (e.g., sizeof(int)) to work.
But even the "better C" languages tend to not really attempt to expand C structurally. The changes amount to fixing the egregious semantics (fixed-size types, no int-promotion, define signed overflow, etc.), add vector types and other operators, maybe tweak ABI a little bit, and add a whole lot of syntactic sugar. And those languages that explore beyond C's limited structural repertoire do so at the cost of C's specificity.
That said, ever since the last time someone asked me this kind of question, I've been trying to design a portable assembly language.
>> I've been trying to design a portable assembly language.
Couldn't something like Forth fulfill this role?
Do you mind expanding on this or pointing me to places where I can read more?
LLVM's floating point instructions assume that there is no floating point environment [1]. And there's no real facility to indicate that floating point instructions might be affected. To remedy this, they've been working on adding constrained floating point intrinsics.
[1] More specifically, that the environment is set up to the default rounding mode (round-nearest), all exceptions are masked, and no one will ever care about sticky bits.
It looks like the only major compiler that actually supports IEEE 754 correctly is icc. MSVC, gcc, and clang all optimize floating-point operations without considering if the dynamic environment is the same.
On a more theoretical level there has been some research on what a better intermediate language would look like. One project I found interesting was Mu VM, which offers some niceties for compiling languages with a garbage collector.
IR's should be terse, simple and dumb. I'm not sure any "real" programming language fits that.
When you have a higher level language with more accurately defined semantics, running it all through C would risk introducing undefined behaviour.
With an IR you can control and define the semantics more closely to what your language needs.
No, it wouldn't. When you target C you need to write a proper backend for its abstract machine, rather than naively rewriting code, of course.
The C abstract machine is a fine IR, specially the later editions of the standard.
Outside of the language-specific front end, compilers generally have no knowledge of the programming language itself. There is no technical advantage to transforming Rust into C when it comes to the middle and back ends, which form the bulk of the compiler.
There are no language-specific optimization opportunities. There are, of course, restrictions on what you can do in some languages that eliminate optimization opportunities, but you're not suddenly going to be able to take advantage of those opportunities by transforming your code into a langue that lacks the restrictions, because then you change the semantics of your code.
There is a key one: the ability to use any C compiler out there (including proprietary ones). This allows you to target all platforms out there.
Plus very few platforms have only support for C and nothing else, unless we are speaking about esoteric embedded CPUs.
Then you can also add custom CPUs and systems, FPGAs, etc. Those are way more rare, but still something people use daily in some industries.
This "C is faster than C++" is a bit dated by now.
"I said C++ is not faster than C" implies that C++ compilers don't beat C compilers, which as many in HPC, HFT and GPGPU computing domains know is false for years now, and no restrict doesn't help that much against template metaprogramming and constexpr.
Template metaprogramming and constexpr doesn't help being faster in HPC or GPGPU, it helps reduce the redundancy of your code, for example if you want a generic algorithm on float, double, int, complex.
What helps speed is being able to control memory allocations and having the tool to place the data required on registers, L1 cache or L2 cache as required by your kernel (and similarly for GPU).
On current architectures, what is hard to optimize is memory and data movement, if your data is at the wrong place or not prefetched at the right time it will be literally 100 times more costly than a saved addition from constexpr.
"Scientific Computing: C++ Versus Fortran" (1997)
https://www.drdobbs.com/cpp/scientific-computing-c-versus-fo...
"Micro-Optimisation in C++: HFT and Beyond"
http://research.ma.cx/NDCTechTown_2017_JMMcG_v1_1.pdf
"The Speed Game: Automated Trading Systems in C++"
https://www.youtube.com/watch?v=ulOLGX3HNCI
"When a Microsecond Is an Eternity: High Performance Trading Systems in C++"
Anyway, I stand by what I say and I'm backed by my high performance code:
- Writing matrix multiplication that is as fast as Assembly, complete with analysis and control on register allocations, L1 and L2 cache tiling and avoiding TLB cache miss:
- https://github.com/numforge/laser/blob/master/laser/primitiv...
- Code, including caveat about hyperthreading: https://github.com/numforge/laser/blob/master/laser/primitiv...
- The code is all pure Nim and is as fast/faster than OpenBLAS when multithreaded, caveat, the single-threaded kernel are slightly slower but it scales better on multiple cores.
- I've also written my own multithreading runtime. It's scale better and has lower overhead than Intel TBB. There is no constexpr, you need type-erasure to handle everything people can use a multithreading runtime for, same comparison on GEMM: https://github.com/mratsim/weave/tree/v0.4.0/benchmarks/matm...
- More resources on the importance of memory bandwidth: optimization convolutions https://github.com/numforge/laser/wiki/Convolution-optimisat...
- Optimizing matrix multiplication on GPUs: https://github.com/NervanaSystems/maxas/wiki/SGEMM, again it's all about memory and caches optimization
- Let's switch to another domain with critical perf need, cryptography. Even when knowing the bounds of iterating on a bigint at compile-time, compiler are very bad at producing optimized code, see GCC vs Clang https://gcc.godbolt.org/z/2h768y
- And crypto is the one thing where integer templates are very useful since you know the bounds.
- Another domain? VM interpretation. The slowness there is due to function call overhead and/or switch dispatching and not properly using hardware prefetchers. Same thing, C++ constexpr doesn't help it's lower-level, see resources: https://github.com/status-im/nimbus/wiki/Interpreter-optimiz...
Also all the polyhedral research, and deep learning compiler research including the Halide compiler, Taichi, Tiramisu, Legion, DaCE confirm that memory is the big bottleneck.
Now since you want to stop on the theory and you mentioned HPC, pick your algorithm, it could be matrix multiplication, QR decomposition, Cholesky, ... Any fast C++ code (or C, or Fortran or Assembly) that you find will be fast because of careful memory layout and all level of caches, not constexpr.
If you have your own library in one of those domains I would be also very happy to have a look.
As a simple example, let's pick an out-of-place transposition kernel to transpose a matrix. Show me how you use constexpr and template metaprogramming to speed it up. Here is a detailed analysis on the impact of 1D-tiling and 2D tiling: https://github.com/numforge/laser/blob/master/benchmarks/tra..., throughput can be increased 4x with proper usage of memory caches.
I guess that is why NVidia has spent 10 years doing hardware design to optimize their cards for C++ execution.
Apparently that was wasted money, they should have kept using C.
I switched to practical applications and walk the talk showing my code, and then you back off and want to go back to opinions.
I see now that you want to back myself with experts since reproducible code and runnable benchmarks is not enough.
Apparently you recognize Nvidia as an expert so let's talk about CuDNN where optimizing convolution is all about memory layout, source: https://github.com/soumith/convnet-benchmarks/issues/93#issu... and it's not about C vs C++ vs PTX.
Or let's hear about what Nvidia says about optimizing GEMM: https://github.com/NVIDIA/cutlass/blob/master/media/docs/eff..., it's all about memory locality and tiling.
Or maybe Stanford, the US government and Nvidia Research are also wrong when pouring significant research in Legion? https://legion.stanford.edu/
> Legion is a data-centric parallel programming system for writing portable high performance programs targeted at distributed heterogeneous architectures. Legion presents abstractions which allow programmers to describe properties of program data (e.g. independence, locality). By making the Legion programming system aware of the structure of program data, it can automate many of the tedious tasks programmers currently face, including correctly extracting task- and data-level parallelism and moving data around complex memory hierarchies. A novel mapping interface provides explicit programmer controlled placement of data in the memory hierarchy and assignment of tasks to processors in a way that is orthogonal to correctness, thereby enabling easy porting and tuning of Legion applications to new architectures.
Are you saying they should have just called it a day once they were done with C++?
Or you can read the DaCE paper on how to beat CuBLAS and CuDNN: https://arxiv.org/pdf/1902.10345.pdf, it's all about data movement. 6.4 Case Study III: Quantum Transport to optimize transistors heat dissipation, Nvidia strided matrix multiplication was improved upon by over 30%, and this part is pure Assembly, the improvement was about better utilizing the hardware caches.
But then since you saw it was a lousing battle going down that path, you pulled the hardware rabbit trick out of the magician hat.
So we moved from C++ is not faster than C assertion, to memory layouts, hardware design and data representation.
Now you are even asserting that it's not about C vs C++ vs PTX, and going down quantum transport lane?
Yeah, whatever.
There is no C vs C++ issue, you keep saying that constexpr and template metaprogramming matter in high performance computing and GPGPU, I have given you links, benchmarks and actual code that showed you that what makes a difference is memory locality.
Ergo, as long as your language is low-level enough to control that locality, be it C, C++, Fortran, Rust, Nim, Zig, ... you can achieve speedups by several order of magnitude and it is absolutely required to get high-performance.
Constexpr and template metaprogramming don't matter in high performance computing, prove me wrong, walk the talk, don't drink the kool-aid.
There are plenty of well studied computation kernels you can use: matrix multiplication, convolution, ray-tracing, recurrent neural network, laplacian, video encoding, Cholesky decomposition, Gaussian filter, Jacobi, Heat, Gauss Seidel, ...
Regarding features that may not be properly optimized, there are exceptions and move semantics I guess, but exceptions are often avoided like the plague anyway and code can be refactored to get the same effect as move semantics.
Am I missing something?
(C itself is not specified very thoroughly but C - a C implementation - is, in the sense that it only does one thing for a given line of code)