The Vision Intel, AMD and Nvidia Are Chasing: Heterogeneous Computing
wccftech.com
wccftech.com
Failure to consider data placement destroys much of the potential perf gain for getting the above right IMO. And unified memory won't make this any better if BW within a GPU is ~1 GB/s and system bandwidth is ~10x less. It will just sweep the problem under the rug.
Further, I have seen attempts to smear parallel computations system-wide, treating all the ALUs as equivalent, but this frequently breaks determinism when the compilers for the two different processor architectures create associatively different implementations of the same computation. And I see no resolution to that problem right now.
That said, GPUs have advanced enormously in the past decade. Branching is nowhere near the big deal it once was, the cache (L1 and register file) for a GPU core (an SMX not NVIDIA's silly redefinition of a SIMD lane) is surprisingly large, and insanely fast atomic ops can resolve a lot of potential concurrency issues (even deterministically if one prudently employs fixed-point math).
Unified memory definitely doesn't change this aspect. It just makes it a whole lot easier to do from a software engineering standpoint.
It is quite painful to deal with GPU buffers, memory mappings and DMA transfers. Unified memory makes this a bit easier and as an added bonus, eases latency hiding by allowing the driver to do DMA transfers transparently and reduces driver overhead from poking the CPU and GPU page tables.
For I think the real pain point is that GPU programming requires the ability to parallelize seemingly serial algorithms in a strongly-typed language in an age where 90+% of the engineers I know do all their coding in single-threaded, weakly-typed Python/R/Ruby/Javascript so they never get the opportunity to develop the relevant mindset to do so. As an aside, to hit close to floating point SOL on a CPU in SSE/AVX often requires a mindset similar to GPU programming (but sssshhhhh... don't tell the "recompile and run" peeps about that because it doesn't fit their comforting narrative).
And it's not a problem for me personally as a former C and assembler videogame and embedded system developer but it is a huge problem to find such talent these days because Python/R/Ruby/Javascript.
All IMO of course.
This is the easy part.
The hard part is getting this right in the big picture, to avoid having the CPU wait for the GPU and vice versa. Pipeline stalls will easily ruin your performance, to the point where single threaded CPU code runs faster because the high latency of data transfers.
This is where modern GPU memory management really helps. For example, in OpenGL, you can use GL_MAP_BUFFER_PERSISTENT and _COHERENT to map your buffers once right after you create them and keep the pointer to the mapped buffer for the entire lifetime of the buffer (see ARB_buffer_storage). When you do this, you're responsible for the synchronization so that the CPU and GPU never touch the same parts of the buffer concurrently. CUDA probably has something similar.
This adds some complexity to the application but now it's done explicitly in the application and there are no surprises. And the CPU/GPU page tables don't have to be modified, TLBs flushed and so on, so you get a bit of extra perf from there.
And I totally agree with the point about the mindset of single threaded programmers with dynamic languages. I was once put into a project where a scientific application was written in Python and some "big matrix" operations were done in C. The matrix ops were on the top of the profile so they assumed that it can be made faster by simply "porting" them over to GPGPU.
Well the matrices weren't that big (a few kilobytes) and the operations were done sequentially, one by one in the application. To make it work, the operations would have had to be batched but this would have meant a big refactoring to the entire Python codebase. This was way too complex for the chemists and phycisists who had written the code. They were really smart people with the domain knowledge but they didn't possess the mindset to think about making it efficient.
Once the underlying algorithm has been deconstructed, one then reconstructs it parallelized in CUDA/OpenCL/whatever. In my experience, the implementation language and APIs can easily lead one to a suboptimal realization of algorithms.
Once done, the original python code can serve as a reference implementation for conformance and unit testing. Of course, if you have 1M+ lines of legacy code, this probably won't work. But in my experience so far, the core algorithms have been less than 50K lines of code, and usually less than 10K.
IMO sometimes one has to stand one's engineering ground to persuade them to make the right decision. And bonus, one ends up a ground floor domain expert at the end.
Finally, I turned down a very prestigious project 4+ years ago because they wouldn't let me take the above approach. They opted instead to do it entirely with compiler directives. That project still hasn't officially shipped. It has, however, driven a significant amount of improvement in said compiler directives.
I can't wait to play with a Tegra X1 SOC! That's a whole lot of power in a an embedded device. Would be excellent for real time embedded video processing or computer graphics.
That should enable 3rd parties to author compilers targetting SPIR-V, allowing other programming languages without having to use C as an intermediate target.
(HSA Intermediate Language LLVM support)
It would seem an IR is already available and it's already integrated on an existing LLVM branch. From skimming around it seems AMD has been working hard to do what is necessary to get things merged into LLVM mainline.
<quote> My highest priority is to get the backend upstreamed as soon as possible, so I would appreciate feedback about any kinds of blocking issues on that.
Thanks
Matt Arsenault </quote>
http://lists.cs.uiuc.edu/pipermail/llvmdev/2015-May/085545.h...
LLVM IR has a few weak points for this kind of IR. It has architecture dependent parts and the format has changed in the past so it is coupled to LLVM development cycles. You can argue whether that's a good or bad thing but it's a thing to be considered.
There were two earlier versions of SPIR, coupled with different LLVM versions. SPIR-V is independent of LLVM but still semantically similar to LLVM IR, so compiling back and forth between the two isn't too difficult. AFAIK there has been discussions about adding a SPIR-V backend to LLVM trunk (can't find the mailing lists).
The SPIR-V spec is already out and there has been some projects already using it, although you can't run it on a GPU yet (unless you work for a GPU vendor, that is).
Some nice designs are out there, see Vobla and Pencil [1] for example, or Harlan [2] as another extreme.
[1] http://carp.doc.ic.ac.uk/external/dissemination.php
[2] http://blog.theincredibleholk.org/blog/2013/06/28/announcing...
I've been following the development of OpenACC, which is has been gaining momentum and lets you write GPU code directly with C with pragmas (like OpenMP). It's a bit more performance portable between platforms than CUDA or OpenCL with a low initial investment. I'm a little concerned about the lack of diversity of compilers supporting it, but it seems like the most feasible route to move those big codebases to heterogeneous platforms.
The problem with delegating tasks to GPGPU's is, that GPGPU cores can do only a very limited subset of what a CPU can do.
First of all, each GPGPU core has only a small amount of fast local memory available. In most applications, tasks need to be able to access databases, large in memory datastructures and what not. The GPGPU architecture is not designed for that. When data outside the core's local memory is needed, data transfer to and from individual cores is costly and when too much of that has to happen, the benefit of the many GPGPU cores rapidly disappears.
Second, a GPGPU core does not have access to other components, e.g. IO controllers. So it can't directly read from disk or write to the network. This again limits the types of work that a GPGPU core can do by a lot. For all network and disk access it needs to communicate with the CPU, and again, this communication is slow.
And finally there's a difficulty of another nature. OpenCL or CUDA does not integrate nicely with higher level languages such as Java. The article mentions Aparapi, which is one solution that tries to deal with this by compiling Java code to OpenCL. The trouble with this is that Java code is living in another world as OpenCL; code that works well for Java might perform horribly on the GPGPU, because the GPGPU's architecture with its memory locality is so different. So to write Java code that will be compiled to well performing OpenCL, you have to code it with OpenCL in mind, and you have to be aware of the manner in which Aparapi will convert your code to OpenCL. This defeats the purpose of Aparapi; it might very well be easier to just write your code in OpenCL so at least you have a clear view what is going on.
Can we have a smart VM that profiles the code and sees that kind of code and put it on GPU?
Modern GPUs can do full dynamic branching and process data structures that aren't flat (ie. vectors/matrices). It's not restricted to such simple tasks any more.
I don't think we'll ever have VMs that are smart enough to do decisions like this better than an engineer can. Or even close. What we really need is smart engineers who better understand what the technology is suitable for and can make good decisions on when to employ it.
As such I didn't think I could figure out how to make use of it for my applications (merging sorted lists of integers, dijkstra's algorithm etc) as typically you're branching and jumping around memory all over the place.
Can you suggest anywhere I can learn about these advances that mean the same rules don't apply?
Firstly, it sounds they are envisioning pipelined algorithms where stages are run on the best compute device. The CPU pipeline would presumably handle all asynchronous Apis and only call the pipelined algorithms once the data was available. It would be much easier to parallelize the code that can be easily be parallelized if the gpu context is the same as the cpu context.
The second part I think is up to compilers. If we can make better compilers for gpu code which integrate more nicely with cpu code then the super low level programming won't be as necessary. GPUs haven't been in the limelight nearly as long as CPUs, so it's understandable that most modern languages are made for CPU code. I'd personally like to see LLVM and PTX playing together nicely in Rust.
Humans will still have to consider the differences in optimal memory access / branching logic patterns, but that's understandable. We already do that - with the CPU in mind - so it will just take a bit of extra learning to consider the GPU.
> Second, a GPGPU core does not have access to other components, e.g. IO controllers.
Didn't AMD promise to solve both with HSA, which should allow (integrated) GPGPU cores to share all system resources with the rest of the CPU?
Some HSA benchmarks from AnandTech: http://www.tomshardware.com/reviews/amd-a10-7800-kaveri-apu-...
Note: the tests seem to use OpenCL and while they mention, "Looking at the HSA-based results..." I find it strange they are comparing the numbers against an Intel Core i3.
It might be more interesting to compare OpenCL HSA vs OpenCL non-HSA.
First: memory. A consumer GPU these days has 3 GiB of memory and a compute/viz GPU has up to 12 GiB. The memory bandwidth is about 300 GiB/s compared to 50 GiB per second for the CPU-DRAM. In other words: in memory intensive applications, the GPU wins. There's no "local memory" in modern GPU architectures (unlike 5 or so years ago), they have L1/L2 caches not unlike the CPU.
> For all network and disk access it needs to communicate with the CPU, and again, this communication is slow.
This is certainly true that the GPU can't do i/o, but "slow" is an ambiguous term. Yes, it is high latency and difficult to orchestrate effectively but the memory bandwidth of a PCI-e bus is 16 GiB/s (v3.x) to 32 GiB/s (v4.x), that's 30-60% of the bandwidth between the CPU and DRAM.
GPUs can handle complex in-memory data structures, and are very effective in processing e.g. tree structures. There's even been applications of GPUs to databases (the PG-storm backend for postgres, for example).
Modern GPUs are entirely different beasts to what the matrix and vector batch processing chips were a few years ago were.
Yes, it's a big shift in paradigms and requires some hard software engineering but the benefits are there. Case in point: the "deep learning revolution", in particular training the networks, would not have happened were it not for GPU tech.
http://www.theplatform.net/2015/07/29/why-hyperscalers-and-c...
If anything the "future" might be in on-demand core configs similar to how modern ARM SOC's are being built from a combination of "small" and "big" cores with different features and performance head space that allows for more efficient power usage for small devices.
So core-specific optimizations for different applications (Database, Web Server, Video Streaming etc.) might be quite possible especially when combined with user-programmable micro-code might be much better than FPGA's. Basically you might have a dedicated CPU (that you might even be able to partial design yourself by submitting VLSI code or something similar) which is specifically tuned for your application e.g. a web application server CPU with 4 big cores that handle executing the application code it self, with additional smaller cores that handle things like HTTP, SSL, session cache etc. each with dedicated register/extensions optimized for those actions e.g. LZMA or AES extensions.
Yes you can program the FPGA to do say LZMA very quickly but you can just as well build a dedicated SIMD ASIC that will do the same and at much lower cost.
If you design your CPU to include current ISA big cores combined with small mission specific SIMD ASIC's it will be better than building a CPU with additional FPGA cores or building a CPU on an FPGA which is an insanity to begin with.
I would yield that FPGA are more flexible but if you can build more or less flexible SIMD ASIC's that are user configurable you will get a similar flexibility to FPGA's with better performance still, eventually there's a limit to how many non-generic operations the CPU handles and there will be a fairly low limit on how many of those will deserve dedicated silicon before you over-optimize and lose performance.
Any problem I was solving on GPUs recently would have been implemented 100x times more efficiently on such an architecture.
But the tooling remains sufficiently awful (and closed) that it's not going to be accessible to average programmers any time soon. And what on earth is the security/multitasking model of an onboard FPGA?