I suppose my question is why would I choose to build something like this over using a GPU?
https://news.ycombinator.com/item?id=4702456
In terms of raw GFLOPS, you shouldn't.
Parallela offers something halfway between a multi-CPU cluster, differing in terms of memory access, and a GPU, differing in that each core is a real core and can be independently making calls, branching, etc.
Another point to note is that every GPU architecture is different and that some support a different degree of control flow parallelism vs. data parallelism.
Ultimately it depends on the kernel you're working with - if you're to a point where you've got strictly linear SIMD and you depend entirely on floating-point math throughput or memory bandwidth, it wouldn't make sense not to use a GPU instead.
Between ARM, OpenPower and Intel's Phi successors this is becoming a hyper-competitive space. Interesting times ahead!
Compared with your GPU, there's also another important factor: These are entirely independent cores. For stuff that your GPU does well - that is, stuff where you're doing the exact same manipulation of a large number of values in parallel - it's likely to keep beating the epiphany chips. For stuff it doesn't do well, that is, stuff where you need indpendent threads of execution for each data stream, the Epiphany chips may become a better fit.
It has 8 FPGAs on there too. Useful for highspeed IO and other tasks. Also, there are 8 HDMI outputs on there. Which would make this a pretty interesting video wall machine.
If it had the 64 core chips instead of the 16 core chips, this thing would be ~ 800 GFLOPS I guess.