Aside: this is not TensorFlow, but XLA, which are two very different things. It's also possible to try this kind of thing and generate a TF graph, but TF is a much less nice compilation target.
4,368 karma · joined December 2, 2012
Reach me at keno@juliacomputing.com
Aside: this is not TensorFlow, but XLA, which are two very different things. It's also possible to try this kind of thing and generate a TF graph, but TF is a much less nice compilation target.
- Does what you have to write to run on X follow the semantics of the language?
- Can you use data structures/code defined in libraries that don't know about your thing?
In the case of julia on TPU here, the answer is yes! (surprisingly perhaps, and getting this to work is pretty hard). In particular, you get a lot of julia's language features: Multiple dispatch, control flow, etc. It's a bit of a subset of the full language (e.g. no mutation at the moment), but everything that's supported is just standard julia and we're working on growing that subset more and more.That approach is very different from something like TensorFlow where you're essentially metaprogramming an expression graph. Numba probably counts for python (yeah, you have to put an annotation on things, but if the python people really wanted, they could probably import Numba into core cpython and make it work more smoothly). Of course in python, you have the additional complication that most of the core implementation is not in python itself, so even if you satisfy my two criteria above for the core language, you're still gonna have to rewrite the whole standard library.
1. To use a small dataset that can be fed from the relatively small Colab VM (so people can play with it themselves)
2. To use a well known model (so people can focus on the implementation parts of it)
There's a resnet example in the previous example in the series (though the dataset is too large to feed from Colab).Also, this is very early stage software, so things are a bit more verbose than they should be :).
Meanwhile, we have absolutely no problem statically checking Julia code (and in fact we do when generating code for TPU). It's a bit of a non-standard thing to do and we're working on improving the tooling for this, but putting Julia on the same level as python here ignores a significant aspect of the design of Julia.
As for collaborations among thousands of engineers, it is true that we don't really have Julia projects that large yet (other than Julia itself) and I'm sure we'll undoubtedly find things to improve in the language to support collaboration at that scale, but i really would encourage you to look at things like the new package manager or the documentation system. A significant amount of work has gone into the usability thereof and I think at that point they warrant criticism more detailed than an off hand dismissal.
Of course for DL, people routinely use reduced precision as you have alluded to. There, we're looking at exascale at the moment. The second machine mentioned in the article (Summit) has ~3EF of Float16 performance (of which their application reached ~1EF). For comparison (and assuming about 50% MXU utilization - which is about the maximum I've seen), you'd need about 200 TPUv2 pods (at $384/hour/pod for the general public) to reach the same scale on this problem. Now Google does have that scale, but it's certainly non-trival.
Half-precision petascale is fairly routine these days. You can get that on the cloud for < $100/hr and I do routinely spin up such systems for ML training. However, petascale fp32/fp64 and exascale (b)f16 as discussed in the article are still fairly rare and usually preceded by months of planning to make sure things go right and the computer power is used usefully.
Disclaimer: not involved in the work, but working closely with the folks who are and I've used both the mentioned systems myself.
(def kernel-source
"extern \"C\"
__global__ void increment (int n, float *a) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
a[i] = a[i] + 1.0f;
}
};")
Is there an example where the kernel is written in Clojure?Yes. Watch HN in the next week or two for an announcement that may interest you ;).