HNHacker News
TopNewBestAskShowJobs

KenoFischer

4,368 karma · joined December 2, 2012

Julia Core Developer Co-Founder at Julia Computing

Reach me at keno@juliacomputing.com

submissionscomments
KenoFischer··on Home Chip Fab
The course number was ES277, though the professor left and it doesn't look like it's currently being offered under that course number. Not sure if there's a replacement class or the department decided to quietly shut it down. Wouldn't be surprised if it was the latter. The class was rumored to be the second most expensive one offered by the university, after the one where you got to grow organs in a petri dish. I also tried taking that one, but apparently not being a biologist was disqualifying.
KenoFischer··on Home Chip Fab
I love this! Nanofabrication was one of my favorite classes and even with fancy equipment and worked out ahead of time instructions making a transistor took a huge amount of time and a good number of missteps. Can't imagine the amount of work required to do it all from scratch. Though I also think the safety lectures scared me sufficiently that I wouldn't want to touch HF acid ever again.
KenoFischer··on First modern coreboot server platform
This is for the BIOS. However, excitingly, this is the same board that I'd spent some time porting various bmc efforts to, so we can now have both on the same platform. I need to get those patches rebased, but I've been running a board locally with both bios and bmc firmware open source.
KenoFischer··on Homomorphic encryption
There is a decent size effort to build a system that runs (a restricted, but hopefully useful subset of) Julia programs fully homomorphically (as well as supporting various sort of secure multiparty computation protocols). At JuliaCon two years ago, the Galois folks talked about their initial prototype of this work: https://www.youtube.com/watch?v=_KLlMg6jKQg (fun to watch even if you don't care about julia to see FHE "in action"). This effort was recently funded with the goal of extending the prototype into a full robust system, so I'm hoping for some good news here over the next couple of years.
KenoFischer··on Julia: Come for the Syntax, Stay for the Speed
Backspace gets out of REPL modes, though I agree that that isn't necessarily obvious for people not used to this kind of modality. I'm sure the ThinkJulia folks would appreciate a note so they can improve the tutorial: https://github.com/BenLauwens/ThinkJulia.jl/issues.
KenoFischer··on Composable multi-threaded parallelism in Julia
> given threaded code generally looks like it was written by Cthulhu

Whereas in Julia, we use Cthulhu to read code ;) - https://github.com/JuliaDebug/Cthulhu.jl.

KenoFischer··on JuliaCon 2019 Live Stream
Schedule here: https://pretalx.com/juliacon2019/schedule/

Each talk will have its own live stream on YouTube and you should be able to immediately watch the raw recording of past talks as well.

KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
Yes, I understand that, but i still think it's misleading. Even in Julia, we're still recording operations, just a) at a much coarser granularity (if there's no control flow in the middle we can record it as one operation) b) in the type of the tape data structure rather than explicitly

For example, one could imagine applying common subexpression compression to a tracer tape, and would get essentially the same thing.

Other places to stores this information are the stack of a closure chain, but it's still fundamentally the same information.

KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
Yes, this is exactly correct. We try very hard to extend Julia with "mechanisms" not "features", such that if you don't like the exact way we're doing it, you can just do your own (with blackjack), while still re-using as much of the rest of the system as possible. The same holds true for AD. We think (and we hope) that Zygote (and related packages) will be an extremely good, general purpose AD system that's a good default, but if you're in a specialized domain, you may be able to do a lot better with a specialized AD transform (e.g. your domain might want to do part of the computation in the frequency domain automatically, or have a domain-specific optimizations that are not generally valid, etc.). Our user base is to varied to be able to prescribe a one-size-fits-all approach. The only thing we can do is provide the necessary mechanisms and high quality default implementations (as well as good explanations as to how things work) and let it loose on the world. Everything being in the same language helps enormously here, since even users coming from the scientific side are well versed enough in the language to at least be able to read and understand the semantics of code in the compiler (even if they think it employs dark magic).
KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
JAX is a sophisticated and well implemented high level tracer, combined with a backend compiler that makes use of XLA. High level tracing has some fundamental draw backs in key features we want (efficient scalar AD for example), so the approach being advocated here is doing AD as a compiler transform on the original source language. Mike has some details in an earlier paper: https://arxiv.org/abs/1810.07951. As a shameless plug, you can plug (heh) Julia into XLA as well, e.g. for targeting TPUs (https://arxiv.org/abs/1810.09868). In fact, Julia and JAX are probably better at that than TensorFlow, because the TF execution model is a bit of a mismatch for what XLA expects.
KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
There's two competing currents here. One the on hand, Julia is extremely powerful and dynamic, so it has very high expressability for any possible differentiable programming you could think of. It also has a fairly simple core, so as long as you know how to properly transform the core, you can get a mathematically correct differential. However, the reason julia works so well is that the compiler is able to understand and eliminate all the dynamism and complexity for you at run time and generate very tight code as a result. When you add AD into the mix, the generated code becomes, much, much more complicated, so the compiler needs to work a lot harder. That can sometimes be a cause for frustration for us, because we get something really cool working with absolutely zero effort, but then need to go back and make the compiler better to reclaim some performance (static languages have the opposite problem, where you need to improve the type system in order to allow the thing to happen, but once you have described it in a specific enough type system, you usually also have good performance - if your language is any good that is).

Overall I think Julia is a pretty great language to implement AD (as evidenced I'd say by the 15 or so different AD packages that people have written for individual use cases before the latest push for a unified one), but it still is a very powerful language, so if you want your AD to handle the whole language (as we do), then you're gonna have to do a bit of work.

KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
Unfortunately, this isn't super easy since the time complexity heavily depends on what optimizations the compiler will apply and how the higher order AD is exactly implemented (there's many ways to do so that all give you the same answer and you probably want the system to pick the best for you). What we could do fairly easily however is to have a compiler introspection tool that runs through the computation, but doesn't do any of the work, just record how much work it would have done. That would easily give you and idea and let you plot a parameter sweep over any dimension you care about with actual results corresponding to the capabilities of the system.
KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
I'm not familiar with that part of the MLIR work, though I wouldn't be surprised if they are working at it. Google Brain is doing some of the finest work on ML for systems programming, so this'd be right up their alley. If it works well and they come up with some useful models, we'll be more than happy to incorporate them (or just use MLIR directly - there's a talk at JuliaCon next week from a Google engineer on using Julia as an MLIR frontend: https://pretalx.com/juliacon2019/talk/3YBZLC/).
KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily on expressability. We've been talking about automatic kernel generation for a while (and when I saw kernel generation what I mean is basically search for access patterns), but note that it's not quite as bad a problem in julia, because you can use higher order functions to get a lot of composability (e.g. if there's a hand-optimized parameterized matmul, fusing in inputs and outputs is just passing in extra functions). There's some academic work at the JuliaLab that's promising. We're also in discussions with commercial partners who care about this kind of thing to see if somebody is willing to fund it, but so far existing solutions have been good enough while we work on higher priority items. I do agree that being limited to a few hand-tuned kernels is a significant drag on research, so I hope we can do something here.
KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
I've read the corresponding paper, and while I think it's a fun read, I'm not sure it does particularly much to clear up the underlying confusion here. Reverse mode AD on straight line primitives is describabable in about three lines of exposition. The tricky part is how do you handle control flow (equivalently recursion). The description thereof then depends on what your underlying IR data structure is, but describing this generally isn't horrible either. The real problem in the literature is that there is lots of confusion over what is the AD algorithm and what are the workarounds applied to make it work in the particular system described.

For example, all the ML frameworks generally combine a high level tracer with a straight line AD transform, which then sometimes gets presented as a fundamental aspect of AD (which it isn't, you're just using the tracer to get an IR you can handle). As a result, this field has a lot of confused terminology, but all the results have been known since the 70s.

The name of the game for all the latest generation tools is to cleanly separate out the semantic AD transform from the rest of the system and then just use an unmodified compiler thereafter. We do it by transforming Julia IR, the Swift folks do it on SIL, and the Scala folks have an implementation using shift/reset and LMS. If you do this, a bunch of traditional AD techniques become just special cases of general purpose compiler transforms (DCE, CSE, etc). See this talk I gave recently for some more details: https://juliacomputing.com/blog/2019/02/19/growing-a-compile...

As an aside, a pet peeve of mine is people calling these algorithms "tape free". High storage requirements, due to the need to remember (or alternatively, recompute) intermediate values are a fundamental property of reverse mode AD and you can't get rid of it. The best you can do is hide it (e.g. in compiler managed stack or closure environments), with all the same fundamental challenges. This is probably a terminology clash, with people wanting to use "tape" as a term for a much more narrow kind of data structure generated from a tracing AD system, but the requirement to have some data structure like it, no matter how hidden is fundamental to the algorithm.

KenoFischer··on A Differentiable Programming System to Bridge ML and Scientific Computing
Happy that this paper finally made it to arxiv. The biggest reason for writing it was to try and showcase some of the breadth of applications we see for really high quality first class AD support at the language level. There are several communities that need this technology, so it makes sense to try and build one system that can address all of them and share tricks. I'm also hoping this gives people a sense of why our list of feature requirements for this system is so extensive (pervasive custom gradients, higher order AD, mixed mode, ultra fast scalar AD, etc). These days AD is often discussed only in the context of deep learning, but as the frontiers of deep learning are pushed and hybrid models become more and more popular, I'm expecting our focus on generality to pay off here.
KenoFischer··on Ask HN: Is there an emulator that could run the code available for Apollo 11
Yes: https://github.com/virtualagc/virtualagc
KenoFischer··on Python Ported to WebAssembly
Sure, but to what extent are they "showing" us that they "Ported Python to WebAssembly". Isn't this more of a "Show HN: Python packaged for Wasmer"? To my mind, part of the purpose of these show HN threads is to ask questions and give feedback about whatever it is that the post is showing, so I'm asking to what extent the Wasmer folks actually did any "porting", since (to me) that seems like the interesting part.
KenoFischer··on Python Ported to WebAssembly
Not sure what the "Show HN" part of this is, but here's a link to the original announcement post for Python-on-WASM by Mike Droettboom and the team at mozilla: https://hacks.mozilla.org/2019/04/pyodide-bringing-the-scien.... This seems to just be repackaging that.
KenoFischer··on Femtolisp: A lightweight, robust, scheme-like Lisp implementation
Used to implement the parser/lowering for Julia! It's probably on the way out at this point, but it's served us very well this past decade.
KenoFischer··on What to Know Before Debating Type Systems (2008)
ML Style constructors are not particularly idiomatic, but you can write basically the same thing in Julia:

    struct Cons{T, S}; head::T; tail::S end
    pairwiseReduction(f, a::Cons{T, Cons{S, Tail}}) where {T,S,Tail} = Cons(f(a.head, a.tail.head), a.tail.tail)
    pairwiseReduction(f, a) = a
(https://github.com/thautwarm/MLStyle.jl if you want the syntax) but that's somewhat beside the point.

I think the primary problem here is one of terminology. Take Stefan's quote from the post you're responding to:

> The primary difference between static and dynamic languages, in my view, is this: in static languages, expressions have types; in dynamic languages, values have types.

There is a clash in terminology here. People coming from a static languages background would use the term "tag" for what dynamic languages people call a type. There isn't really a super good word in the dynamic languages world for what static languages people call a type, and the word type is generally re-used, since the distinction isn't as important (generally there is an embedding of the tags into the type system, so using "type" for both can be sensible). I'm gonna adopt (my best interpretation of) the static languages terminology here for this comment.

Some languages have both types and tags. The biggest difference then between static and dynamic languages are which of the two is primarily dispositive (in the sense of determining the semantics of the language). In dynamic languages, the tags are, in static languages the types are. This choice has deep seated implication for the kind of programs that can be written and the behavior of the program.

In fact, you sometimes end up in situations where static languages have multiple tag systems (but generally only one type system) and dynamic languages have multiple type systems (but generally only one tag system). In julia, the tag system is primary, but we have a very rich type system that is embedded in the tag system by parametricity and is used for dispatch. This type system is constructed in such a way that the subtyping relationship is induced by tag system (for types T, S, (T <: S iff for all values v s.t. tagof(v) <: T, tagof(v) <: S)) [1]. You basically want to choose a decidable type system that satisfies this constraint, but is at the same time sufficiently expressive to be useful for what people want to do.

We also have another, larger type system that is used for inference judgements, but that is generally transparent to the user. It is in principle possible to type check over any of these type static system, but type checking is not part of the semantics of the language.

The problem with statements like "Julia's type system does not allow for X", is that it presumes that the type system having some feature is a pre-requisite for having the corresponding functionality. So while Julia's type system does happen have support for unions (and universally qualified unions), the significance of that is for dispatch, not for whether the language can have expressions that can take on values of different tags.

In any case, I can assure you that many people working on Julia are well familiar with Haskell's type system. Hopefully I was able to convince you that there is at least something interesting about the way Julia's type system works.

[1] I should credit the Northeastern PL group for the term "tag-based semantic subtyping" here to describe this notion (https://github.com/julbinb/ftfjp-2019/blob/master/paper/mini...).

KenoFischer··on Multi-Level Intermediate Representation Compiler Infrastructure
Yep, I've been reading the docs with great interest. I'm also looking forward to getting the XLA/HLO dialect available. I'm thinking that might be a neat way to get immediate use out of MLIR in the julia world. There's a bunch of extra HLO passes I need to write for TPU performance and I figure might as well try to do that using MLIR.

Good to hear that getting this into LLVM is still in the cards - totally understood that there are organizational challenges, just want to keep pushing in the right direction :). My schedule doesn't allow me to be at EuroLLVM this year unfortunately, but I'm sure we'll get an opportunity to catch up before long. Now that MLIR is out it'd be fun to dig in in detail.

KenoFischer··on Multi-Level Intermediate Representation Compiler Infrastructure
Congrats to the MLIR team for getting the source out. I know it must have been a lot of work. MLIR seems like (or at least aspires to be once complete) a nice step up from LLVM and addresses a lot of the issues that LLVM has around extensibility. Looking forward to see where this goes. In particular, it'll be interesting to see to what extent re-use of passes will work out between dialects in practice.

I'm also looking forward to digging into the polyhedral compilation stuff more. It's about time that somebody attempted a ground up re-think of that, so I'm looking forward to understanding it better. I've talked to Chris a number of times about the rest of the design, but we never got into the specifics of the polyhedral compilation.

That said, I think it's a bit unfortunate that this ended up in TF rather than LLVM. TF of course needs good compiler tech more than most projects at this point, which is why MLIR came from that direction, but MLIR is much more of a general compiler project. I worry that putting in TF will discourage the rest of the ML industry from adopting it as eagerly (everybody is already using LLVM, so that branding would make that much easier). Also, looking at the code, MLIR looks structurally a lot more like an LLVM-heritage C++ project than a TF one (hell, you build it by cloning llvm and putting MLIR in the projects/ directory). Perhaps moving it under the LLVM umbrella is still in the cards - I'll keep advocating for it at least ;).

KenoFischer··on A Julia Interpreter and Debugger
The history here is a bit as follows. Back about four years ago, I set out to write a debugger for Julia and while I was at it, I figured I might as well fix all of the other things that have always bothered me about debugging. As an example of that, it had the following features:

  - Being completely written from scratch in Julia
  - Mixed debugging between Julia and C/C++ code
  - Integration with https://github.com/mozilla/rr for time traveling debugging
  - Debugging windows programs running on a linux host under wine (and in particular even as an rr replay).
  - Zero overhead debugging by using mixed modes (DWARF based debugging generally, a much more accurate interpreter based one when possible).
  - Very precise location tracking, on an expression basis, not just a line-by-line basis
  - Multi-process debugging by riding along RPC calls, even across machine and even as separate rr recordings across machines
As you can tell from this feature list, that project was a bit over ambitious and in the end failed to be a particularly useful debugger for Julia. It was useable for a bunch of basic cases, but fairly brittle. Also, then 0.6 came around and all of Julia's internal APIs changed and porting over the entirety of Gallium would have been quite hard.

After 0.6 was released, I decided this wasn't working out for a Julia debugging story and pulled out just the interpreter and UI parts from Gallium (that package was called ASTInterpreter2.jl). This worked ok, but was lacking a bunch of features that I never got around to implementing, because the lead up to 1.0 required a whole bunch of compiler work that nobody else was available to do. This work now is based on that work from 0.6, but actually making it work nicely and paying a whole lot of attention to it that I was never able to. I'm very happy that Tim, Kristoffer and Sebastian decided to take this on, because I feel like my workload has only increased over the past year ;).

I still very much want all the ambitious features from above, and we do still have the code (and much of it is actually in use, just not for debugging) and I'm fully expecting them to make a recurrence. However, this time around, we'll start with a stable, usable system and gradually make it faster and add features, rather than building an unstable system with all the features and trying to fix it. The former approach is much better for getting usable tools out (though sometimes the latter can lead to more interesting results in a research setting). The other thing to note is that Julia is much more stable now, so it's easier to keep things running without breaking every few months.

KenoFischer··on Programmer Migration Patterns
Yes, we do indeed see a surprising number of C++ programmers in the user base (essentially people who are stuck in C++ because of performance considerations, but would like to iterate faster). MATLAB/Python/R users tend to come to Julia when they outgrow the performance constraints of their older systems.
KenoFischer··on Iodide: An experimental Mozilla tool for data exploration on the web
My preferred way to do this would be a compiler pass that rewrites everything to use the explicitly managed stack. I had started on writing that transform, but ran out of time. That's currently the major blocker for making the Julia/Iodide integration work well.
KenoFischer··on Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
> For example, the way dual numbers are used to extend the reals for automatic differentiation doesn't have a deep connection to duality in vector spaces.

Yes, the right way to think about dual numbers (esp once you generalize them beyond just the single e^2=0), is to think of them as tangent vectors (sections of the tangent bundle). I've never really liked the "dual number" terminology here. That's why I deliberately chose to use the duality of forward and reverse mode AD, because that notion of duality agrees with the underlying linear algebra (or in general differential geometry). I do agree it's a mess of terminology.

KenoFischer··on Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
Forward AD is the pushforward of a tangent vector (an element of the tangent space), Reverse AD is a pullback of a cotangent vector (an element of the cotangent space). The duality notion between tangent and cotangent spaces is the same as the duality notion of spaces in optimization. Unfortunately, I'm only passingly familiar with discrete optimization, but I would suspect the notion extends from optimization. That's not to say that they are fundamentally the same or that writing this down helps anybody in any way, but a lot of these "dual" notions do have some sort of dual vector space under the hood.
KenoFischer··on Julia on Google TPU: Shakespeare RNN
Another thing I should have mentioned here is that Julia's multiple dispatch helps a lot with this problem, since you can provide specializations in the dependent package. So e.g. Flux.jl only needs to have generic code, CPU and GPU code and I can provide TPU specializations (where necessary - hopefully not often), for any function that needs it (yes, talking to upstream is required here to, to make sure they're aware we're doing it, but at least they don't have to maintain it).
KenoFischer··on Julia on Google TPU: Shakespeare RNN
Oh, I should have mentioned somewhere. Credit for putting the notebook together goes to Elliot Saba (https://github.com/staticfloat) :).
← PreviousPage 9 of 17Next →