I don't know. I'd imagine that the folks working on the EHT already are making use of plenty of HPC for image reconstruction. It's quite a different problem from the Celeste application of course, so this work isn't directly applicable, but if they ever wanted to rewrite their code in Julia, they should give us a shout ;).
The data is publicly available (as is the code), so you could just use it. The data itself is organized by region of the sky being imaged, so that calculation has already been done for us (and at the distance the stars/galaxies are at the difference in distance due to the Earth orbit doesn't matter). In the actual application, both the noise and the psf are modeled based on the actual telescope, so just adding arbitrary noise is a bit of a different problem, but probably a good start. Do note also that Celeste is a generative model (including modeling the noise), so you could just use it to generate a bunch of images for you.
Nobody is forcing anybody to do anything, but even compared to tech itself, the open source community tends to be less diverse. I think it's fairly clear that there is a large number of qualified people who are not adequately represented in the open source community at the moment. In order to not miss out on these people, we need to a) try to get more such people involved with the open source community through outreach, etc and b) once we get to that point in our hiring, make sure that our hiring funnel is not just restricted to the subset of the population that happens to already be active in the open source community. In my opinion, the key is just to have a sufficiently wide funnel at the top to avoid injecting bias at the top of the funnel.
I don't feel qualified to speculate on the implications for astronomy because that's not my field. However, Celeste is indeed a generative model and you can draw an image sample from the Celeste model and compare it to the original (there's a few examples in the Celeste issue tracker - https://github.com/jeff-regier/Celeste.jl/issues/625 is one I found on a quick look but there's more), so I think it qualifies in that sense. Of course you may want a more sophisticated model than Celeste for a complete model of the sky.
I'm not sure what the state of AVX-512 codegen was in the CLR/JVM back in early 2017, but given that LLVM's was still in buggy early stages, I would guess that it was a WIP at best so performance was probably not ideal (we had to work quite hard to get decent quality code generation). On KNL, you're either running AVX512 all the time or you're pretty much dead in terms of performance.
I worked on the computational side of this, so while I understand the science, I don't have a very good overview on the state of the art in astronomy. However, my understanding from the statisticians and astronomers on the project is that variational inference had not been attempted on this scale before (in Astronomy or otherwise).
Yep, that's something we need to fix. So far we've done most of our hiring from the open source community, which is unfortunately overwhelmingly male. We've done some work to try to increase diversity in the community and Jane graciously agreed to take a break from her PhD studies at Caltech to help with outreach as part of that (also thanks to the Sloan Foundation for funding the corresponding effort). Clearly a lot more work is needed on that front, but I'm hoping the work that Jane and others have been doing this past year have helped on the community side. Of course there's work to do on the company side independently, but I imagine the community will continue to be an important hiring channel, so I consider improving diversity there as a necessary ingredient.
| Are there significant research advantages to super-fast turnaround (less than 15 minutes, enabled by massive parallelism) in this domain?
No, the 15 minute turn around time is not important given the dataset we have at the moment, but showing that it was possible to do was considered important from a science perspective, because of the upcoming LSST telescope. LSST will generate an amount of data equivalent to the full dataset we had available every 3-4 days, so being able to scale this up far enough to accommodate that as well as future planned extensions to the algorithm, necessitated showing scale. The actual science runs by the project are usually done on a few hundred nodes over a couple of hours.
| Do you feel like a massively parallel system with Xeon Phi nodes is a good match for this problem? Or did the code get optimized to run at high scale on Cori Phase II because that's where you were given compute resources?
Cori Phase II worked well for this problem, though I wouldn't be surprised if GPUs wouldn't have been a better fit (though harder to program of course and at the time the Julia GPU infrastructure probably wasn't quite ready yet - even KNL was a struggle since LLVM was still in the process of completing support for it). The Celeste project is still ongoing (working on science goals more so than extra parallelism or performance improvements at the moment), but I wouldn't be surprised if there was an attempt to run on Summit at some point, especially now that Julia's GPU compiler is much more mature.
One of the biggest problems we failed to anticipate actually was getting the data from disk to compute units quickly enough. Early in the project we crashed the interconnect on the machine, so for the challenge run we weren't allowed to do anything other than pull the data directly from disk (lest we bring down the machine again while other challenge runs were ongoing). I haven't really looked at the interconnect on Summit, so I can't say how well it would handle that.
| Does this approach effectively scale down to e.g. a university that can afford a large storage array and beefy commercial servers
Yes, it scales fairly well. In fact you could probably do it with spot instances on a public could fairly well. The biggest thing would once again be getting the data to the compute units quick enough. That's quite demanding on the network (and ideally you want to pre-stage the data in memory). Certainly it's feasible to do this on a large-ish university cluster on the SDSS data set in a few hours. Probably less feasible on LSST data once that comes online, but maybe by that point improvements in computation speed and storage speed will have made up for that and it'll become feasible again.
The JIT compiler was active, but most of the computationally heavy parts were ahead of time compiled. The compiler had a number of tweaks to increase the quality of the generated code (though at the expense of a good amount of compile time). Generating the ahead of time image probably took a good 5-10 minutes (much more if we had tried to do it on the KNL nodes that we actually ran on), so it was infeasible and would have been a waste of time to try to do so in a JITted fashion. Some JIT compilation probably still happened (for code paths that weren't in the pre-compilation trace, e.g. logging code or exception code paths hit by a few processes).
As always in these sorts of discussions, it depends how you define your terminology. However, by most definitions of static typing, Julia's type system is not static. The julia type system is very much a property of the runtime language and behaves as such. It is quite strong, true, but still not statically enforced as you would expect from a static language. In particular, (to the extent that you can identify one), you never get any sort of compile-time type errors in julia.
Static typing has very little to do with what the compiler can say about your code. You can have dynamic languages with very strong type systems and semantics as well as static languages with weak semantics. The only difference between static and dynamic languages is whether the compiler enforces completeness of the analysis or not.
There's a bit back story there, so as a fun historical note, let me recount it here. In 2016, there was the Dirty CoW exploit (CVE-2016-5195), which got fixed by Linus in 19be0eaff [1]. Unfortunately, in that patch he forgot about transparent huge pages (they're easy to forget ;), which caused processes to lock up if you were using FOLL_FORCE on a memory range that happened to be backed by thp. That patch was widely backported to stable kernels, so I soon noticed my processes randomly freezing on various Linux machines. With a bit of debugging, I sent in a patch to fix that (https://lore.kernel.org/patchwork/patch/748110/), though I'm not sure that Linus ever saw it (or I guess he could have assumed we were using ptrace). Nevertheless, I assume that this being a common exploit vector for DirtyCoW made him want to remove it. As a result that was of course the second time within the course of six months that this stopped working (plus Linus' fault both times), so that email was written from a bit of a "you've got to be kidding me" mindset.
As an addendum in writing this comment, I became aware that my patch to fix FOLL_FORCE with thp actually re-introduced a version of the original DirtyCoW exploit apparently known as HugeDirtyCoW (CVE-2017–1000405) [2] - or rather my patch interacted badly with a pre-existing mistake in the thp code and caused this. Oops.
I love this site, the method is very clever. I did a re-implementation of this at one point for fun. One of my favorite things to do with this simulation is to actually plot the path of the photons as they approach the black hole (well, leave the black hole but you reverse time for ray tracing purposes). Conveys the idea pretty well that there's something funky going on with spacetime. E.g. here https://imgur.com/OPHLMJL (just a screen shot from back then, it's much more fun with sliders that led you move things around).
Well, yeah, that's because it's a piece of paper, not a calculator. More seriously, you have to be very careful "spooky action at a distance" in programming language design. For example, you might say something like "Well, if the compiler can prove that every constant involved is provable in rational arithmetic, do that, otherwise use floating point arithmetic", which seems like a perfectly sensible thing to do, until you realize that it means that very small changes to the code will have drastic effects on the correctness of your program (if suddenly the compiler is no longer able to prove rational arithmetic is the same answer). We do have that sort of thing happen with performance in Julia (and in many other languages too, of course) - if the compiler can prove it can do something faster, it'll try to, but we try, very, very hard to give everything predictable, sequential execution semantics (i.e. you should be able to take code, run it step by step and generally get the same answer as if you had executed it all at once). One particular example of this is comprehensions. In early versions of julia, the type of `[f(x) for x in a]` would depend on what the compiler was able to prove about `f` (and some other things), e.g. if the compiler knew that `f` always returned an `Int`, you'd get a `Vector{Int}`, otherwise a `Vector{Any}` (or whatever the compiler could prove). This turned out to be hugely annoying to people because the semantics of the computation changed (slightly, only an eltype which you generally don't look at), depending on what the compiler was or was not able to prove about your program. These days, we have runtime semantics for non-empty comprehensions and only use static information to improve performance (which as I said before, we do anyway, so users are used to it). That's not to say that you couldn't do a language that changes semantics based on whole program analysis (you could probably do it in Haskell for example), but you'd lose part of the gradual transition from REPL to full program, which is very important to a system like Julia.
You're probably thinking of complaints that our CI was flakey at times (failing despite changes being ok), which is true (though it's been pretty stable lately). However, people sometimes equate that with bad test coverage, which is absolutely not the same thing. Easiest way to get a test to pass is to delete it.
From my perspective at least, the julia test suite is quite comprehensive (if something passes the test suite, I'm generally happy to put it on master and have people use it). We're also lucky that there is so much open source julia code available, which we can use as essentially one giant regression test. We used that heavily in the lead up to 1.0 to make sure that there weren't any surprise bugs waiting for people.
Open a julia editor or the julia repl, type `\^1<tab>` and copy and paste the unicode superscript ¹ into your post. Or whatever else your preferred method of getting unicode characters is ;).
> However, what does the library function return to me as arrays it allocates?
Depends what the library function does of course. If it's shape preserving (e.g. if it's a map over the array), it'll generally preserve the axes of the input array.
> What if I do an outer product of two arrays with different base indices?
You'll get an offset array indexed by the product of the axes of the input:
julia> A = OffsetArray(1:3, 0:2)
OffsetArray(::UnitRange{Int64}, 0:2) with eltype Int64 with indices 0:2:
1
2
3
julia> B = 4:6
4:6
julia> A*B'
OffsetArray(::Array{Int64,2}, 0:2, 1:3) with eltype Int64 with indices 0:2×1:3:
4 5 6
8 10 12
12 15 18
> Whatever your answer, I suspect it is more cognitive overhead to remember than "always 1 based" or "always 0 based".
Sure, but the real point here is that in most situations, you don't actually care what the axes are, because you use the higher level abstractions (e.g. iteration over elements or the index space, linear algebra operations, broadcasts, etc.), which all know how to deal with arbitrary axes. The only time you should ever really have to think about what your axes are is if there's some practical relevance to your problem (OffsetArrays are used heavily in the images stack).
No, but it'd be fairly easy to activate, since LLVM supports it in code generation. The primary reason we don't is that it messes with stack traces too much.
I can understand why you might get the impression, but I'd encourage you to try out julia and see that we're really quite good at using index-agnostic abstractions, so most code doesn't care what your arrays are indexed with. If a certain set of indices make sense in your domain (0-based for FFTs as you say, symmetric indices about the original for image filters, 1-based for just regular lists of things, etc), just use it, and it'll be convenient interactively, but most library code doesn't really think about it that much.
Which is why julia doesn't make any assumptions on how your axes are indexed. If you're working in a numerical domain where 0-indexed arrays, or symmetric arrays about the origin, or arbitrarily other transformed axes make sense, just use those.
Pre 1.0, we've generally provided backwards compatibility for one version with deprecations. Somewhat ironically that has often led to people just living with walls of warnings until the version that actually broke it came out, which led to a worse experience. We also have automated upgrade tools now, which can do many of the simple (and some not so simple ones) automatically. The situation on 1.0 is slightly worse than in previous releases because we released 0.7 and 1.0 simultaneously to avoid having to ship 1.0 with active deprecations. Of course that means that people will have to fix their packages now, rather than waiting until next year.
Exception handling is one of the few subsystems that hasn't really had a revamp. It's probably one of the areas that'll get a good bit of thinking post 1.0.