Julia 2.0 isn’t coming anytime soon, and why that is a good thing
logankilpatrick.medium.com
logankilpatrick.medium.com
It's the opposite. Julia 2.0 would not bring standalone executable compilation, but instead would require compiler work for language changes. The whole point of saying "we don't need to do a 2.0 right now" is that folks on the compiler team believe working on things like standalone compilation and improved caching of LLVM code (i.e. fixing TTFP) are a better use of time than changes to language semantics (i.e. a 2.0).
And TTFP does need to be addressed. It's a current shortcoming of the compiler that native and LLVM code is not cached during the precompilation stages. If such code is able to precompile into binaries, then startup time would be dramatically decreased because then a lot of package code would no longer have to JIT compile. Tim Holy and Valentin Churavy gave a nice talk at JuliaCon 2022 about the current progress of making this work: https://www.youtube.com/watch?v=GnsONc9DYg0 .
This is all tied up with startup time and are all in some sense the same issue. Currently, the only way to get LLVM code cached, and thus startup time essentially eliminated, is to build it into what's called the "system image". That system image is the binary that package compiler builds (https://github.com/JuliaLang/PackageCompiler.jl). Julia then ships with a default system image that includes the standard library in order to remove the major chunk of code that "most" libraries share, which is why all of Julia Base works without JIT lag. However, that means everyone wants to have their thing, be it sparse matrices to statistics, in the standard library so that it gets the JIT-lag free build by default. This means the system image is huge, which is why PackageCompiler, which is simply a system for building binaries by appending package code to the system image, builds big binaries. What needs to happen is for packages to be able to precompile in a way that then caches LLVM and native code. Then there's no major compile time advantage to being in the system image, which will allow things to be pulled out of the system image to have a leaner Julia Base build without major drawbacks, which would then help make the system compile. That will then make it so that an LLVM and BLAS build does not have to be in every binary (which is what takes up most of the space and RAM), which would then allow Julia to much more comfortably move beyond the niche of scientific computing.
I am happy that there is progress on the code caching front and will look to the link you shared. Thanks for elucidating this further!
At least for my workloads, I often write a small .jl script that imports everything I want and then runs some functions on a list of files, for instance, or I just do my work in a Jupyter notebook, and I pay the load cost once for hours or days of computation.. Annuitized over the entire time I might use the kernel the startup time is basically nothing.
I'm not saying the load time should be a lot lot faster! That would be awesome! But I see folks complain about this and I'm wondering if they really need to be booting a new kernel constantly or if they could adapt how they work a little bit and get their desired result. (For instance, by running a .jl script instead of a bash script that calls Julia or something.)
I think the problem is that there needs to be a way to store the compiled code to disk and reimport it, which I think there are some plans toward that path. There were some developments discussed [here](https://www.youtube.com/watch?v=GnsONc9DYg0) to help with this, but the holy grail of fully compiled function caching across Julia kernels is not there yet.
PHP took over 10 years between 5 and 7. A bit push to upgrade from 5.6 was around 2017/2018 when support would stop. It was a mess because most developers simply didn't care or didn't know enough to handle this.
Now they're doing yearly releases (7.2, 7.3, 7.4) but also PHP 8 was released already (along with 8.1 and 8.2).
So what would an interface look like? Best I can figure out an interface is equivalent to a requirement that a couple of methods are implemented and that a bunch of fields exist, which sort of gets taken care of by the duck-typing, the only problem is that it's not documented in code.
One approach that I'd been thinking about would be defining a method IMyInterface that returns a tuple of fields and methods defined by the interface and saying a type implements that interface if there's an implementation. Technically this performs the same job, but unfortunately it doesn't seem like it'd be easy to use without hefty syntactic sugar.
What do you imagine interfaces would look like in Julia?
Technically "all" you need is to extend the 'where' keyword a "little":
Iterable{T,S} = TIter where {
iterate(iter::TIter) <: Tuple{T,S},
iterate(iter::TIter, state::S) <: Tuple{T,S}}interface{S,T} Iterable{T} Next = Tuple{Union{T,Nothing},S} iterate(self::Self)::Next iterate(self::Self, state::S)::Next end
This means "Iterable has two type parameters, one of which (T) is named explicitly in the type and the other of which (S) is implicitly used in the definition of the interface but is otherwise not nameable.".
I think the real question is how to implement structural sub typing of interfaces -- you wouldn't want to have to list the interfaces a struct implements when defining it because then you're limited to only those traits you (the struct's author) know about. Better (and more Julian) to automatically implement Iterable for any struct with those two iterate methods defined on it, but it seems like a pretty big task to keep track of that at runtime. Especially if you care about matching the return types of the interface's functions since in general those can't be type checked until they're actually called (or at the least compiled, but the whole point of Julia is to not compile the world ahead of time).
In theory an interface ought to just be a shorthand for 'this duck-typing works', but in a way that ideally can be verified statically.
A static type checker is the only possible (though major) advantage I see. It's debatable if it even makes sense for Julia to check interfaces at runtime, if something doesn't satisfy an interface but the code compiles, then why would it be wrong?
Okay cool, that does sound sensible. But now let's take a case where A{T <: AbstractFloat} defines the interface by doing direct dispatches for every floating point type in Julia Base. It won't error because iterate(::A{Float32}), Float64, etc. are all defined. Now someone adds the package ArbFloats.jl. Do you error when `using ArbFloats` because the interface is now not satisfied for `A{ArbFloat64}`? That would be annoying for users who never use type A, but if you don't error then your interfaces are not guaranteed to work. This would mean interfaces should only ever be used with `A{T <: T2}` where all interface definitions are done with `iterate(A{T}) where {T <: T2}` in the generic form so that it automatically extends well. That would be good code, but does that mean you should enforce that `iterate(::A{<:AbstractFloat})` is defined and error if it's not, even if every `T <: AbstractFloat` case is handled in the current REPL?
There's a lot to think about. But no matter how that's done, I cannot see it not breaking some code somewhere, hence probably being a 2.0.
I would also like to see a better test suite, with the ability to run a single test if I want to. Test Driven Development on a large project is a nightmare (although it's gotten a little better recently). I still can't believe how people contribute to Base Julia, because running the tests takes forever.
Lastly, better support for debugging in Neovim, Vim, Emacs etc. Better linting tools too. There's a push in other languages to write core language tooling in Rust or Zig, I think Julia would benefit from that approach (there's `juliaup` but I'd like to see more official tools similar to this).
Personally I'd like to see SetField.jl or Accessors.jl in Base. I'd also like to see improvements in the ergonomics of using immutable structs. In Julia, users should want to use immutable structs all the time, and ONLY if they need reference semantics they should want to reach for mutable structs. I think the naming and the sematics here is REALLLYY confusing when trying to get a team to write performant code. I can't tell you how many times I've had this kind of conversation with various members in my team.
Me: Create a struct for holding all that related data together
F: Okay, let me create a struct. - Refactors code base for a day - Realizes the data is being updated - Uses mutable structs
Me: Don't use mutable structs unless you want reference semantics
F: What even is reference semantics? - Tries to refactor by removing the `mutable` keyword, code breaks when doing `a.b += 1`, writes messy code like `a = A(B(a.b + 1, a.b.c, a.b.d, ...), a.c, a.d, ...)`.
Me: Ugh, why didn't you use `SetField` instead?
F: No unnecessary extra dependencies to reduce complexity.
Project Manager: Where are we with this task? Just use mutable struct and move on.
Me: Sigh. Fine.
On a non-language side of things,
1) it would be nice to have a bigger focus on teaching how to write performant Julia code. Currently, most if not all the material is geared toward someone who already knows how to write performance code (in C++ for example) and showing them the "Julian" way to do it.
2) I'd like to see a usability survey done from Julia users to see what features should be tackled next.
I agree that one of these should be standardized to be a Base functionality. But that won't need a v2.0 for that to happen: it can happen in a v1.x.
> it would be nice to have a bigger focus on teaching how to write performant Julia code. Currently, most if not all the material is geared toward someone who already knows how to write performance code (in C++ for example) and showing them the "Julian" way to do it.
The issue is that many of these tricks are actively being addressed, for example with escape analysis to reduce allocations. So what really needs to be taught is how to profile and understand what the code is doing, and how to effect it to do what one wants for performance. That then increases the barrier to entry. I think we shouldn't be teaching people "how to write performant Julia code" much more than is already out there though: since most of the tricks are really just workarounds for missing optimizations, those compiler optimizations should just get added (and there's some work showing it is possible).