Loom: Project Loom Early-Access Builds
jdk.java.net
jdk.java.net
As someone that codes Java (using various async libraries) and has dabbled in Go, I really look forward to this, it can't come soon enough.
Tiny link on the right called "State of Loom" seems to have the most background information though, and should be helpful to people like me who had no idea what Loom is/supposed to be:
http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...
>... delimited continuations (of some form), and related features, such as explicit tail-call.
I expect internal implementation of continuations in Loom to have non-negligible overhead which may be justified for heavy blocking I/O operations but not for CPU-bound workloads. One benchmark I'd like to see (I may want to create it myself at some point) would run a large number of light threads (fibers) which would exchange data via blocking queues (to mimic goroutines with channels). No I/O, a totally CPU-bound workload with a lot of concurrency and a good potential for scalability. Somehow, I'm not very optimistic about Loom's performance in that context (please tell me if and where I'm wrong).
2. For CPU-bound workloads, virtually any overhead that isn't zero can come to dominate in many situations. The overhead can be made to be zero when the tasks are known in advance and can be inlined. This happens in two situations: generators, where there is one producer that can be inlined together with the consumer, and parallel workloads where all the tasks are the same. I am not aware of any single implementation, or even a single user-facing construct, that gives you the optimal experience both in those situations and in I/O-driven concurrency. If you have only one continuation construct, you must choose whether you want to favour one or the other. C++ favours the former, while Java, like Go, the latter, partly because Java already has a decent parallelism construct, and partly because we believe that the concurrent use-case delivers more value to more people. If parallel use-cases that benefit from continuations and/or generators come in high demand, we can address them later.
Then again, the same would be true in Go. Loom simply brings goroutines to the JVM.
Go might perform better, but that can be due to many other things not specific to the concurrency implementation (like better memory utilization because not every struct is stored on the heap in Go)
That is the promise, yes, but you specifically mentioned a CPU-bound task. A CPU-bound workload cannot hope to do much better than doing a concurrent task spread equally among the available CPU's, spawning more threads will just add coordination overhead.
If you're spawning one million threads, each of which will perform a blocking operation, then you're likely not CPU bound (unless you're doing a System.sleep(1) or the equivalent).
> I just don't believe their scheduler is so good.
I believe you pick your own scheduler. By default it uses ForkJoin, which is supposed to be pretty good, uses the same conceptual implementation as Go (work stealing), but you're free to use something else.
Imagine a DAG with million nodes. Each node takes data from all its input edges, processes it and sends to all its output edges. Now that I have light threads I want to use them to implement all nodes. Edges are naturally implemented as blocking queues. If I feed enough data into this structure I get a CPU-bound workload with a lot of concurrency (and scalability). Yet the nodes will have to block on their input queues because data will not be ready in most cases.
Now, to complicate the design, I want to have I/O tasks that read data from a file/network and feed it into the DAG and send its resulting output to the UI thread. Should I use different "threads" for different parts of the design? I'd hate to, especially given the fact that all those parts are dynamic in nature.
So, of course, I'm not currently spawning a million threads to execute all those tasks but Loom says I could do that for all my tasks and my concern is that to get decent performance I'd have to explicitly separate my "CPU-bound" tasks from my "I/O-bound" tasks and use different kinds of threads for each (which I'm kind of doing right now but again the promise was...).
As long as you do IO or other blocking operations which block for a significant amount of time (ms instead of ns), Loom will be beneficial. Otherwise, you'd do better with regular threads in a pool, but even then I'd imagine the performance difference to be tiny.
Valhalla was first announced in 2014, and Loom in 2017.
Does anyone have some insight on when either of these might finally be ready?
For those not familiar:
Valhalla: introduces "inline types" (structs) and allows using primitive types as generics. Has the potential for quite significant performance improvements because it can reduces all the pointer chasing currently required.
Loom: allows Java to work like Go: code that looks single-threaded but where IO is actually driven by a runtime on a hand full of threads. As in: async without the development and debugging complexity.
So I am not concerned that engineering efforts that allow .jar files from 2000 to keep running in Valhalla/Loom aware JVMs (minus modules issues) to also take equally amount of time.
Generally I think that programs can be structured nicely and have them perform optimally by using traditional threads, backed by idiomatic usage of the java.util.concurrent package and integration of those vanilla threads with the NIO package.
IOW, using NIO doesn't inevitably imply using an async framework (be it bespoke, or official like Loom)?
Personally I've always had my reservations around async, it seems a paradigm that makes things gratuitously harder.
I think that's too high a bar. I think the right question should be "will Loom make it easier to write well performing and maintainable solutions to real problems?" I think by making threads a much cheaper resource the answer will be yes.
For example, the simplest way to write a network server that can handle more than one client at a time is to fork a new thread for every client. Using that model you quickly run into a wall with OS threads in Java, and if you want more performance need to look into using NIO and event loops. With virtual threads you'll be able to go much further with the thread-per-client model.
This seems much debatable. For me the simplest way is creating a thread-per-core pool, and have clients integrate with said pool via a queue. It's a well-understood paradigm that has worked for a long time.
I get the sense of abstraction of having a "thread" per client, but is it really the simplest choice if it needed creating a multi-year project like Loom first?
A precedent that comes to mind is Clojure core.async. It pursues the same sense of abstraction. But does one gain much if the implementation has its own quirks, flaws, etc? Any given abstraction is one bug away from leaking.
So, it's well plausible that the same can happen with Loom - more stuff to learn, more underlying complexity, bigger surface area for bugs.
It all seems too gratuitous too me, so my high-bar question/questioning remains :)
You can see a need for this at least in the Android space: RxJava, Kotlin coroutines, etc.
Also, in a user-managed thread multiplexing system you can't guarantee "fair" scheduling. If you have some DOS bugs in your server you can lock up an entire serving thread. A language-level multiplexing system will not have these same kinds of edge cases.
There are many places this sort of thing becomes helpful. We could also see concurrent processing utilities that could be very helpful. See this for more info on how this could lead to cleaner programs: https://www.youtube.com/watch?v=f6kdp27TYZs
Loom is not a panacea and doesn’t make any new thing possible, it makes certain things easier. And it still will have its limitations.
Let's say we want a concurrency of 1000 and have 8 CPUs available. I'd have small thread pools, one per logical kind of task. Maybe this would result in 20 threads total for the whole JVM.
Concurrency would be represented by the items being held in BlockingQueues of max size 1000. Items would be transferred from thread to thread by communicating via said queues (pipeline pattern).
CPU utilization would be practically ideal as there's always a thread having some kind of work at hand.
And the system would be actually concurrent since the queues can have interleaved workloads (e.g. work for users #1, #2, #1 again, #3, etc).
The max queue size allows to communicate backpressure to e.g. a load balancer.
Particularly, the queue items will be picked up in a nondeterministic order, driven by CPU utilization.
> Is the idea here that you’ve reimplemented scheduling on top of the jvm?
No, I'm implementing a fairly vanilla pipeline pattern, just with simpler, more explicit blocks than those used by async frameworks/abstractions. IIRC it's described well in the JCIP book.
To get reasonably many requests started and waiting on those responses, either you need to write in async/callback oriented style, or you need a mechanism to make many thousands of naive, blocking routines coexist efficiently.
The situation is quite simple, and governed by Little's law. If your number of threads is greater than the average request rate times the average processing latency (of processing a request entirely on a single thread), then you're all good and it's a well-understood paradigm that's worked for many years. If, on the other hand, that number of threads is smaller than that multiple, then that well-understood paradigm is well-understood to have crashed servers for many years, requiring many more of them even when their utilisation is low. If you're on one side of this, then you don't have a problem to begin with. Loom exists because increasingly more people are on the other side of this problem, where they need multiple servers.
Loom comes from the same place of skepticism of async that you are expressing. Its goal is to make the imperative, blocking style as efficient as the async style while retaining familiar debugging tooling.
Personally, I'm more skeptical of imperative, blocking programming but Loom does seem like it can only be a good thing to have access to on the JVM platform.
If you're running on Linux, this is rather less the case. People see 1MB Xss values and assume that 1MB of 'real' memory is required to spawn a thread. In reality, on a standard linux system with overcommit enabled, stack pages are only consumed as they're written to - the rest is just virtual.
Possibly you're talking about something else, but I commonly see this raised as a concern about creating threads in Java.
1. Once stack memory is committed, it is never uncommitted, and creating a new thread is relatively expensive.
2. Committing virtual memory is done at a page granularity.
These make threads far too costly to serve as a construct for any single concurrent task.
Look, I'm looking forward to Loom as much as anyone. It is clearly going to be a big leap forward to the java ecosystem, and I'm looking forward to some threaded model being appropriate for any job rather than just most of them. It's just I see this terror of creating a thread or two out in the wider world and it seems a touch overwrought to me.
I don't think this is true if you create have approximately 1 thread for each CPU core (which is the ideal scenario in a non-async app - why would you create more threads than CPUs?).
A JVM thread is basically an OS thread. So on a 64-core server you'd have 64 threads, give or take. While a server can handle a few thousands threads quite seamlessly.
As far as using a fixed pool goes: others have replied similarly but by doing that you are completely letting your CPU cores go to waste whenever you are waiting for any underlying IO. For many classes of applications, they will end up topping out at only a few percent CPU usage.
For workloads that involves a lot of IO. If you perform much more than 2-3 IO ops per request, a lot of time spent per request is simply waiting for IO response, meaning your CPUs are mostly idle.
One can allocate thread pools sensibly relative to CPU count and achieve ideal task<->CPU allocation.
So a modern choice would be either to have specialized background job processes, or microservices. Each such process would have manageable thread pools and a good CPU fit.
It also is a non-trivial fact that servers can perfectly have 128 cores or such, just like TB-sized memory isn't unheard of for JVMs. So that's another option for "less modern" apps: go the thread pool route, increase CPU count.
That makes the context switching problem worse. If you're running 100 services with 8 threads each on an 8 core CPU, that's got the same overhead problem as running 800 threads in a single service - potentially worse if the kernel has to do a more thorough context switch (e.g. TLB flush for spectre mitigation) when switching between separate processes. (And if you're going to solve that problem by running each service in its own container or VM then again that only makes the context switches heavier - fundamentally if you're running 800 threads on 8 physical cores then you've got to switch between them somehow).
> It also is a non-trivial fact that servers can perfectly have 128 cores or such, just like TB-sized memory isn't unheard of for JVMs. So that's another option for "less modern" apps: go the thread pool route, increase CPU count.
Well no shit if you just buy more CPU cores whenever you want to run more code then you don't need to worry about getting decent performance out of your hardware. But most of us are in businesses that can't afford one physical CPU core per IO-performing function in your codebase, which is what you'd need to sustain that approach (not to mention perfectly predicting how much load every single function needs). I guess when you add the 129th function to your codebase you then throw away all your super-expensive servers and buy even more expensive 256-core replacements? Or you forcibly split each service at the 128-function boundary so that you can deploy it as two microservices on two machines?
Quite obviously I meant running a microservice / job processor in a different "node" (aws instance, whatever).
It barely makes sense to let microservices compete with each other for CPU - one precisely seeks greater isolation.
> Well no shit if you just buy more CPU cores whenever you want to run more code then you don't need to worry about getting decent performance out of your hardware.
Large codebases relate strongly to large businesses that can afford hardware matching their scale.
> I guess when you add the 129th function to your codebase you then throw away all your super-expensive servers and buy even more expensive 256-core replacements?
Ok I regret spending time over your replies.
That's still just sweeping the problem under the carpet. Either you're getting a physical core for each function, in which case you're paying for lots of cores. Or you're multiplexing functions onto cores at some level (whether that's your own code, shared hosting, AWS lambda, or whatever) in which case someone (you or your service provider) is doing a bunch of context switches that have to be paid for somehow.
> It barely makes sense to let microservices compete with each other for CPU - one precisely seeks greater isolation.
For services big enough to need a full CPU (or a full core) all the time, you may as well give them a full CPU. But that implies a much coarser level of granularity than you've been suggesting. If your services really are "micro" then it makes a huge amount of sense to share a CPU between multiple services because most of them don't need a whole CPU the whole time.
> Large codebases relate strongly to large businesses that can afford hardware matching their scale.
The code grows faster than the hardware use. I've worked on codebases that were deployed on thousands of machines - but they had millions of lines of code and tens if not hundreds of thousands of functions. One of them even did take the approach of running multiple steps of the same logical computation on separate machines - but this was still done by having a yield point in the code and a scheduling framework that dispatched tasks onto grid compute machines, both because even though the code gets compiled into a state machine representation (i.e. one-way flows, de facto unstructured programming - the same as your pipeline approach) you don't want to have to maintain it in that form, and even if you did have code that looks like that, manually deciding how many cores to allocate to processing each state would not have been tractable.
> Ok I regret spending time over your replies.
Look, I honestly thought you were doing some kind of elaborate troll. What on earth kind of environment are you working in where you have more cores than code?
Right, so then how do you structure your program - particularly an I/O-bound program - to use that small pool of threads effectively?
Imagine you're building a REST frontend that queries a couple of datastores and aggregates the results - you get a web request, fire off a request to some coordinator datastore, get a response back from that, fire off a bunch of requests to other datastores based on that, then form the results into some JSON and send them back to the client. If you do blocking requests then you waste most of your threads most of the time (they'll be idle waiting for responses from the datastores).
If you use NIO then that solves that problem, but you've still got to get each thread to actually do the right kind of processing - you want each thread to run an event loop where it checks for returned results from the datastores, matches those up with the requests that they belong to, and does the next step of processing for that request whether it's firing off more requests to other the datastores or composing together the results and sending them back to the original client. And you've got to also handle timing out stale requests etc.
Now you can write a server literally like that, with a global buffers for each possible request type that hold the state that you need to pick up handling that request again. But it's a nightmare to maintain, as you're effectively forced to write unstructured programs where you do a GOTO for each I/O operation - there's no easy way to trace through the handling of a single request or run part of your program for testing.
So people generally prefer to use an async framework that will handle keeping track of each suspended request and what to do next when the result comes back, and multiplex those onto the native threads for you. That could be something where you call an I/O API and pass a callback to run when the result comes back (and the framework takes care of running the callback on some thread when the result is ready, without blocking a thread in the meantime), or something where there's a special operator to say "suspend this function here, package up the rest of it as a continuation, and continue with that continuation when the I/O result comes back". Or, as Loom is doing, it could be something similar that works completely invisibly whenever the programmer calls particular functions.
There are tradeoffs to all these approaches to async, but any of them is going to perform a lot better than the naive approach of just using a native thread for each request, and they're all (hopefully) going to be more maintainable than manually writing an event loop that does the right thing each time.
I've hinted over the responses to my OP how I tend to structure the programs I design.
Other than that, arguing over specifics is somewhat besides the point.
It would be great to have a standardized benchmark/challenge so one could let the code speak for itself.
(I know C10k is a thing... not ideal, particularly because it makes concurrency a goal in itself, as opposed to a solution to a root problem)
The trouble is that threads are heavyweight, particularly in context switching. So the user has to work around it by explicitly creating async functions, task schedulers, futures and so on, to avoid having to block a crucial thread (as in GUIs or network event loops).
Languages like go and erlang help you avoid that by having the entire ecosystem (libraries included) give you permission to write blocking code, and that under the hood, they would take care of fast context switching and async I/O. That is what loom hopes to do. Ideally in this world, the ordinary developer would not have to use nio at all ... simple blocking APIs should suffice.
That said, Loom is a neat tool and I expect new, more convenient patterns will pop up from Loom.
I'll be interested to see how Java tackles OS thread problems after Loom. We'll still live in a world where most(all?) GUIs frameworks revolve around single thread concurrency and GPU access is still single threaded. I don't see how the implicit threading of loom is going to work for that, and it also seems like a line in the sand has been drawn in the face of async/await. We'll just have to see what comes of it.
Rather virtual threads get created by default and an explicit call, or thread factory can be used to get the value back.
At least that is how I remember it.
This is one of the major driving forces behind Loom being done so differently to how they’re done elsewhere.
Async became so popular due to sync code not scaling for large concurrent use cases. Loom is about leaving sync application code be, and optimising the underlying runtime instead.
The implicit threading of Loom works well for things like transparently dealing with IO without blocking an OS thread.
For things like posting back to a UI thread, you'll still need explicit handling not unlike async/await. It seems like Java is taking a scoped approach. https://wiki.openjdk.java.net/display/loom/Structured+Concur...
By doing it this way, the entire JVM ecosystem get's the benefits of async/await without changing the Java language. It's also easier for developers to learn (do what you're used to with threads, just use this .virtualThreadPool function instead and be extravagant with the number of threads in the pool).
> International use restrictions
> Due to limited intellectual property protection and enforcement in certain countries, the source code may only be distributed to an authorized list of countries. You will not be able to access the source code if you are downloading from a country that is not on this list. We are continuously reviewing this list for addition of other countries.
Maybe it's just that Oracle that doesn't want to convey the software to certain nationals, but anyone else with a valid GPL license may convey it?
Besides, they either own the code, licensed said code by a contributor or another software maker to Oracle on a perpetual, sublicensable terms or used Expat-licensed (MIT-licensed) code. All of these does not restrict Oracle from assigning any copyright conditions apart from the required attributions on Expat-licensed code. It could be closed-source and they will still be fine.
You can’t forbid yourself from applying whatever terms you want on your own code.