kcsrk

I am not sure I understand the suggestion that preemptive scheduling is important with the explosion of CPU core count. You may need preemptive scheduling even on a single core if your application expects some fairness of scheduling guarantees. Am I missing something?

We can do preemptive scheduling of user-level threads by registering a timer interrupt with the OS and preempting the currently running user-level thread in favor of another one. This is up to the individual libraries to implement. These issues are discussed in [1] Sec 4.3.

(I work on Multicore OCaml)

Multicore OCaml separates the notion of concurrency (overlapped execution) and parallelism (simultaneous execution). You can have one without the other but also have both. The main principle that we're going for is that the compiler should expose just enough support for concurrency and parallelism such that interesting features can live outside the compiler as libraries. This permits the library evolution not to be tied with the language evolution but also permits diversity of libraries beyond what the language developers can aim to achieve. That said, we will build and maintain a few core libraries for concurrency and parallelism.

For concurrency, Multicore OCaml exposes effect handlers [1,2,3] which allows you to build generators, async/await, coroutines, etc as library functions (as opposed to primitives natively supported by the compiler). For example, have a look at aeio [4] which permits the construction of scalable asynchronous I/O in direct-style (no callbacks). aeio is also discussed in [1]. We also plan to extend Lwt to support parallelism.

For parallelism, the compiler exposes Domains, which are a wrapper over OS-threads [5]. Domains expose a low-level API that is not suitable for end user consumption. On top of domains, we support channels with bounded buffers [6] (similar to channels in Go, Mvars in GHC Haskell) as well as software transactional memory (STM) that supports shared memory and message passing concurrency and which can take advantage of hardware transactional memory [7].

[1] Concurrent System Programming with Effect Handlers: http://kcsrk.info/papers/system_effects_feb_18.pdf

[2] CUFP effects tutorial: https://github.com/ocamllabs/ocaml-effects-tutorial

[3] Effects Examples: https://github.com/kayceesrk/effects-examples

[4] Aeio: Asynchronous Effect-based IO https://github.com/kayceesrk/ocaml-aeio

[5] Domains: https://github.com/ocaml-multicore/ocaml-multicore/blob/para...

[6] domainslib: https://github.com/ocaml-multicore/domainslib

[7] Reagents: https://github.com/ocaml-multicore/reagents

Of interest is Crowbar[0,1], a testing tool for OCaml that combines Quickcheck like property testing with AFL fuzz.

[0] https://github.com/stedolan/crowbar

[1] https://ocaml.org/meetings/ocaml/2017/extended-abstract__201...

Secure applications are pervasive. Think point of sale terminals, IoT devices and wearables handling personal information, all fly by wire systems, etc. Shakti is a whole family of processors from tiny microcontrollers to server class processors.

We're working on it https://github.com/ocamllabs/ocaml-multicore

We're working on it. You can install and play around with Multicore compiler today though the wider library ecosystem is yet to catch up. https://github.com/ocamllabs/ocaml-multicore

The author of the talk here. I am excited about the Multicore OCaml upstreaming plan.

We're going to phase it into 3 distinct phases. First, we will upstream the multicore runtime so that it co-exists with the current runtime as a non-default option. This will give a chance to test and tune the new GC in the wild. Once we are happy with it, it will be made the default.

The second PR would be support for fibers -- linear delimited continuations without the syntax extensions (exposed as primitives in the Obj module). This will help us test the runtime support for fibers and prototype programs.

The last PR would be effect handlers with an effect system. Any unhandled user-defined effects would be caught statically by the type system. The effect system is particularly useful not just for writing correct programs, but to target JavaScript using the js_of_ocaml compiler where we can selectively CPS translate the effectful bits à la Koka. As a result, we will retain the performance of direct-style code and pay extra cost only for the code that uses effect handlers. In the future, we will extend it to track built-in OCaml effects such as IO and mutable refs. At that point, we can statically distinguish pure (as Haskell defines it) functions from impure ones without having to use Monad transformers for the code that mixes multiple effects. I'd recommend Leo White's talk [0] on effect handlers for a preview of how this system would look like.

[0] https://www.janestreet.com/tech-talks/effective-programming/

There is already one! See Malfunction [0,1]. There is an Idris and Agda backend for malfunction.

[0]: https://github.com/stedolan/malfunction

[1]: https://www.cl.cam.ac.uk/~sd601/papers/malfunction.pdf

With algebraic effect handlers (the concurrency story of multicore OCaml), you can write your I/O code in direct-style, but retain the advantage of non-blocking I/O. As such, you have the advantage of goroutines, but this is also an opt-in. The asynchronous I/O is implemented as a library and is not baked into the language, and hence you can write your own I/O library. For more details, see our recent draft paper: http://kcsrk.info/papers/system_effects_may_17.pdf.

There has been developments along some of these issues:

- jbuilder (https://github.com/janestreet/jbuilder) has markedly improved the build system story. Importantly, the community understands that this is an issue and is actively working towards making the situation better.

- Type declaration at the code level is optional. You could always annotate your code with types, and the type checker will ensure that the annotated types are compatible with the inferred ones. If you would like to know "what type is this 'e' variable", there is merlin (https://github.com/ocaml/merlin), which integrates with popular editors.

- Reg. typeclasses, there has been enormous amount of work put into modular implicits (http://ocamllabs.io/doc/implicits.html). It is likely to be the next major features in OCaml along with multicore.

Multicore OCaml comes with native support for concurrency through algebraic effect handlers which generalize common control flow abstractions such as exceptions, async/await, generators, non-determinism, backtracking. All of these mechanisms can be implemented directly in OCaml. [0] introduces the model, [1,2] has examples, and for further reading see [3].

[0]: http://kcsrk.info/ocaml/multicore/2015/05/20/effects-multico...

[1]: http://kcsrk.info/ocaml/multicore/effects/2015/05/27/more-ef...

[2]: https://github.com/kayceesrk/effects-examples

[3]: See recent pubs here http://kcsrk.info/.

Performance backwards compatibility has been a key goal in the Multicore OCaml project. Currently, the overhead for running legacy sequential OCaml programs on the multicore GC is a few percentage points on average. The overhead is low enough that we may not need to provide a "sequential-only" compilation flag.

Multicore support isn't for every one, but there are some important use cases where multicore support is very useful. One example is Facebook's Hack language type checker [0] which offers good opportunity for parallelism, but the lack of multicore support in OCaml means that they had to implement a custom off-heap lock free hash table shared by multiple processes. There are many industrial uses of OCaml including static type checkers (FB's Hack, Flow, Infer, etc), program verification tools (Coq, why3), math libraries (Owl) [1] etc., all of which benefit from multicore support.

[0]: https://www.youtube.com/watch?v=uXuYVUdFY48

[1]: https://news.ycombinator.com/item?id=14751236

(I work on multicore OCaml) Multicore OCaml has been steadily under development, with the latest updates here: https://ocaml.io/w/Multicore. While I can't promise when multicore lands in trunk, a stable-alpha version installable through opam should be ready in the next few weeks.

(author here) The collected CAS operations are batched together using a lock-free k-CAS implementation. The library does not presuppose a hardware k-CAS mechanism.

The built up transactions carry around some information internally (blocking, # of CASes, etc). You could possibly extend the API to query them. But it is not clear to me immediately why you'd want to expose all of this info in the types.

There is the small standard library that ships with the compiler. And then there are more extensive ones such as Batteries and Core. It is easy to generate docs in Reason syntax from OCaml projects using `redoc` tool. Here is Reason docs for:

- OCaml compiler's stdlib: http://kcsrk.info/reason_stdlib_docs/index.html

- Batteries: http://kcsrk.info/reason_batteries_docs/index.html

How about building functional operating systems and unikernels: https://mirage.io/. Since Reason is fully compatible with OCaml, you can build your own Reason unikernels on top of MirageOS libraries. Here is a list of mirage OS pioneer projects: https://github.com/mirage/mirage-www/wiki/Pioneer-Projects

Multicore support is progressing nicely. Only earlier today I made a PR for adding nativecode compilation for multicore: https://github.com/ocamllabs/ocaml-multicore/pull/47. A lot more information about the multicore OCaml project can be found here: https://ocaml.io/w/Multicore

Here is some context: http://amirchaudhry.com/bitcoin-pinata/