Hacker News Reader

sbineton ROOT: analyzing petabytes of data scientifically

back in the days, one always had to have 2 terminals open to work with ROOT: one to work and the other to 'kill -9 root.exe' thanks to CINT happily completely destroying your TTY.

sbineton ROOT: analyzing petabytes of data scientifically

that's odd. TEfficiency is a relatively simple thing to read/write :

- https://github.com/go-hep/hep/blob/main/groot/rhist/efficien...

sbineton ROOT: analyzing petabytes of data scientifically

IMHO, ROOT[3-5] is too many things with a lot of poorly designed API and most importantly a lack of separation between ROOT-the-library and ROOT-the-program (lots of globals and assumptions that ROOT-the-program is how people should use it). ROOT 6 started to correct some of these things, but it takes time (and IMHO, they are buying too much into llvm and clang, increasing even more the build times and worsening the hackability of ROOT as a project)

Also, for the longest time, the I/O format wasn't very well documented, with only 1 implementation.

Now, thanks to groot [1], uproot (that was developed building on the work from groot) and others (freehep, openscientist, ...), it's to read/write ROOT data w/o bringing the whole TWorld. Interoperability. For data, I'd say it's very much paramount in my book to have some hope to be able to read back that unique data in 20, 30, ... years down the line.

[1] https://go-hep.org/x/hep/groot (I am the main dev behind go-hep)

sbineton Go Modules in 2019

there are actually a couple of real interpreters for Go:

- neugram.io

- https://github.com/cosmos72/gomacro

here is a mybinder that used to use the former, and now uses the latter:

- https://mybinder.org/v2/gh/go-hep/binder/master

sbineton Go Modules in 2019

> These are things that are great for product development and devops and not in fact all that valuable in scientific computing

I disagree. Again, this may very well be science-domain dependent, but in High Energy Physics (where, finally, Python is recognized as a mainstream language, BTW) many -- if not all -- of the pain points that slow down undergrads, PhDs, post-docs and researchers at large, are these Go features.

yes, the time from "idea" to "little script that shows a bunch of plots" on a subset of the overall data is definitely shorter in Python (exploratory analysis is really really great in Python). but, at least for LHC analyses, python doesn't cut it when it comes to Tb of data to swift through, distribution over the grid/cloud, ... you die by a thousand cuts. and that's when you are alone on one little script. LHC analyses can see well over 10-20 people assembling a bunch of modules and scripts. You really long for a more robust to (automated) refactoring language than python, pretty quickly.

sbineton Go Modules in 2019

wrt n-dim arrays: yes, it's a pain point. we (Gonum devs) tried to push for an "n-dim slice" type a couple of years ago. it didn't pan out.

generics may help in that department.

in the meantime, there's Apache Arrow:

- https://blog.gopheracademy.com/advent-2018/go-arrow/

sbineton Go Modules in 2019

this is being worked on:

- https://github.com/golang/tools/blob/master/cmd/golsp/main.g...

sbineton Go Modules in 2019

I guess it's scientific domain dependent. I've very rarely needed something else than float/double in my C++ days or REAL*8 in my F77 ones.

sure, when you need it, you need it. but float64 caters for a good 99% of my usual work day.

From a user POV, seamless installation of packages is a great boon. From a grid/cloud operator POV, static binaries are great too.

sbineton Go Modules in 2019

others have already mentioned Gonum:

- https://gonum.org

Gonum is almost on par with _e.g._ NumPy/SciPy (most notably lacking: ODEs). So still ways to Go but it's getting there.

Go-HEP is my attempt to bring a few High Energy Physics oriented packages to particle physicists:

- https://go-hep.org

I've also written a few words on why I think Go is great for science:

- https://sbinet.github.io/posts/2018-07-31-go-hep-manifesto/

TL;DR: Go is great b/c it brings great s/w engineering practices and a s/w engineering-friendly environment to scientists.

Admittedly, generics will change how packages are written. So some code churn will take place when/if they land, but the Go community learned the lessons from Python2/3 and Perl5/6. Expect a better migration path.

Lastly, I guess the 2 remaining weak points of Go are:

- runtime performances sub-par wrt C++ or Rust

- GUIs (which may or may not fall into "interactive visualization")

That said, the Go community worked on a Go kernel for Jupyter:

- https://github.com/gopherdata/gophernotes

- http://gopherdata.io/

hth, -s

sbineton Show HN: TinyGo – LLVM-based Go compiler for microcontrollers

Next step is to directly generate IR with (link: https://github.com/llir/llvm) github.com/llir/llvm. And then rewrite all of llvm in Go :)

sbineton Show HN: Life – A secure, blazing-fast, cross-platform WebAssembly VM in Go

- the VM is written in Go and can execute wasm binaries

- the next version of the Go toolchain (Go-1.11) will have a wasm target

sbineton Go gets preliminary WebAssembly support

there's actually (the beginning of) an interpreter in Go, for Go, that consummes wasm :)

https://github.com/go-interpreter/wagon

sbineton Gonum – Numerical Computing for Go

as I am working on https://github.com/go-interpreter/wagon, I'd be very interested in these issues you're talking about.

sbineton Gonum – Numerical Computing for Go

> But Go doesn't have a JIT, and its performance is good enough that I don't expect one to emerge any time soon.

that's true. that is... until a (real) Go interpreter shows up. something that's bound to happen when Go will be used for (data) exploratory work.

sbineton Gonum – Numerical Computing for Go

I have been using Gonum for some time now (also contributed, mostly in the plotting area).

Last summer, I tried an experiment: have a student migrate a little python-based analysis to a Go-based one. The analysis was fitting some cosmological constants out of the so called Hubble diagram.

I was pleased to see that, in the span of 2-3 months, the student who had limited knowledge in programming (a bit of python), managed to pull off the minimization of a 740 supernovae dataset with a 2220x2220 nuisance parameters matrix.

and the run time was 2x faster than the python one (with scipy/minuit for the minimization, so everything in C/C++, really).

success. :)

(and this motivated us to completely switch to Go as a teaching language for our master in particle physics / cosmology.)

sbineton Gonum – Numerical Computing for Go

also, there's been work to make for-loops preemptible:

- https://github.com/golang/go/issues/10958

- https://go-review.googlesource.com/c/go/+/33910

- https://go-review.googlesource.com/c/go/+/36206

- https://go-review.googlesource.com/c/go/+/46410

- https://go-review.googlesource.com/c/go/+/43050

sbineton What can Rust do for astrophysics?

and for those doing the same but in Go:

https://github.com/astrogo/fitsio

(a pure Go version)

sbineton Gorgonia: a library like Theano or TensorFlow, mainly written in Go

out of interest, what do you find lacking in https://github.com/gonum offering as a scientific stack?

sbineton Gorgonia: a library like Theano or TensorFlow, mainly written in Go

I was challenging your "Go is not set up to have a REPL" assertion. we are in violent agreement about the (non) "REPL-ness" of _gore_ and the fact that "just keep compiling a module incrementally" is a dead end.

I don't see how reimplementing Go in Go but with a REPL (on top of, let's say, a SSA interpreter) is a Go-like REPL at best and not just a Go REPL. Could you expand a bit?

AFAICT, once one has a Go interpreter (the x/go/ssa is basically just that), implementing a REPL is "just" providing a program counter to the interactive interpreter layer, together with the ability to modify that PC... (many details have been dropped on the floor, of course)

sbineton Gorgonia: a library like Theano or TensorFlow, mainly written in Go

I beg to differ. See: https://github.com/sbinet/igo

It is a true REPL (not like "gore", mentioned somewhere in that thread). It doesn't implement the whole Go specs, but there's AFAICT nothing in the Go specs that prevent `igo` to implement the whole Go specs. It's "just" work. Moreover, with the introduction of the "buildmode=plugin" in the yet-to-be-released Go-1.8, import-ing packages on the fly should be easy implementable.

("my" new vehicle to provide a true REPL for Go is at https://github.com/go-interpreter)

sbineton Gorgonia: a library like Theano or TensorFlow, mainly written in Go

you can even use it on the web: https://github.com/gopherds/mybinder-go

sbineton Gorgonia: a library like Theano or TensorFlow, mainly written in Go

https://github.com/go-python/gopy

it automatically creates a CPython-2 extension module out of a Go-1.5 package.

I plan to update it for Go>=1.6 and also directly generate a "cffi" python module so CPython-{2,3} and PyPy can be directly supported out of the box.

sbineton Julia: A Fast Language for Numerical Computing

generating structs at runtime might land for go-1.7: https://go-review.googlesource.com/#/c/9251/4

sbineton Proposal for a Go Interpreter

state. as I said elsewhere in this HN thread, many of the first REPLs that appeared when Go was first released used this rather effective scheme of recompiling on the fly a single main.go file after each new input the user entered and re-running the whole thing.

nothing's wrong with that except that you accumulate side-effects:

  gorepl> f := func() { launchMissiles(); fmt.Printf(">>> missiles launched\n") }
  gorepl> f()
  >>> missiles launched
  gorepl> var a = 2
  >>> missiles launched
  gorepl> fmt.Printf("oops!\n")
  >>> missiles launched
  oops!

that's not the kind of REPL I want, and that's not what "my" REPL does (https://github.com/sbinet/igo) nor the one built on top of LLVM (llgoi: http://llvm.org/viewvc/llvm-project/llgo/trunk/cmd/llgoi/)

sbineton Proposal for a Go Interpreter

that's my personal bias. GoRe's REPL recompiles and re-runs the resulting binary after each line entered, accumulating side effects:

  $ gore
  gore version 0.2.5  :help for help
  gore> var i = 41
  gore> f := func() { i++; fmt.Printf("universe=%d\n", i) }
  (func())(0x4012b0)
  gore> f()
  universe=42
  gore> f()
  universe=42
  universe=43

when Go was released back in 2009 there has been a string of such REPLs, which don't match "my" definition of a REPL, at least, not the one I want to be able to use for exploratory work (the kind you get from a python or IPython prompt)

again, that's my personal bias. that said, GoRe is a nice project (gocode-completion, especially, is nice)

sbineton Proposal for a Go Interpreter

ah, didn't know about CAAS. of course LLVM/CLang was also on our radar. we are still pondering on whether we should target LLVM bitcode so we could easily build upon LLVM toolchain (and cross-pollinate with other LLVM-based interpreters. e.g. seamlessly interoperate with IJulia and CLing interpreters, sharing code and values at runtime thru LLVM.) to me, the main issue with LLVM is that it's a real pain to develop against, I know this for a fact having worked a little bit with CLing, the CLang-based C++ interpreter (the edit-compile-test cycle is that of a set of large C++ libraries, gophers tend to be spoiled in that department)

tangential: actually, you can refactor with the CLang libraries: http://llvm.org/releases/3.7.0/tools/clang/docs/LibTooling.h... (well, to some extent)

sbineton Proposal for a Go Interpreter

interesting, it's true having a toolbox for creating go interpreters would be much more powerful.

I am not sure I completely get what a compiler-as-a-service really is. could you expand please?

(wrt JITing, yes, we considered it and it's on our radar. we know some version of it worked at some point, at least at the proof-of-concept level: https://github.com/nelhage/gojit)

thanks for your input!

sbineton Proposal for a Go Interpreter

actually, somebody already wrote a kernel for (at the time) IPython, using "my" github.com/sbinet/go-eval go interpreter: https://github.com/takluyver/igo

my plan for this new go interpreter proposal, is to be able to use for scientific exploratory work (like we do with IPython right now when in a python-based ecosystem or CLing when in C++ land).