HNHacker News
TopNewBestAskShowJobs

tgamblin

594 karma · joined June 11, 2015

submissionscomments
tgamblin··on Ask HN: Is your company sticking to on-premise servers? Why?
Yes but we still have many OP systems on the floor (see the link above).
tgamblin··on Ask HN: Is your company sticking to on-premise servers? Why?
I work in Livermore Computing at LLNL.

We manage upwards of 30 different compute clusters (many listed here: https://hpc.llnl.gov/hardware/platforms). You can read about the machine slated to hit the floor in 2022/2023 here: https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capita....

All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. For our workloads on "commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd pay for equivalent cloud nodes, and for the really high end systems like Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life of the machine.

The way machines are procured here is different from what smaller shops might be used to. For example, the El Capitan machine mentioned above was procured via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We write a 100+ page statement of work describing what the machine must do, and we release a set of benchmarks characterizing our workload. Vendors submit proposals for how they could meet our requirements, along with performance numbers and test results for the benchmarks. Then we pick the best proposal. We do something similar with LANL and SNL for the so-called commodity clusters (see https://hpc.llnl.gov/cts-2-rfi for the latest one). As part of these processes, we learn a lot about what vendors are planning to offer 5 years out, so we're not picking off the shelf stuff -- we're getting large volumes of the latest hardware.

In addition to the cost savings from running on-prem, it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.

tgamblin··on Frontier: ORNL's 2021 exascale supercomputer will run on AMD CPUs and GPUs
On the NNSA side of DOE, the codes very intentionally do not depend on CUDA. They use RAJA (https://github.com/llnl/raja) or Kokkos (https://github.com/kokkos/kokkos) to ensure that they're not tied to a particular GPU. Using those frameworks you can easily switch between CUDA, OpenMP, or whatever in a single-source codebase.
tgamblin··on Frontier: ORNL's 2021 exascale supercomputer will run on AMD CPUs and GPUs
At LLNL the story is similar, but the framework is RAJA: https://github.com/llnl/raja. RAJA is a bit simpler than Kokkos in that it does not require you to adopt its data structures.

The Kokkos and RAJA teams are also working together on some common utility libraries now.

tgamblin··on Ask HN: Who is hiring? (April 2019)
Lawrence Livermore National Laboratory (LLNL) | Livermore, CA ONSITE | Spack Developer | ability to obtain a Q Clearance (US citizenship) required

Want to work on open source for science? Come work on the Spack package manager (https://spack.io, https://github.com/spack/spack) at LLNL!

Spack is a tool for building and installing scientific software on laptops, clusters, and the world’s largest supercomputers. It allows users to build optimized packages with many different compilers, build options, optimization flags, and dependency versions. Spack facilitates individual development workflows, but also allows supercomputing facilities to deploy large suites of software for their users.

LLNL (https://llnl.gov) is home to the world’s 2nd fastest supercomputer, Sierra (see https://www.nextplatform.com/2018/06/26/peeling-the-covers-o...), as well as ~25 other large HPC clusters. Your work will directly support simulations run on these machines, and you’ll get to collaborate with other DOE national laboratories via the Exascale Computing Project (https://exascaleproject.org). Spack has an international community, and you'll also collaborate with major supercomputing sites around the world.

We’re looking for the following skills/experience (not all required): strong Python skills, with lower-level languages (C, C++, Fortran), strong systems programming skills, building HPC and scientific libraries, build systems (CMake, autotools, make). Experience with development of any package manager is a plus.

Bonus skills: experience with large OSS projects, experience with SAT, SMT, ILP solvers or Prolog.

Email tgamblin@llnl.gov with your resume, and a bit about yourself and relevant experience.

tgamblin··on Show HN: Visalist – Find visa requirements for countries around the world
How do you handle some of the more obscure cases? For example, citizens of some countries, including the US, can do a short, visa-free stay in China if they’re transiting through the major airports (https://www.travelchinaguide.com/embassy/visa/transit.htm).

I saw Beijing this way coming back to SF from Taiwan. It was a pain to find all the details about how to apply, and a site like this could help people find out about such opportunities. e.g., you could match trip itinieraries to interesting stopovers.

tgamblin··on The U.S Just Beat China to develop the world's fastest supercomputer
In what sense? The ranking is still based on LINPACK scores, and it's intended to represent the fastest machines for scientific computing.

We can argue about whether modern simulation codes look like LINPACK (they don't) but neither Google's nor Amazon's cloud is competitive for that type of workload. Even if the clouds are physically bigger, nobody at Google or Amazon is running really big numerical jobs of the sort that DOE does. Many are running smaller scale HPC and ML jobs, but it's expensive, and for latency-bound physics simulations (most of what we run at LLNL, ORNL, etc.), cloud networks will not perform at scale.

The question of whether China's machines ever performed well for big multi-physics applications is up in the air, but they were certainly beating us on LINPACK. The Chinese set out to show that they're serious about HPC, and I think they accomplished at least that. I wouldn't be surprised if you saw this race go back and forth a few times in the coming years.

tgamblin··on GPU Fluid
There are plenty of scientific and engineering fluid dynamics codes that take advantage of GPUs. The current fastest supercomputer in the US, Titan at ORNL, uses GPUs. Here’s a CFD code that ran on it: https://www.olcf.ornl.gov/2014/06/13/ramgen-takes-turbomachi.... ORNL likes their OpenACC; here’s a GTC presentation on the code: http://on-demand.gputechconf.com/gtc/2014/presentations/S475...

LLNL and ORNL are right now rolling out two new gigantic GPU systems, Sierra and Summit: https://www.google.com/amp/s/www.nextplatform.com/2017/09/19.... There will be plenty more GPU CFD simulations of running on those systems.

tgamblin··on Why Threads are a Bad Idea (for most purposes) (1995) [pdf]
If you're talking about MPI and its use in HPC, you might've been mostly correct 5 years ago. Now, HPC applications at the largest sites are hybrid MPI+OpenMP (or some other threading model, or they use GPUs). On Xeon Phi systems like the latest Crays, you pretty much need OpenMP to take advantage of all the cores. The PPC A2 processors on the (formerly #1, now #4) Blue Gene/Q system at LLNL won't get full instruction issue per cycle unless you use multiple hardware threads (the threads are exposed to the hardware and each core will only pull instructions from a single process at a time, so this is somewhat unlike hyperthreading on Intel chips).

There can be overheads to running very large numbers of MPI processes, depending on how scalable the MPI implementation is, and there are overheads to having too many threads contend for memory on a single node. At least in the current state of things in HPC, there is a delicate balance to choosing the right process-thread ratio, and it differs from application to application and depends on the node count for the job. People struggle to find the sweet spot, but we're definitely not running all MPI anymore.

tgamblin··on Homebrew 1.2.0
It's not yet the production tool that Homebrew is, but there is Spack: https://spack.io, which allows installing arbitrarily many builds of a package, with different versions, compilers, and options (disclaimer: I'm the original author).
tgamblin··on Compiling an application for use in highly radio-active environments
Another place this use case comes up is supercomputing. Not because of the unusual levels of radioactivity, but because of unusual numbers of processors. When you have more components, a single bit flip somewhere is increasingly likely. Resilience has been a research area in HPC for a while now, and people have looked at fault tolerant algorithms, redundancy schemes, faster checkpointing, and other ways to make sure your HPC application running on a million cores won't die because one core gets a bit flip.

So far this has luckily remained a research area in practice, because the vendors tend to do a good job of hardening their machines against errors as they get larger (most of the gloom and doom predictions take the error rates of current hardware and extrapolate). It will be interesting to see if it remains that way.

Relevant article: http://superfri.org/superfri/article/view/14

tgamblin··on Managing dotfiles with GNU Stow
I manage my dotfiles in a separate git repo in ~/.dotfiles, and I have simple scripts that link and unlink everything. It's worked well for me for quite some time:

https://github.com/tgamblin/dotfiles

tgamblin··on Jon Skeet: The 'Chuck Norris' of Programming
The common complaint about stackoverflow reputation is that it's a function of the number of questions answered and not actual skill. http://stackrating.com/ gives an Elo rating of stackoverflow users, where you're rated by how well you answer relative to others (sort of like a ladder).

Jon Skeet is #2 on the Elo scale, so not only is he answering a lot of questions; his answers get upvoted more than others. He really is giving good answers, in addition to being quite prolific. Eric Lippert might be the more interesting data point, though -- despite being 23rd in reputation, he's #1 on the Elo scale.

tgamblin··on So you want to write a package manager
In Spack this would look like:

    spack install guis ^guile@<new version>
How do you handle optional dependencies in this model? Or, a more complex case, dependencies that are only present for certain versions of the package? Look at the gcc package in Spack:

    https://github.com/LLNL/spack/blob/develop/var/spack/repos/builtin/packages/gcc/package.py#L54
This builds gcc 4.x and 5.x versions, and it's pretty concise. The `mpc` and `isl` dependencies are only present when the version of gcc is 4.5 or higher, or 5.0 or higher, respectively.

There's an iterative DAG-building step called concretization that looks at these types of constraints and tries to solve for a satisfactory DAG, which is then built in a functional style. I don't see how I would model that in Guix, and but in Spack it's a few lines, and there's command-line syntax to customize versions on the command line, not just programmatically. e.g.:

    spack install gcc@5.0 ^isl@0.11.1 ^mpc@1.0.2
tgamblin··on So you want to write a package manager
These are the things that Spack attempts to solve. If you look at slides 14-15 in the presentation above, the concretization process automates something like your second example. You can build with an arbitrary package version, e.g. `spack install ruby@1.8 ^libxml2@1.2`, and Spack will evaluate constraints on packages to build a new DAG. If some other dependency in the DAG says it can't build with the older libxml, or if the older libxml implies other dependency version constraints, they'll be integrated in the new DAG. The output of this process is a concrete specification, with all versions/compiler/variants/etc. filled in, which could be used to reproduce the build later.

If spack doesn't know about the version, it can try to scrape the package webpage to find it automatically, or you can just add another one-line `version()` directive in the package file, rather than creating a new package entirely or making a global config change.

My point is mainly that to automate this type of change, you need something like constraint solving to adjust the DAG. That can be used in conjunction with a functional Nix-like build, and actually adds some value to it. That is what Spack is doing. This is why I claim the constraint solving, etc., is not obsolete.

tgamblin··on So you want to write a package manager
My point is mainly that the package specification in Guix is very verbose -- I must hack (and in some cases generate) package files to do what I want to do. The approaches you mention for generating package variants seem to generate new packages -- how do you deal with the profusion of packages that result here?

Don't get me wrong -- Nix is the inspiration for a lot of what Spack is doing, but we've added to it by making an attempt to template the packages by version, compiler, and options. So the usr don't have to "create" the package variant at all: they just `spack install numpy %intel@15.0.1` to compile with the Intel compiler, because the compiler can be swapped in/out of any build. We do similar things with versioned interfaces like MPI -- you can swap OpenMPI or MVAPICH in/out of a build. I have not seen anything to lead me to believe Guix allows this in a simple way, without generating a new package and copying a lot of boilerplate from the old one. The graph transformation stuff you mentioned in your other comment is promising, though.

tgamblin··on So you want to write a package manager
How does this work security-wise? Can any user deploy on your clusters? AFAIK Guix requires root/chroot. This would never fly at our site...
tgamblin··on So you want to write a package manager
> The novel idea in functional package management is that a package can be viewed as a pure function whose arguments are the dependencies, the source code, the build script, etc. and whose output is the built binary.

And this is an awesome idea. Cryptographically hashing your install and keeping the install spec around for later is very powerful.

> Lock files, dependency resolution algorithms, etc. However, these are obsolete.

These are still very important (and hard) problems. Claiming that they’re obsolete ignores what they’re used for.

> The functional package management paradigm approaches the problem from a different angle which is able to precisely describe the full dependency graph for a piece of software, all the down to the bootstrap binaries (the C compiler's compiler, etc.).

I suppose this is fine from the perspective of a system package manager. But at the application development level, there are not very many users who want to specify all that.

If you want to play around with different software stacks in Guix and Nix, you have to actually write packages, which involves tweaking a lot of files. There aren’t so many people with the intestinal fortitude to get down to that level. I want a system where I can say “try running with a different version/build/configuration of this particular dependency”, and then the PM figures out what else needs to be done to accommodate that. Users don’t want to tweak package files a lot, and even less do they want to manage the profusion of version/configuration-specific package files that result from doing such experimentation.

We’ve developed Spack as kind of a compromise between these extremes. Spack [1] has builds parameterized by version, compiler, build variants, compiler flags (soon), etc. and it attempts to let the user experiment with a large combinatorial space of packages. You can do that with a command-line syntax (without editing package files), and you can specify tweaks down to the dependencies with a recursive syntax [2].

I support code teams at LLNL who want to experiment with many different compiler flags, dependency versions, etc., across Blue Gene/Q, Cray, Linux, GPU, and Xeon Phi machines, and none of them want to specify everything quite as rigorously as Nix/Guix demand. What we want is really good build parameterization and a concise way to specify it. I don’t see many Guix/Nix builds written that way — they’re all tied to specific versions and you’d need a new package file to support a new version.

> functional package managers overcome the problems that plague traditional package managers, allowing for unprivileged package management, transactional upgrades and roll backs, multiple variants of a package coexisting without conflict, etc.

That’s one way to look at it. The other is that these systems ignore these problems. What you’re really offering is a really good reproducible build system, at the price of rigorous specification. Reproducibility is a big deal, and there are great reasons to do this, but you can’t say the other systems are “obsolete” when they have very different goals.

What Nix/Guix are not doing is reducing the burden of specification. You write down all the details of a very specific, reproducible software stack, but you do not make that software much more composable or extensible than it already was. The user can install the version combination that you packaged, but can they easily try their own? Can they easily try to build with a different compiler/compiler version/set of compiler flags/dependency version/etc.?

npm, pip, etc. hide a lot of details from the user, and that is why people likely continue to use them (i.e., they are not “obsolete”). Constraint solving, etc., are still necessary to hide a lot of the complexity that users don’t want to deal with. Nix and Guix are great for reproducing a snapshot, but app devs want to explore the build space more than that.

Spack attempts to find a happy medium. We cryptographically hash builds, but we also let the user build things that the original package author may not have tried yet, without modifying the package. That does require all the constraint solving nastiness, but it doesn’t kill reproducibility. Spack stores the generated build provenance after doing some constraint solving [3], but the tool helps fill in the missing details of the dependency graph, not the human. Nix and Guix, AFAIK, do not do that. Spack isn’t fully reproducible because it’s not a full system package manager and it doesn’t use chroot, but you can try the same build again using the provenance that was generated when all the solving was done. The tool just helped the user along the way, and I think that’s still a very useful (not obsolete) thing.

[1] https://www.computer.org/csdl/proceedings/sc/2015/3723/00/28...

[2] Slides 9-11: https://tgamblin.github.io/files/Gamblin-Spack-SC15-Talk.pdf

[3] Slide 14: https://tgamblin.github.io/files/Gamblin-Spack-SC15-Talk.pdf

tgamblin··on So you want to write a package manager
Spack has, to some degree, a compromise between the language and system package management sides of things. Languages in Spack can specify how their extensions should be activated/deactivated within the installation, and modules like numpy can ‘extend’ the language packages. So, you can plug and play with different python modules. I haven’t looked at how Go does language extensions, but it would be interesting to see if this can be mapped to Go, Lua, etc.

[2] http://software.llnl.gov/spack/basic_usage.html#extensions-p...

[3] http://software.llnl.gov/spack/packaging_guide.html#extensio...

tgamblin··on So you want to write a package manager
Got to agree with davexunit here. Language package managers aren't going to handle your non-<language> dependencies. One place where this comes up in spades is with numerical software. Example: frequently you want to build numpy/scipy with fast math libraries (BLAS, mkl, etc.), which might be in C, Fortran or whatever. Python's myriad custom package managers have no support for that level of customization so you're back to manual installs.

Spack [1] attempts to solve this by allowing extensions [2,3] for languages that have their own module systems. Basically you can embed language-specific module enablement logic in the package, and you can enable/disable packages in particular Python installs. You can then build several versions of numpy with different dependency libraries and swap them in and out. It's a compromise between language and system package managers. Right now it's too stateful in that the modules are linked/unlinked into a python installation. We have plans to provide something more like virtualenv or conda environments.

Functional package managers are nice, as are reproducible builds, but they come up short for customizability. I'd have to write an awful lot of packages with Nix and Guix to try out many different versions, compilers, dependencies, etc. Spack has a syntax [4] that allows these to be composed arbitrarily. It's not fully reproducible right now (though it could be if we went down to glibc and used a chroot jail), but the packages are much more composable than the current crop of functional package managers.

[1] http://github.com/llnl/spack/

[2] http://software.llnl.gov/spack/basic_usage.html#extensions-p...

[3] http://software.llnl.gov/spack/packaging_guide.html#extensio...

[4] http://software.llnl.gov/spack/basic_usage.html#specs-depend...

tgamblin··on Maybe: run a command, see what it does to your files without actually doing it
It's intercepting system calls with ptrace, so it could easily fool the process into thinking its calls did exactly what was expected. A cursory glance through the code indicates that it mostly does. However, it looks like it doesn't record changes to files so that they could be read back in by the application. So this will work for applications that read once and write once, but if the same file is written and read, it'll probably break.
tgamblin··on Flat Tree Dependency Resolution in Npm v3
Spack [1,2] is our attempt to do this for HPC, where we have two things going on: 1) Our applications aren't single-language. They include Fortran, Python, C, C++, Yorick (!), Lua, etc. etc. 2) We have a major combinatorial versioning problem, because HPC clusters (and scientific software in general) rely on ABI-incompatible libraries like MPI, and MANY different compilers.

Spack is build-from-source and takes a lot of inspiration from Nix and its descendants, but it adds support for things like swapping out compilers/MPI versions/etc., as well as a syntax for composing new builds of things on the fly That is, you can build something with a slightly different version of a dependency without ever editing a package file. Our users to be able to build a package with 6 different compilers and 3 different MPI library versions, and have their 18 versions coexist on the same system. This has been super helpful for testing production code at LLNL, because we have to run our stuff on other peoples' often very different clusters. For instance, at LANL, we prefer PGI compilers. At LLNL we tend to like Intel or Clang.

One reason languages tend to grow their own package managers is that there are all kinds of peculiarities about how different languages manage modules/extensions. Have you seen how many different ways there are to deploy Python modules? It's very hard to have multiple versions of things coexisting in the same Python env, and things like pkg_resources require editing client code [3]. Ew. Spack tries to address this by allowing different language packages to implement their own module activation/deactivation logic, so you can kind of have a language-specfic package manager for each install of the language. I'd rather have something more like Nix's profiles, though.

I wouldn't say that the project is huge yet, but we've had some interest from mostly other HPC sites. It would be cool if the project caught on in the "real" world, too.

[1] https://github.com/llnl/spack

[2] http://www.computer.org/csdl/proceedings/sc/2015/3723/00/280...

[3] http://stackoverflow.com/questions/5265731

← PreviousPage 3 of 3