HNHacker News
TopNewBestAskShowJobs

tgamblin

594 karma · joined June 11, 2015

submissionscomments
tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
HPC isn't exactly special, and neither to some extent is Spack. Spack's gotten some attention in talks at, e.g., CppCon.

The big differentiator is really the degree to which people want to tune and customize their builds in HPC vs. other communities, and the diversity of hardware that needs to be supported. Things that stand out to me:

  - different applications' needs to customize the builds of their dependencies
    - e.g., one app might need HDF5 with MPI support, another might want it 
      built without. Those are two different, incompatible HDF5 builds.
  - need for specific microarchitecture builds to take advantage of vectorization
  - need for GPU support (for NVIDIA, AMD, *and* Intel GPUs)
  - need to use sometimes vendor-specific tuned system libraries like MPI, cray-libsci, mol
  - need for specific *versions* of dependencies (solvers, mesh libs, etc.) for
    numerical reproducibility
  - need to integrate across languages, e.g. C, C++, Fortran, Python, Lua, perl, R,
    and dare I even say Yorick.
Most of these requirements are not so dissimilar from peoples' dev environments in, say, the AI community, where people really want their special version of PyTorch. Where monorepos are common in industry, they really haven't taken off in the distributed, worldwide scientific community, so you get things like Spack that let you keep rebuilding the world -- and all the microcosms in it.

So I'd say Spack is not so much a special package manager as a much more general one. You can use it for combinatorial deployments at HPC centers, dev workflows for people dealing with multi-physics and other complex codes, and as sort of a distributed poly repo with lock files.

The intent was never to be specific to HPC, and I would love to see broader adoption outside this community.

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
Yep! I'm not a nix expert but my understanding is that this is typically done with overlays. So you can, say, swap out openmpi for mpich, but mpich requires a different version of hwloc that conflicts with the one defined in some other branch of your derivation, you're out of luck. In Spack that is handled through dependency resolution. There have been discussions of how to deal with this in Nix, e.g.: https://discourse.nixos.org/t/concept-use-any-package-versio....

Similarly, for packages like HDF5 that tend to be depended on with options that affect its API (+parallel, ~parallel, etc.), I do not think there are automatic ways to get nix to ensure that your request for hdf5+parallel is consistent with other packages in the graph.

Possibly worth a mention: for uarch flags like the ones shown here: https://nixos.wiki/wiki/Build_flags, the user is responsible for setting them specifically for the requested compiler. We've abstracted that with `archspec` (https://github.com/archspec/archspec) so that you can just ask for a target and the correct compiler flags are injected. e.g., `target=cascadelake` or `target=zen3`. So if you switch compiler, you do not have to look up these details. See https://tgamblin.github.io/pubs/archspec-canopie-hpc-2020.pd... for more on how it works.

The salient parts of archspec aren't in a python library but in a JSON file (https://github.com/archspec/archspec-json/blob/master/cpu/mi...) so it's something that could probably be incorporated into Nix.

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
Sorry to hear this. We have a lot of connections with the high energy physics community, which is why ROOT and Gaudi are even in Spack to begin with. There is an #hep channel on https://slack.spack.io, and we talk to those folks fairly frequently to figure out what we can add to support these codes better.

On `LD_LIBRARY_PATH` specifically: we intentionally inject RPATH-ing compiler wrappers into the build to avoid these types of problems, which are on by default for the ROOT and Gaudi builds. Would be curious to hear where the remaining pain points are.

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
The Spack DSL for defining versions, options, etc. is declarative. That part of a package exposes options to the solver and very much allows reuse, in the sense that the same package can be built many different ways, and even with different dependencies. The whole system is parameterized in ways that Nix is not.

The imperative part of a Spack package is the build recipe, which AFAICT is not so different from bundling a bash script in a Nix derivation. Just like Nix, it's included in our configuration hash, so if you change the recipe you get a different installation of the software.

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
Main differences with Nix:

* Spack has a dependency solver (concretizer). nixpkgs is managed by humans who do the dependency solving for you over time. What this means practically is that you can install 5 versions of some package with different dependencies and flag combinations with a few commands in Spack, while with Nix you would probably need to check out different commits of the nixpkgs repo and maybe do some hacking on nix derivations to get the same thing. Spack is fundamentally designed to help with combinatorics.

* Nix builds all the way down to libc; Spack (currently) doesn't -- it's designed to live on an existing system.

* Nix build environments are isolated (but require root to run); Spack's aren't as completely isolated and don't require root (so you can run Spack in your home directory.

* Spack installation hashes are what Nix would call full configuration hashes -- they're configuration metadata hashes, not content hashes of the installation. So you could say we're not quite as committed to exact binary reproducibility, but you could also say that this allows us to support relocatable binaries (which Nix does not). Also, our metadata is pretty detailed.

* Spack's DSL is Python; Nix is, well, Nix.

Spack is very much inspired by Nix, and we talked about this a bit in the original paper here:

* https://tgamblin.github.io/pubs/spack-sc15.pdf

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
I'd love to have both of these, to be honest.
tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
We have a guide for this written by Adam Stewart, probably our most prolific (package) contributor:

* https://spack.readthedocs.io/en/latest/replace_conda_homebre...

The thing you'll likely notice most about Spack vs. conda/mamba/rattler(?) is that it lets you pick your versions and options much more fluidly, and it'll build you a package from source if there's no binary for it. Spack is much more seamless for integrating binary installations with source builds. `conda-build` is a whole other tool outside the normal user workflow.

Spack has over 7,000 packages at this point, so I don't think the lack of packages is going to hurt too much. We don't offer nearly as many public binaries yet, though, so the source builds may make the UX more painful, depending on what you're trying to do.

tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
If I have to pick one difference, the fundamental one is that Spack does dependency resolution with a solver, while EB does not.

So, in Spack, you write a package.py file (like say the one for gromacs: https://github.com/spack/spack/blob/develop/var/spack/repos/...), and you can build that package with any of the versions/options/dependencies/etc. expressed in the package.

With Easybuild, someone (maybe you?) has to write a specific configuration (easyconfig) for every build you want to do. Want to tweak a version? Write a new easyconfig. The dependencies point at specific versions, too, so now write configs for all the dependencies, and so on. There is a lot of copying/tweaking of file trees in the EB workflow.

Aside from that:

  - Spack has a package database and allows you to query what is installed.
    - EB does not have "uninstall", just `rm -rf`
  - Spack builds from source and also handles binary packages (no binaries in EB)
  - Spack supports environments via `spack.yaml` / `spack.lock`; EB doesn't have that.
  - Spack doesn't require Lmod to work -- you can just `spack load` or `spack env activate` to use packages.
  - Spack builds with RPATH by default (much like Nix does) -- dependency libraries are found automatically even if you run an executable straight from its directory.
There's more but Spack is a package manager, while EB is really set up to automate a certain type of installation common in HPC.
tgamblin··on Spack – scientific software package manager for supercomputers, Linux, and macOS
Original author here -- since I know that people click the comments to decide whether to read further, here's a (biased?) summary:

Spack is aimed at end users, developers, and admins. It's used primarily in HPC at the moment but it would be great to see use outside of the HPC community.

Spack supports:

1. Basic installation (spack install foo, foo@3.1.2, foo@3.1.2 +option, etc.)

2. Reproducible environments, via `spack.yaml` and `spack.lock` files that you can version in a repo. You can use this to build a custom software stack, build some dependencies for a project, etc.

3. Building many versions of the same package, e.g. these are perfectly fine and can coexist:

    - spack install hdf5
    - spack install hdf5 %clang
    - spack install hdf5 %oneapi@2023.2.0
    - spack install hdf5@1.18.0 cxxflags="-O3 -fast" target=cascadelake
    - spack install hdf5 +mpi ^mpich
    - spack install hdf5 +mpi ^openmpi
4. Both source builds and installation from relocatable binary caches

5. Lots of conditional features in packages that other systems do not offer, e.g.:

  - injecting compiler flags
  - building for specific uarch targets
  - optional dependencies
  - very fine-grained options (e.g. +/- cuda, set cuda_arch, etc.)
  - building the same package with different compilers
  - building the same package with different potentially ABI-incompatible dependencies like MPI or different boost versions.
5. Autogenerating things like container recipes, CI pipelines, Lmod/environment modules, etc.

6. Good support for building against external packages that may already be on your system.

At a very high level, Spack has:

* Nix's installation model and configuration hashing

* Homebrew-like packages, but in a more expressive Python DSL, and with more versions/options

* A very powerful dependency resolver that doesn't just pick from a set of available configurations -- it configures your build according to possible configurations.

You could think of it like Nix with dependency resolution, but with a nice Python DSL. There is more on the "concretizer" (resolver) and how we've used ASP for it here:

* "Using Answer Set Programming for HPC Dependency Solving", https://arxiv.org/abs/2210.08404

tgamblin··on The birth of a package manager
Have you looked at Spack (https://github.com/spack/spack)? Disclaimer: I lead the project.

Spack essentially is nix with dependency resolution. spack packages can have conditional (nearly) anything: versions, options, compiler choices, dependencies (including virtual deps). The resolver can make choices about all of those.

See here for more details: https://arxiv.org/abs/2210.08404

tgamblin··on Mapping Out the HPC Dependency Chaos
Spack installs the software and generates the modules (you can customize how). See https://spack-tutorial.readthedocs.io/en/latest/tutorial_mod...
tgamblin··on Mapping Out the HPC Dependency Chaos
Co-author/spack guy/occasional frequenter of Nix fora here.

Aside from what p4ul said above, the main architectural differences between Nix and Spack are: 1. Spack has a dependency solver (the "concretizer"), Nix does not 2. Spack packages are parameterized, and we solve for parameters like build options, compilers, virtual dependencies, etc., so the user can say what tweaks they want on the CLI or in an environment. To do this same with Nix packages you typically need to hack on some nixpkgs. The goal of Spack is to be able to compose and swap libraries, compilers, and other options easily. 3. Spack has more support for external packages (or "impure" packages as the Nix people say). You can point it at an existing installation on your system to use as a dependency.

Hashing installation directories, the store-based installation model described in the paper are things the two have in common.

More on the solver aspect in this other paper that we're also presenting next week: https://arxiv.org/abs/2210.08404

tgamblin··on Ask HN: Who's having a ton of fun at work and what are you doing?
You can do this kind of work and also get paid decently at a national lab. For example, LLNL is hiring!
tgamblin··on Single-chip processors have reached their limits
All of this is true, but I’ll just be nitpicky to make a point:

> Does a future event's processing circumstances maybe depend on all events received up until now?

In a parallel prefix sum, the final sum does depend on all prior inputs, but a good parallel implementation runs in O(log(n)) time. It is, of course, not a total ordering problem, but that’s not obvious at first glance — I’ve always thought it was a beautiful example of something that appears to be entirely sequential but actually parallelizes really well.

All of which is to say that, yeah, we’re latency bound at the end of the day, but there’s a lot more innovation and performance left to be wrung out of most systems. A little creativity goes a long way, and I like the idea of a universe where the software is the main determinant of performance — where you can’t get away with just waiting for the next generation of chips.

tgamblin··on Ask HN: Those who changed industries from web dev, where did you go?
I did a year of server-side web dev from 2002-2003, then decided I wanted to do something different. I went back to grad school, got my Ph.D., and now I’m in high performance computing.
tgamblin··on Ask HN: Where can we see simulations of a nuclear war?
Not a nuclear war simulation, but what you’re asking about is the mission of NARAC at LLNL: https://narac.llnl.gov

NARAC monitors and predicts the spread of wind/water carried nuclear, biological, chemical, and other hazardous materials in emergencies. You can see videos of recent responses on their website; the highest profile one was probably the Fukushima disaster (if you remember the various evacuation recommendations, NARAC guided the US ones). Here are some more: https://narac.llnl.gov/about/timeline, including the Eyjafjallajökull volcano.

They run in some of their own (comparatively) smaller clusters for quick turnaround simulations, as well as on LLNL’s bigger HPC systems (https://hpc.llnl.gov/hardware/compute-platforms) when there’s a need.

tgamblin··on NixOS 21.05
Guix and Nix don’t actually solve constraint problems, though. By design, they omit the dependency resolution that other package managers do, and humans manage the dependencies. If you want to change versions, you need to do some hacking to update things and propagate requirements through the package graph.

You could think of Spack (github.com/spack/spack) as a nix/guix-like model, but with a solver. Spack uses ASP (which is prolog-like but reduces to SAT with optimization) but packages are written in an embedded python DSL.

tgamblin··on What to do about GPU packages on PyPI?
Check out the tensorflow package in Spack. We patch Bazel slightly to allow external dependencies/library paths, and the dependencies are factored out into their own packages:

https://spack.readthedocs.io/en/latest/package_list.html#py-...

Spack can build with two levels of parallelism (within a build and among builds) on a single node or multiple nodes with a shared filesystem:

https://spack.readthedocs.io/en/latest/packaging_guide.html#...

You can also generate GitLab pipelines of your build, which will break the jobs out as well:

https://spack.readthedocs.io/en/latest/pipelines.html

tgamblin··on What to do about GPU packages on PyPI?
This is why Spack supports external packages. For any package, you can tell spack the prefix where it lives and build with that install prefix instead of a spack-built version:

https://spack.readthedocs.io/en/latest/build_settings.html#e...

People use this the tell Spack where the system cuda libs or system MPI are.

tgamblin··on What to do about GPU packages on PyPI?
And with dependency resolution (spack calls it “concretization”), which nix and guix do not have.
tgamblin··on Cosmopolitan Libc 1.0
See https://www.python.org/dev/peps/pep-0513/#key-causes-of-inte....

Software built with a newer glibc is not compatible with software built with an older glibc. You can't just build an EFL binary on an arbitrary distro.

Moreover, and probably more notably for cosmopolitan, binaries built with glibc aren't compatible with BSD libc or musl. IIUC, cosmopolitan works across the BSDs and musl-based Linux distros, as well.

tgamblin··on Cosmopolitan Libc 1.0
> I guess I just don’t see the use case where this is compelling. If I write a handy Unix utility in C, I’ll just keep the source code around, and compile it as needed.

Isn't this exactly why you don't see the use case? You're willing to compile.

As someone working on a cross-platform, cross-language packaging tool (https://github.com/spack/spack), it's very appealing to not have to build for every OS and Linux distro. Currently we build binaries per-OS/per-distro. This would eliminate a couple dimensions from our combinatorial builds.

We still care a lot about non-x86_64 architectures, so that's still an issue, but the work here is great for distributors of binaries. It seems like it has the potential to replace more cumbersome techniques like manylinux (https://github.com/pypa/manylinux).

tgamblin··on Sbang lets you run scripts with long shebang lines
Yeah I guess I read that too quickly -- sorry.

This:

    #!python3
Seems to work (i.e., search $PATH) on macos but AFAIK nowhere else. It fails on both ubuntu20 and centos8. Per this: https://www.in-ulm.de/~mascheck/various/shebang/, relative paths are supposed to be interpreted as relative to ., so I'm guessing that is why everyone uses /usr/bin/env.

Given that shebang is implemented in the kernel (and not in user-space like coreutils) I think env is more likely to change than the shebang mechanism, though who knows -- there have been recent changes to shebang so perhaps further work on this will help: https://lwn.net/Articles/779997/.

tgamblin··on Sbang lets you run scripts with long shebang lines
Yeah it would be nice if POSIX would move a little faster, but I guess the slowness is the point.

This is probably obvious but bears repeating since we're on this tangent -- even -S and -P do not prevent the OS from truncating the shebang if it's too long. So env is great for passing multiple args, but it still doesn't let you use really long shebangs (the original point of the thread :)

tgamblin··on Sbang lets you run scripts with long shebang lines
It's there because on some systems all the args are passed as one. -S splits them out. You can try this on Linux and env even warns you about it:

    # cat test.py
    #!/usr/bin/python3
    import sys
    print(sys.argv)

    $ cat test
    #!/usr/bin/env test.py foo bar baz

    $ PATH=. ./test   
    /usr/bin/env: 'test.py foo bar baz': No such file or directory
    /usr/bin/env: use -[v]S to pass options in shebang lines
with -S, env does the right thing, and you get this:

    $ cat test 
    #!/usr/bin/env -S test.py foo bar baz

    $ PATH=. ./test
    ['./test.py', 'foo', 'bar', 'baz', './test']
tgamblin··on Sbang lets you run scripts with long shebang lines
I'm not sure how this solves the problem. The goal is to execute an interpreter in a long path via the shebang line, but this is a regular script that executes a program in a long path. You can't use this to make a file invoke /some/long/path when executed. Or am I missing something?
tgamblin··on Sbang lets you run scripts with long shebang lines
True. The two disadvantages are:

- you still get the python3 in PATH, not a specific one (sbang aims to use a specific interpreter)

- The -S argument to env is not POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/e...

tgamblin··on Sbang lets you run scripts with long shebang lines
In Spack https://github.com/spack/spack), we encounter this when users install packages in deep paths within their home directories, or in deep paths within shared project directories (e.g., in NFS or Lustre on HPC machines). So we patch installed scripts that have long shebangs.

We want installed packages to work exactly as built, with the right versions of dependencies, and we don't want to rely on the user getting their environment right to do that. Spack users may install several versions of python or other interpreters. This ensures that scripts work without a special environment.

There is another use case that is less emphasized here. sbang also lets you pass arbitrarily many arguments on the shebang line. If you do this:

    #!/usr/bin/perl arg1 arg2 arg3
At least on Linux, you'll get one argument: "arg1 arg2 arg3". I think perl gets around this by parsing the shebang line itself, but sbang is more general. You can use it with /usr/bin/env to pass more arguments than would otherwise be allowed.

See https://www.in-ulm.de/~mascheck/various/shebang/ for an extremely comprehensive list of limitations on shebangs.

tgamblin··on Ask HN: Is your company sticking to on-premise servers? Why?
Some, but LLNL codes are mostly C++, especially now that they need to make effective use of GPUs.

More on codes:

- https://computing.llnl.gov/sierra-supercomputer-dedication

Some repos of interest:

- https://github.com/LLNL/axom

- https://github.com/LLNL/RAJA

tgamblin··on Ask HN: Is your company sticking to on-premise servers? Why?
thanks!
← PreviousPage 2 of 3Next →