Hacking anything with GNU Guix | Hacker News Reader

Hacking anything with GNU Guix

gexp.no

307 pointspodiki102 comments

Another great use of guix shell that I tend to do is for one off commands. I don't need OBS installed to use it occasionally, I can just do "guix shell obs -- obs" to run it once and not have it in my PATH, application launcher, etc. when I don't need it. Or better yet for running python or CL scripts, with something like 'guix shell python [python packages] -- python3 myscript.py "some argument"'. No need to carry around these libraries. This can be done with a manifest or guix.scm file as well, that guix shell will use automatically (once authorized, so you don't accidentally run something).

This keeps things tidy in terms of just having packages I need all the time installed, while still being very quick after the first run. (There are some caveats about garbage collection, but I rarely do that.)

And as the article mentioned there's also containers...lots of useful tools.

'guix shell' is killer. I use it for all my projects and one-off experiments. I also hook it up to Emacs 'M-x compile' so that compilation happens in the context of the guix shell. Beyond the simple command line package specification, you can add a guix.scm file to the root of your project repo and fill it with code that specifies all the packages needed for a development environment. Some of my guix.scm files are simple lists of existing packages, others are full of custom code to modify existing packages or define new ones. No matter how simple or complex it is behind the scenes, I just run 'guix shell' and get on with working on my project.

You might be interested in buffer-env, it can automatically load an environment on a buffer-to-buffer basis and make it appear as though it were globally installed. No manual "guix shell ... -- ..." calls needed.

Thanks, I've been meaning to check it out, so long as I can integrate guix shell with it.

Could you elaborate a bit on how this works? Am a longtime emacs user and very motivated to get this working!

I wrote down my notes on how to use it here: https://amodernist.com/texts/emacs-guix.html.

Thank you!!

Any pointers to your setup?

I was looking for an example as well, when I saw the nickname of the GP. He created a nice project named Chikadee and the source code can be seen at https://git.dthompson.us/chickadee.git/tree/, where you can also find the guix.scm file.

As zelphirkalt pointed out, one project I use with guix shell is named Chickadee. It's a little game dev library.

Here is the guix.scm file: https://git.dthompson.us/chickadee.git/tree/guix.scm

Here is another project (a static site generator) with a much simpler guix.scm file: https://git.dthompson.us/haunt.git/tree/guix.scm

In both cases, this same file can be used to create a development environment via 'guix shell', build from scratch in an isolated environment (handy to run before a release) via 'guix build', and it can even be installed with 'guix package'. It's super handy.

On a high level, this reads like what Docker is kind of achieving.

{docker, nix, guix} take any two. What are their differences?

Guix shell does not by default work inside a container. There is --container, as mentioned in the article, but by default guix shell is setting up a clean shell to work in, which has the dependencies available.

A guix.scm file is a mostly declarative file, where one specifies the "what" not the "how", in contrast to a Dockerfile for example, where one writes sequential steps to take, to install all that is needed, which is of course much more prone to making mistakes. In guix one relies on the package specifications, which are already in guix for the dependencies and guix' machinery to process them. In docker one uses the directives offered by docker to write a Dockerfile instead.

I guess one could say guix is a bit higher level than docker and does not require you to know how to install something you depend on. On the other hand you need to know about guix' way of describing a package, if you want to create one. But for mere dependency specification, guix is much simpler than writing a Dockerfile.

Just to emphasize, the lack of docker is a good thing - full virtualization is nice but as with anything comes at the cost of overhead, complexity (you need to connect things between docker instances and disparate data sources? Probably need some fancy solution with its own quirks).

I haven't used it yet, but I love the concept - I use small environmental scripts all the time, explicitly to not have to deal with vms, this sounds like it is much more flexible and crucially much more reproducable.

I had an idea recently to try out Fedora silverblue with Guix on top.

Silverblue gives you an immutable root filesystem, which is great for building a reliable and reproducible system, but it makes installing software more complicated. Containers are the main solution, and stuff like flatpaks and toolbox. That (mostly) works, but it’s a bit heavy handed and not without annoyances.

But with guix, your packages live in the guix store, and don’t touch your os packages. So that seems like a good way to manage software on top of an immutable OS.

Unfortunately when I went to try it out, I learned that the only way to change the default location of the guix store (since /gnu/store won’t work in silverblue, and Guix isn’t available as a package for fedora) is to build it from source... and then I ran into some build issues when trying to do that and gave up.

...So maybe someone less lazy than I will pull it off and report back whether it’s actually a good idea or not!

I'm downloading the silverblue iso right now to play with this in a VM. It is surprising to me that ostree does not let you do any sort of bind mount or symlink of a non-reserved location like /gnu to some other filesystem, so I'll have a whack at it.

> It is surprising to me that ostree does not let you do any sort of bind mount or symlink of a non-reserved location like /gnu to some other filesystem, so I'll have a whack at it.

...Well, it might. I actually had zero experience with both ostree/silverblue and guix when I tried doing this, and may have made some hasty assumptions.

EDIT: I do know however that guix doesn't work if the store is a symlink. I tried doing that a while back and it didn't work.

So you can create /gnu/store on Silverblue with the commands (as root)

  chattr -i /
  mkdir -p /gnu/store
  chattr +i /

The last command restores the immutable attribute on the root filesystem. I am able to write arbitrary files (e.g. `touch /gnu/store/test`) even with that attribute set, so it isn't inherited by subdirectories (maybe obvious).

Now, persistence of the /gnu/store data could be an issue, I'm not sure, but maybe this gets you started?

EDIT: next step is to get Guix installed and then try to update the system, we shall see how it goes

I just tried doing that to create the bind mount at /gnu/store via fstab, and it disappeared on reboot (and threw me into an emergency console). Removing the fstab line allowed me to boot normally again.

I think the solution will require using systemd to recreate that folder on boot, and do it early enough to not make the system panic like that.

Maybe that's a bad idea, I don't know enough about ostree/silverblue. However, after some more digging it seems like this solution would be preferable to changing the default guix store with a custom build, since doing that means you can't use any of the pre-compiled binary packages from the official guix servers.

So, having removed the immutable attribute from / and run the guix-install.sh script, the first issue I have is that the script thinks the group 'kvm' exists, because 'getent group kvm' returns successfully. However, if you try to create a user and add them to the kvm group, you get an error about the group not existing. Looking at the contents of /etc/group, there are only 5 groups and none of them are 'kvm'. At the very least it seems that the Guix install script is making some assumptions about the system that Silverblue breaks, which does make me wonder a bit about using the two side by side.

Additionally, I have found some posts as far back as 2017 regarding using Guix or Nix on Fedora Atomic Linux or Fedora Container Linux (it seems the Silverblue name is more recent) and the fact that we're still puzzling it out 5 years later without much in the way of documented success does make me question the viability of the idea a bit.

Wouldn't that require that the mountpoint exists in the underlying filesystem? That is, you can't create a /gnu directory to mount into if root is read-only

EDIT: Or is the filesystem not literally read-only? Sibling comment seems to imply that it's just chattr +I'd

I just tried it, and the folder created via chattr disappered after rebooting. So it seems like there's more to it.

Not on Silverblue but I've used a bindmount to have /gnu/store elsewhere (due to disk space on a laptop). I can dig up some notes if that is helpful.

I didn't think to try doing a bind mount, so that might actually work. I'll need to try it again at some point!

Used this same trick to use `$HOME/nix` for my nix store on a remote system that had a slow and ephemeral disk for the root fs, but a fast and persistent one for my home folder.

Not sure about guix, but for nixos we have https://github.com/nix-community/nix-user-chroot

You can do a pivot_root based install of Guix: https://github.com/pjotrp/guix-notes/blob/master/GUIX-NO-ROO...

What does silverblue give you on top of what Guix and Nix already provide?

In Guix's case, a working desktop with wayland + proprietary Nvidia drivers. (I'm actually using Kinoite, which is the KDE spin of silverblue)

I could install Guix on top of something like Ubuntu or Fedora workstation for the same effect, but then I'll have two system package managers side by side, which is pointless and probably error prone.

You can do that with NixOS trivially.

And of course with Guix System. (odd to always see Nix* mentioned when someone talks about Guix.)

Comparing https://wiki.systemcrafters.cc/guix/nvidia/ with https://nixos.wiki/wiki/Nvidia makes me think that in this specific case there's a difference.

A question for nonguix contributors, including OP: is there a reason that nonguix has an nvidia package but not an nvidia service that handles more of this in a streamlined way, even just some stuff copied and pasted from this wiki?

There's an ongoing effort to smooth out the entire Nvidia process, e.g. https://gitlab.com/nonguix/nonguix/-/issues/198 has the current discussion. As anyone that has dealt with Nvidia on Linux, it is always a bit of a hassle and the effort hasn't yet happened to make it as smooth in Guix as other distros. We're getting there though.

Well, Guix is also mentioned when someone talks about Nix, seems fair to me (and of course, Guix may be more relevant sometimes!).

Well, Nix was mentioned a few posts up... But in general, Guix is directly based off of Nix and is seen by many as "Nix + Guile + GNU". Is it odd to see people talking about a different BSD in a {Free,Open,Net,*}BSD thread?

Guix is quite different from it's origins, though some of the oldest part of the daemon are still from the Nix days, if I understand. It will one day be rewritten in all Guile, presumably.

An FHS base system that still gives you some of the nice features of Nix and Guix, like immutability and rollbacks.

In theory something like it could be a giant escape hatch so that if you run into Guix packaging difficulties for some software that, e.g., tries to fetch from the internet at runtime, you can just build with autoconf and make or `npm build` or whatever against the Fedora base system. Silverblue might be quirky enough that it has its own hiccups though, idk.

Nix provides FHS environment sandboxes, which are also a great escape hatch, or an alternative for quick packaging: https://nixos.org/manual/nixpkgs/stable/#sec-fhs-environment...

I think nonguix also uses equivalent machinery for its Steam package: https://gitlab.com/nonguix/nonguix/-/blob/master/nongnu/pack...

Maybe there are docs on repurposing that as a general packaging escape hatch, which someone more familiar with nonguix could point to in a reply.

Apparently there is an rpm-ostree-friendly RPM build of Nix that can be used to install Nix on Silverblue as well: https://github.com/nix-community/nix-installers/pull/8

It includes some code to get around the group/user management quirks that some other commenters have noted b0rked the Guix installer with Silverblue, so perhaps it gives an outline of how to install Guix on Silverblue for a hacker who'd like to try it :)

See my comment above about adding a general FHS container to Guix itself; it works, just need to put the final touches on the patches.

Awesome! That is great to hear.

There's an FHS emulating option for guix shell containers that has been proposed (it needs some minor code changes but is functional; I use it): https://issues.guix.gnu.org/56677

So doing a guix shell --container --emulate-fhs <packages> will have you in a container with the usual /lib, /bin, and so on, which should work just as a "typical" Linux distro. You'll want to share things from the host to the container (e.g. graphics) but that is all standard guix container options.

That is very cool. Could one then persist binaries built against such a container with something like `guix pack`? That would be absolutely killer. (Not reproducible at all, but an awesome escape hatch for building software in a traditional way, ad-hoc and imperative, on GuixSD. Could be really nice when packaging an odd piece of software feels overwhelming.)

Either way, that's pretty exciting and I hope it can be merged soon!

I'm not sure how that could work exactly, but yeah, did think of some sort of guix shell build environment container. Though I guess inside it should be the same as "regular" Linux, but you've specified the environment exactly (i.e. you could capture the exact Guix commits to make a reproducible container). Unfortunately, once in there you have state to worry about, so you'll lose reproducibility control as you say, but do think there could be some way to make this a nice stop-gap.

So would guix be something like an even slimmer version of docker (which in turn isna slim version of a VM)?

It should be pointed out that "guix shell" is currently only available in the unstable "latest" version, not the stable "standard" release (v1.3.0). If you are running an older version, "guix environment" is similar.

https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix...

https://guix.gnu.org/en/manual/en/html_node/Invoking-guix-en...

Running `guix pull` and restarting the daemon should update guix to the latest given it's a rolling release.

Can anyone explain to me the appeal of guix over nix? Is the learning curve any easier? I really badly wanted to like nix...but its just so strange and unconventional.

Whenever I need a disposable environment I reach for lxc/lxd.

What in particular did you find strange and unconventional?

For me it was the core premise that is both the blessing and the curse. That it is only possible to install software by first packaging it.

That's because the basic idea is immutability and that doesn't go well with OSes designed around mutability.

Nix/Guix are basically the "glue" that connects two worlds.

Haven't tried guix yet, and haven't used nix in years, but if what you struggled with is the language, guix might be less strange and more conventional since it uses Scheme as the language. It uses strict evaluation like most languages and there's probably more documentation about it.

I think the main difference is that Guix will only support open software officially, whereas Nix will also happily allow proprietary stuff, like nVidia drivers.

See e.g.:

https://gitlab.com/nonguix/nonguix

> Guix channel for packages that can't be included upstream. Please do NOT promote or refer to this repository on any official Guix communication channels.

Far from the 'main difference' -- if there was a project that just forked Nix and made that change, you could say that about it, but not about Guix, which has tons of work-hours invested and pretty much none of those work hours have to do with that.

True but that's from mostly a technical perspective. It is also (more?) important to know how a community thinks.

I think the main difference is that Guix uses an actual language and writing things for it seems pretty reasonable, while Nix language is frustrating to work with and look at. There are probably hundreds of functions in nixpkgs that would make your life easy, but only 3.5 people know about them or how to use them.

Guix is strange and unconventional in the same fundamental way as Nix, namely that each package is installed to its own, immutable, quasi-content-addressed prefix. This is because that design decision is what gives both package managers their superpowers.

If you are already a Scheme user, you may find Guix more accessible for you. If you are not a C++ programmer, you may find hacking on the package manager itself easier with Guix.

In terms of the CLI interface, Guix really shines here. Guix also has a more centralized approach to documentation, which you may find helpful.

Beyond the surface-level language differences, Guix has gone with different abstractions than Nix in some key areas, and has some features that Nix lacks. One big one is that GuixSD uses a different model for defining the options that can be used to configure the system. Guix's approach¹ is more explicit, and features some provenance tracking for configured options— it can draw a graph for you showing where each setting on your system came from.

Guix also takes a different approach to pinning package versions and defining repositories of source packages, and its conventions for doing those things are more settled than their equivalents and alternatives in the Nix world.

Guix also supports a feature called 'grafts'² that allows you to avoid rebuilding the world in case of things like mission-critical security updates to glibc. This is a really cool and useful feature!

I'm sure there are other things that more serious Guix users can better highlight than this dilettante. :)

The Guix blog is really excellent! I strongly recommend it for getting a sense of what problems Guix tries to solve and how it sometimes approaches them differently than Nix does.

1: https://guix.gnu.org/manual/en/html_node/Defining-Services.h...

2: https://guix.gnu.org/blog/2020/grafts-continued/

I tried and bounced off Nix on other systems, and I'm now running nixOS on my personal laptop (which is a secondary device).

I think there's absolutely room to solve the same set of problems better than Nix does:

1. The number 1 problem for me has been documentation of nixpkgs. Nix lang is a bit funky but even if I was writing python the problem is that you're writing code to assign magic objects to magic variables and the only way to find the right ones is to read the nixpkgs source (and given the size of all-packages.nix and the limit of github's web viewer, maintain a local checkout).

2. Second place goes to the flake/non-flake divide where the nix community generally implies flakes are better but apart from the nix cli detailed docs, most things refuse to acknowledge its existence in the official docs.

3. Portability between macOS and Linux, where flakes actually make the situation worse as the root config is now system specific.

4. Tools like flakes, niv, the suggested way to write a shell.nix all want you to handle full commit hashes directly which is kinda unergonomic.

None of these are inherent to the problem space. If I was to keep writing the list, maybe around 9 and 10 are the things around nixlang being a funky language or /nix directory that the "you just need to understand functional languages/content addressable stores" discussion seems to think are the top ones.

Agreed on points 1 and 2. I think 3 is basically solvable for Nix, and it's easy to paper over with a Nix library for now.

Re: 4, Flakes do let you define inputs in terms of Git tags and branches and then have the computer resolve those to commit hashes for you, which is good.

Overall, I agree that it's not fair to think of package managers that work in the same basic paradigm as Nix as mere also-rans or clones. There's a lot of room to meaningfully experiment in the space and Guix's developers have proven thoughtful about where they want to differ in technical and ergonomic matters.

From what I can tell (I've only used NixOS/Nix), Guix has considerably more thought put into the UX: Nix is currently undergoing a transition to a newer CLI; Nix's nomenclature is confusing; Guix has much better documentation; Guix has its equivalent of Nix's home-manager built-in.

There is still some division in the Nix ecosystem between the pre-flake and the experimental post-flake way of doing things.

Guix uses an established programming language for configuration, which some might find attractive (I actually quite like Nixlang after getting used to it).

Guix makes installing non-free software a hassle (you have to include community sources). Nixpkgs doesn't impose this restriction, though you still have to explicitly allow the install of non-free packages when running Nix.

Overall, Guix seems like the more polished product, though NixOS/Nix is similar in functionality and has a larger collection of packages and more traction in general.

> Overall, Guix seems like the more polished product

Possibly when comparing the Guix vs. Nix package managers, but for Linux distributions GuixSD ("Guix System" now) is very far behind NixOS in this regard. I've tried to install GuixSD on different hardware several times over the past couple years, and failed every time, between a lack of drivers and unpolished or buggy installer. Last time the installer wiped out my partition table without prompting when I went in to manually partition (to set up dual boot).

NixOS on the other hand has always been flawless to install, and now there's even a modern GUI installer.

I like Guix better in theory, but Nix wins in practicality.

Well the thing with the "Guix System" is that is uses the LinuxLibre kernel. So if the hardware you have requires proprietary bits or driver, it just isn't included. That isn't to defend Guix in anyway, but the problem doesn't seem to be Guix when you read the fine print of the distro, the problem is your hardware.

So, do I agree necessarily agree with GNU's official distributions having this philosophy? Not really. Firmware and drivers is probably the one area I think the GNU people should finally throw in the towel. It keeps people out of running it, or doing so with relative ease. And the battle is for the most part lost on that end. But they do essentially tell you that it is likely it won't work across a lot of hardware out of the box.

But here is the download page [0] https://guix.gnu.org/en/download/ That mentions the Libre kernel with link to information about the Libre kernel [1] https://www.fsfla.org/ikiwiki/selibre/linux-libre/ With the following information:

"Linux, the kernel developed and distributed by Linus Torvalds et al, contains non-Free Software, i.e., software that does not respect your essential freedoms, and it induces you to install additional non-Free Software that it doesn't contain. Even after allegedly moving all firmware to a separate project as of release 4.14, Linux so-called "sources" published by Mr Torvalds still contain non-Free firmware disguised as source code. Stux, a cute penguin. Few realize he's not Free

GNU Linux-libre is a project to maintain and publish 100% Free distributions of Linux, suitable for use in Free System Distributions, removing software that is included without source code, with obfuscated or obscured source code, under non-Free Software licenses, that do not permit you to change the software so that it does what you wish, and that induces or requires you to install additional pieces of non-Free Software."

I think that allow unfree in nix is practically the same as adding nonguix repo to your channels. Both are essentially one line of config in some file.

(I am former nixos user but now on guix)

In addition to the other answers here: the Guix repositories only accept Free Software, even excluding things like Firefox, and intentionally make it somewhat difficult to install non-Free-Software (as opposed to Nix, where installing things like Firefox is relatively easy[1]). This may appeal to some people.

[1] https://nixos.wiki/wiki/Firefox

Reminds me of the VHS vs Betamax war, where VHS won because Betamax (Sony) didn't allow adult videos (at least, so the legend goes).

Guix does not make non-free software "intentionally more difficult", it just excludes it from the main repository. There is a nonfree repo that you can add to your guix channels.

the non free repo explicitly asks you not to talk about it

https://gitlab.com/nonguix/nonguix

looks intentional to me

Well, all it says that you shouldn't promote it on the "official channels" (i.e. the mailing list and the #guix libera.chat channel). Guix, the package manager itself, does not make installing non-free packages any more difficult than free packages. I suppose it could be said that Guix (the project) makes finding non-free packages harder, although anecdotally I will say that nonguix is the first thing I've heard about guix, since it seems to be the most controversial part of it.

While it's technically correct to say that the software itself does not make it particularly difficult to install non-free packages, it also doesn't come with non-free repos when you set it up, and the documentation and official support channels intentionally lack information on non-free repos. As default settings and documentation are a critical part of a software project, it still has the end result of making it harder (much harder for non-technically-inclined users) to do so.

Non-free software is just off-topic in official channels, there is no need to interpret malicious intent into it. It is no more difficult to enable nonguix than it is to enable any other repository.

I don't think anyone was implying malice, just that there was intent. Nothing wrong with having principles.

I'm sorry, but if ask how to install Firefox and no one tells me in official channel or, worse, suggests alternative browser/fork of firefox. That's a pretty intentional way of making things hard.

I'm sure someone will tell you, perhaps privately. Everyone is quite nice there and no one is going around trying to shame people or kick them out.

One of my personal pet peeves are that Guix does not support Apple, while Nix does.

guix is written in basically scheme. Nix is some weird stringly typed DSL thing.

...except KDE[1]:

> GNOME, Xfce, LXDE, and Enlightenment are available (see Desktop Services), as well as a number of X11 window managers. However, KDE is currently missing.

[1]: https://guix.gnu.org/manual/en/html_node/Limitations.html

> PYTHONPATH, CPATH, etc are all set up and ready to go.

Does that mean that Guix just exports the required environment variables in the shell rather than wrapping each executable with a bash script [1] like nix does?

If yes, that's great, because the wrapper approach feels like an ugly hack. I found some executables on my nixos installation that are behind three layers of wrappers, and that's probably not the maximum.

I guess nix could improve this situation by making `wrapProgram` smarter (if the executable to be wrapped is already a wrapper, merge the inner and outer wrapper), but even single-layer wrappers are annoying, and I imagine they have some performance impact.

EDIT:

I forgot about nix-shell, which does actually export the right environment variables directly to the shell.

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-supp...

Does nix-shell/`nix develop` use makeWrapper?

Thank you, I totally forgot how nix-shell works. You're right, it should export the right env variables without makeWrapper.

I haven't used nixos in a while and I got confused about this because to install python packages globally I used `python3.withPackages` and that does use the wrappers.

Guix has a mechanism called search-paths which is defined for any package like Python that searches for things based on envars; it exports the relevant search paths into the environment. Though it also has shell wrappers, which are essentially a hack around non-propagating dependencies: it allows you to expose things that would normally be visible to everyone, like executables, to just the specific program.

Plus: You can make it all reproducible using `guix time-machine`.

It seems so obvious to me now that the fundamental design decision in Nix/Guix (to encapsulate dependencies via a content-addressed store, to make only those specific dependencies visible to the dependent app or library, to express all of this declaratively somehow, and to basically treat the entire build process and in fact the entire system configuration like a pure function with a deterministic result) is the way forward.

I had a couple people attempt this for the dev environment of https://www.oilshell.org/ with Nix, and it wasn't entirely successful. (Not Guix, but my understanding is that Guix would have the same issues)

As background, we've long had a set of evolving shell scripts that fetch and build dependencies at specific versions -- like bash/dash/zsh/mksh/busybox to test against, re2c to generate code, CommonMark, Python 3.10, MyPy with pip dependencies, and (bonus) R with CRAN dependencies.

I wrote about 2 problems here: https://lobste.rs/s/s5co2f/where_contributors_have_problems_...

1. OS X and libc, which is not really a problem since our existing scripts don't solve it either. It was just one motivation for Nix that didn't quite work out.

2. The file system layout becomes different, and Oil's shell tests rely on that. So containers ended up being easier. The whole build and test system runs in OCI containers under Docker and podman now, so it's pretty reproducible and automated.

But I still think it would be nice if someone who actually knows Nix and Guix (unlike me) tries again. The dependencies are more stable now than 2 years ago.

I think you have to write like 10 Nix or Guix expressions from scratch with the exact tarballs that we use. (Otherwise the tests will break even more, because Oil's tests are extremely detailed and find bugs in specific versions of specific shells.)

Right now we have a 134 line shell.nix that tries to reuse Oil's scripts, but I think it doesn't gibe with the way that Nix and Guix are meant to be used. Probably the real solution would be more like 1000 lines from scratch?

I remember that Nix Flakes was what I thought Nix was going to be, but at the time it wasn't ready. I thought Nix was supposed to solve the "it works on my machine" problem but it actually doesn't -- you still need a CI because it's possible to write .nix expressions in ways that break the sandboxing (unlike Bazel where you always get it).

I had ran this by someone who knows Guix and my takeaway was that Guix is basically the same in that regard.

A key problem is that, at least for awhile, I want it to work in parallel with our existing system ... not have a "big break".

Also, reading back on that, I think the hardest part would be "PyPI dependencies" of MyPy, and "CRAN dependencies of R scripts like dplyr". These have what I call the "rewriting upstream" problem.

Though maybe a hybrid approach can work? Or does that defeat the purpose? I think a problem with these kinds of systems is that they can be "all or nothing". If you're halfway in, and halfway out, you don't get any benefits.

For lang-specific package managers that have a lockfile format, it's possible to write tooling that lets you run `cargo lock` or whatever and then generate the required Nix expressions for those dependencies. For stuff that doesn't support such a format, if you had your scripts export to your own little json lockfile format, you could have Nix read that (Nix expressions can ingest json).

There's some ongoing work to better support 'generating' those Nix expressions and using them in builds without explicitly storing them anywhere, as well (John Ericson's RFCs on 'import-from-derivation' and related functionality), and for unifying such tooling (dream2nix). (This kind of thing is what the `guix import` CLI does, but it has some limitations.)

There's some talk of possibly using Oil Shell for a port of Nixpkgs to Windows, although another language might be chosen by whoever really takes up that work. People interested in adding native Windows support to Nix might be interested in helping you sort out Nix packaging for Oil (I see there is a package currently, but it's not a source build).

Hm interesting, are there any examples of that? I want to add lockfiles for all the package managers we use, e.g.

https://github.com/oilshell/oil/issues/1081

> But I believe libc is fundamentally a “hole” in the sandboxing of Nix, at least on OS X.

Nix doesn't sandbox builds on macOS at all by default, and turning sandboxing on breaks some builds: https://github.com/NixOS/nix/pull/1821

But the libc issue goes deeper. This blog post has some explanation: https://matthewbauer.us/blog/darwin-stdenv-update.html

> Linux is unique among operating systems due to the fact that the Kernel and Libc are developed independently. Linux is maintained by creator Linus Torvalds and a community of contributors. Glibc, the most popular Libc for Linux, is maintained by the GNU project. As a result, Linux has a strong separation between Syscalls and Libc.

> [ ... ]

> To accomplish this, Linux provides a stable list of syscalls that it has maintained across many versions. This is specified for i386 at arch/x86/entry/syscalls/syscall_32.tbl in the kernel tree. The syscalls specified here are the interface through which the Libc communicates with the kernel. As a result, applications built in 1992 can run on a modern kernel, provided it comes with copies of all its libraries.

> The macOS Libc is called libSystem. It is available on all macOS systems at /usr/lib/libSystem.B.dylib. This library is the main interface that binary compatibility is maintained in macOS. Unlike Linux, macOS maintains a stable interface in libSystem that all executables are expected to link to. This interface is guaranteed by Apple to be stable between versions.

> In Nixpkgs, we maintain this compatibility through a list of symbols that are exported by libSystem. This is a simple text list and is available for viewing at NixOS/nixpkgs/pkgs/os-specific/darwin/apple-source-releases/Libsystem/system_c_symbols. The symbol list is created by listing symbols (nm) on the minimum macOS version that we support (for my PR, 10.12). We do some linking tricks to ensure that everything that we build in Nixpkgs only contains those symbols. This means that we can reproducibly build on newer versions of macOS, while maintaining compatibility with older macOS versions. Unfortunately, newer symbols introduced in later versions cannot be used even on systems that have those symbols.

> A side effect of macOS design, is that fully static executables are not supported in macOS as they are on Linux. Without a stable syscall interface, there is nothing to provide compatibility between versions. As a result, Apple does not support this type of linking.

So Nixpkgs does do some things to try to let you peg builds only to what's available in older versions of the macOS libc, but it doesn't try to let you bring your own libc.

There's at least one libc that does claim this kind of portability of course, and is the subject of lots of good posts here on HN. Maybe the author of Cosmopolitan libc ( https://justine.lol/cosmopolitan/index.html ) might have something insightful to say here.

Yes I agree it's an Apple-specific issue, and not relevant for what I care about in Guix / Nix

(Although Oil will obviously run on OS X because it uses plain libc from source, as is officially supported)

Cosmopolitan libc does seem interesting, because a shell basically only depends on libc.

It would be interesting to get Oil to run like that, on platforms where fork() is supported (not bare metal I assume).

Would it work on Windows? How do they do fork() emulation? I think that's very hard on Windows

Oh yeah it looks like it's slow on Windows (and OpenBSD)

https://justine.lol/cosmopolitan/functions.html

So I don't think it would make sense as a path for portability for a "production" shell

I think performance expectations for shells on Windows are pretty different. The first time I open PowerShell in a given session on my work computer, it takes ~9 seconds on average, without a particularly complex config.

That said, I think it can also makes sense to aim higher in terms of performance at the cost of a more bespoke Windows implementation. lol

As long as you don't hardcode "expected paths" (like in the FHS) in the filesystem, and expose the expectations you have via some configuration (environment variables and flags are fine), then packaging your software for Nix is pretty straightforward.

Want to try creating it? I posted a comparison with Gitpod here:

https://lobste.rs/s/jiyetr/hacking_anything_with_gnu_guix#c_...

https://github.com/oilshell/oil/blob/master/README.md -- click "Contribute with Gitpod"

I'm not extremely competent in low-level SW engineering, so would anyone be willing to enlighten how all these shells and environment isolation techniques work?

Even Python's virtual environment mechanisms are opaque to me and guix seems to be able to basically do that - but for anything you can think of (?).

In its most primitive use case "guix shell" sets environment variables and doesn't isolate. With "--pure" it also unsets existing variables. With "--container" it unshares a bunch of user namespaces so that the process has a different view on the file system (and others).

User namespaces are a Linux kernel feature. You can learn more about them from the man pages: https://www.man7.org/linux/man-pages/man7/user_namespaces.7....

Simplified: In *nix environments primarily, many programs have similar building blocks. These building blocks are often separate projects and are managed as dependencies, which may in turn have their own dependencies. You end up with a tree or dependencies for each program, where parts in the tree may be the same for different programs. Only one copy of each part used in all trees is kept on the system so any program that needs it has it available. Usually all these libraries (dependencies) are kept in a few common locations on the system, and done global system settings list what those locations are. If you look you'll find a bunch of libraries all thrown into the same directory with one another as a result.

But what happens when program A needs a library X version between 1.0 and 1.9, and program B needs library X version between 2.0 and 5.0? Both programs assume the system will have only one version on library X, and need it to meet their version requirements. Enter Guix (and it's cousin Nix).

With Guix, instead of the single global setting listing a couple directories to find all libraries in, a unique environment is crafted for running program A where a large number of folders appear to be listed in the setting, each including exactly one version of one library. In parallel a different unique environment is crafted for program B in the same way. But in A's environment the folder for library X is the one with library X version 1.5, and in B's environment the folder for library X is the one for version 3.2, but they might both use the same folder for library Y version 0.6. This may seem obvious, but it's not as easy as this all sounds. There are actually a lot of global settings for where to find dependencies, some of them have to match each other, and which ones get used depends on what the programs being run are. Also there's more at play then just the version number of the library, what optional parts are included also plays a role. This gives a huge number of possible variations of packages, not all of which are easily summarized by something as simple as a version number. Therefore Guix had to have some way of consistently referring to a package that used a specific version of code, included specific options, and a while list of other factors that affect what results (called a "build closure"). How do you make sure you actually found everything that affects the build result though? You run it again under conditions that shouldn't change it and confirm it's the same output. Successfully doing this is called "deterministic builds", and allows you to conclusively state that a list of these inputs (the build closure) will always produce exactly these outputs. This may involve doing things like making the build of the package always think it's 1970-01-01 00:00:00 UTC+0 so builds that embed a timestamp always have the same timestamp, which is why Guix has had to create a large list of packages you need to pull from. One you have these deterministic builds, you can ask for a package via build closure, Guix can download (or locally build and populate) the package into your cache, and every subsequent request in a Guix-managed she'll used to run a program can point to the library folder in the cache.

Making this even more difficult, building code has it's own dependencies added to the ones necessary for running it ("run time" vs "build time" dependencies). For compiled code this can be tools like the compiler used for example.

When trying to figure out what's in the build closure, it can be very difficult. One way to make sure you really do control everything is to isolate it. That's what containers do.

The article is about having the correct environment set up - attacking the problem from the environment level.

When I think of "hacking code you didn't write" one of the biggest hurdles for me is "inserting my changes to the behavior of the code with minimal refactoring of the original" - attacking the problem from, perhaps, compiler instrumentation level.

e.g. suppose when I use some library function f, there is a deeply nested function, g, that has access to some data x, which is cleaned up somewhere before the API returns its result.

How do I tell my compiler "I want a new function h, which, when given the same arguments as f, returns x, by returning from within g" ?

This problem is trivial within emacs thanks to the advice system, and something I've always missed in literally any other programming environment. In Guile, I bet `(parameterize)` could be used to similar ends, but you don't get to choose the language of every project you hack on.

Can anyone comment on how this compares to apt-get source? (Optionally run inside docker?)

AFAIK that won't install all the necessary dependencies, right? And even if it does, these are installed globally. Guix shell will only make the dependencies visible while the shell is active, and "hide" them again as soon as the process terminates.

Not an apt user but my quick search suggests this just downloads the source of the package? The guix shell --development command in this article is for getting all the dependencies needed to build the package (but not the source itself, though you can do that through other guix commands I believe). In other words, with guix shell you can now run make or whatever you need to build the package in question, without needing to fetch the development tools, other libraries, set env variables, etc.

Edit: for the source of a package guix build --source thepackage will return the path to the source (as stored in the store). This includes any patches or transformations (e.g. you can pass patches to be included or a git/branch/commit location to pull the source from instead of what is defined in the package definition)

It's more like `apt build-dep`, but you don't install all the development libraries and headers to your system in a global, persistent way. Instead you just make them available for that shell session.

It's likely a lot smaller than a Docker container that contains a whole Ubuntu runtime, and it'll automatically share a cache of deoendencies with any Guix packaged you may have installed locally. It also has normal/native access to your filesystem (and thus your dotfiles and your homedir), since it doesn't live in a container.

xmake [1] gives you something like this via `xrepo env shell`.

[1] https://xrepo.xmake.io/

thank you hn! this is a such a gem find

Reply on news.ycombinator.com