NixOS RFC 136 approved: A plan to stabilize the new CLI and Flakes incrementally
github.com
github.com
Building third party flakes takes forever since every flake uses its own version of nixpkgs. If you don't pin your third party flake urls they might also change under your nose as you run the same command again a day later.
Flakes are coupled to git. You need to remember to stage changes whenever you do Ctrl+S in your editor before rebuilding. I've wasted more time than I'd like to admit wondering why what I'm building doesn't include my latest change.
Flakes copy the entire directory into the nix store. This is terrible for mono-repos - and especially if it contains any large files. There is an issue about making the copying of the entire directory into /nix/store lazy - though this doesn't really address the problem. It just treats some common symptoms.
This kind of copying would likely be a lot more acceptable if it was specified explicitly in flake.nix, rather than implicitly. Then you could also use your own filters for ignoring files, rather than relying on the coupling to git and its staging area.
Could you not just do a sparse checkout for monorepos?
Let's say I'm working on a huge monorepo locally, with GBs of code and data in it, most of which I want to ignore in my nix build. Let's say the files I actually care about are also not in a single subfolder. Then I can't just move the flake.nix to that subfolder either.
definitely not ideal and I agree that until lazy file mapping, content addressed derivations, and file deduplication get implemented that the UX will be rough in some cases.
You're probably already aware, but for the benefit of newcomers in this thread, here's a link to the massive PR trying to work out more efficient copying of local flakes to the Nix store:
https://github.com/NixOS/nix/pull/6530
A few dedicated folks are working really hard to break that thing down and test it and get it merged. I can hardly wait!
The “official” (as in, community consensus) solution seems to be to bring your own nixpkgs, and manually override the nixpkgs of every dependency (and their transitive dendencies) and verify your build still works. Which just doesn’t seem right to me.
Flakes are great for building an ecosystem of composable, reproducable software. The more they succeed at that, the bigger the problem becomes. Surely it’s not too late to figure out a principled solution before stabilizing flakes?
The way I would do it with nix is: because everything gets hashed, just check that the built output with a new nix-pkg version matches the hash with the old version and fail out/fetch the older version as a second step. The rest is just finding ways to cache that build fail information so that you can share the result across users transparently.
That way you get the benefit of a minimal-sized system with no size overhead for the most part, yet out-of-tree git repos get reproducible at any commit, and it is even easier than adding a new repo to apt, etc.
With content-addressable hashing, the problem may largely solve itself, reusing unchanged dependencies between older flakes.
Works well for me. Though when doing configuration changes I usually pass --no-substitutes (or smth) to not query all my subs for derivations that'll never be in a remote cache.
This is decidely not quite true. You must have the file visible to Git. Flakes still just uses the files, as long as they're tracked by git. As long as you have no new files that aren't in the index, staging unstaged changes will not change eval.
That said, yes, when you add a new file to a flakes repo, you start to develop a habit of always adding it to the stage/index for eval.
but you should ONLY ever see this as a very very obvious "file not found", it is not a subtle problem as this post implies.
Given that the Git thing is meant to stop you from shooting yourself in the foot, I think maybe it's time to conclude it wasn't really that great of an idea. However, I still see why it works that way.
That's not quite right. It may use a predefined version of nixpkgs and there may be a reason the version is hardcoded to something. But if that's not what you want, you can use something like `nix run ... --override-input nixpkgs nixpkgs` and it will run with your current system nixpkgs version instead.
then what's the point of using nix and flakes if flakes destroy the very point of nix, reproducibility?
You only need to `git add` the file once.
You do not need to stage every change after that before flakes can "see" the change.
Does the same workplace not allow people to include config related to their editors and so on?
There's an upside to this: I've been in situations where I've lost the definition of a previous configuration because I didn't check the state of it at that point into source control. By "forcing" the user (me) to at least add the current state to source control (if not commit it), it makes it more likely for me to have the configuration/definition for the state saved at the same time as the state itself.
That's what flake.lock is for, it remembers the versions picked. No need to put versions in the flake urls just for that.
If I have an arch system now, how useful is it to use Nix or Homemanager on a non-nixos linux system? I dont really want to dump a working system and duel booting always has whatever you need on the other system.
Do you get anything when not using NixOS to manage everything? What good is it if half my system is installed by pacman, half by asdf-per-project and the rest by nix? Has anyone gone down this route? What is the actual lived experience of half-assing it?
I would say that the biggest benefit you get is the hugeness and currentness of nixpkgs and the fact that it becomes a lot less important what software you have "installed".
Honestly, it’s Home Manager that really adds the maximum value, and getting all the way there takes a full weekend, but what were you gonna do anyway?
Indeed. As no less than Ron Swanson rendered unto us - an exemplar in classic American yarn form:
... wise words.
I feel like Nix does have such killer features, but they are different for everyone.
For me it was:
- system management (nixos + nix-darwin. I wish there was something like nix-darwin for non-nixos linux distros, binding daemon systemd units and desktop files into the "hosting" distro)
- on-the-spot running without globally installing for one shot tasks (nix-shell -p iotop --run iotop)
- "just works" project environment (default.nix+shell.nix or flakes)
These are the ones that made me tip over and decide to go from tiptoeing to full tilt
Another benefit is that "traditional" package managers may litter your system with configuration files and post-install scripts (for the software you are installing and all of its dependencies), some of which can persist even after uninstalling the packages.
Nix prevents that by design.
It's fire and forget and I can just focus on what I'm set to do.
Once I'm done with the one shot task, it disappears and could be garbage collected for all I care.
Some tools I did promote to being part of my user env but crossing that bar is a very deliberate thing, some are promoted to systemwide (root) and the bar for that is even higher.
It removes a ton of mental deadweight.
It's in early stages, but it exists now: https://github.com/numtide/system-manager
go outside, talk to friends, do something productive, etc...
Fully reproducible environments with home manager and nixos are neat but IMHO you can just use ansible to do the same thing on debian, arch, etc. if those systems have the packages you need.
When you inevitably find software that isn't in nix (or are packaging your own software), be prepared to take a quantum leap in complexity as you now need to fully understand the nix language and its concepts like derivations. This isn't necessarily hard but it is not documented well at all, and there are factions in nix at odds with each other (flakes vs. nix-env) which make learning it even more confusing.
I think this is where most of the "i went back" posts I read came from. That and hitting issues with programs that expect to be able to write to some XDG dirs.
Exactly this was why I am so hesitant to switch to nix. I tried to dig into the details of nix, language, derivation and whatnot, but documentations and community discussions are, in my experience pretty segregated that left me confused, and I eventually went back to arch.
Though I will give you that there is enough complexity in specifying which nixpkgs/flake gets built, where is it places, etc as is, especially with the `nix-\w+` vs `nix \w+` variants being slightly different, to the point that I always have to look it up in the man page/in my history whether I need -A and the like.
I've used Ansible for years, then moved to Nix and have been on it for a year and a half. Yes the configuration language is more complex than Ansible's but it's also way more consistent and predictable. The bit of complexity comes with a lot of additional benefit. Ansible half-asses it, only giving you idempotency if you stay within certain boundaries in how you write configs. And a leaky system is a system that will break. Making it even worse is that you are building on top of a moving target with a distribution underneath that will break your Ansible scripts regularly as the system changes. Ansible is a huge mess and not suitable for the job IMHO.
With NixOS, the entire system is controlled by config, so there is only one source of truth for the state of the system, unlike Ansible. This is a fundamental flaw. Ansible is a hack on top of pre-existing setups.
With Nix, my system only gets more stable with time. It's the opposite of anything I've ever used, where systems get crufty and have to be reinstalled every couple years. Since the state of the system is fully determined by your config, you can always get to an exact state without a bunch of extra shit missing or added.
And the language isn't really that hard for anyone who's done a bit of programming. It's certainly easier than most general purpose languages, as it's static data 90% of the time with a bit of logic thrown in when needed.
That'll depend if you're on stable or not. If you're on stable and you need something from unstable that's also possible without any real risk of breakage on NixOS.
It's hard to tell from how you say it; if you aren't using it for any projects and would just be using it for a slice of your system/user packages, I suspect you'll end up not really seeing what the fuss is about. If you will try to use it with some projects, I'd start there and see how it goes.
(It should be fairly low stakes. If it doesn't stick, you won't disrupt yourself that much. If it does stick, you'll have a better sense of what you want to hand over to it and why.)
Is it possible with complex projects like based on Electron? Is it reasonable to use Nix in Docker (eg. I am on Mac but want to build it like it's on CI)?
Combining nix with direnv is a popular combo for dev environments. Example: https://determinate.systems/posts/nix-direnv
If you haven't worked with nix before, the quick start here is a good place to start https://zero-to-nix.com/
Not sure I can boil it down to tips, but: more or less any time I need to do something regularly that involves third-party code, I'll be using Nix to manage it. Specifics depends on the project. There are templates around (and templating is built into flakes via `nix flake init`), but I haven't used those much myself.
Maybe a sense of general patterns helps?
Most of my projects have at least a shell.nix (devShell expression in flake.nix) for setting up an environment to work on the project. I.e., add appropriate deps to path, define some utility shell functions, start up database servers and so on.
Sometimes that's all. For example, my blog just has a shell.nix for providing dependencies like hugo, gist, make, jq, etc., defining some helper functions for scaffolding new posts and such, and then I use Makefile targets for creating a dev build, production build, and publishing.
If it's something others might use, I'll generally have a default.nix/flake expression for building the package. Then I can use that for testing it, for integrating it in other projects or my own systems or whatever.
I'll often have one or more expressions for ~testing it. (If I have more than one, the first might be a straight test, and the 2nd might be a performance suite comparing it against similar tools, for example.)
If I'm researching several potential alternatives, I may build a Nix expression that builds all of them, pulls them all into scope, and then invokes them all to compare whatever I care about. Then I can readily repeat the same check locally, or go run it a bunch in CI.
If it's something I have CI set up for, I may bother to have a ~normal build that runs against pinned dependency versions whenever I make code changes, and then also set up a ~canary build that regularly re-runs my project's main/master against the latest nixpkgs so that I get some notice if something I depend on happens to be broken.
> Is it possible with complex projects like based on Electron?
I'm not familiar with the Electron packaging, sorry. I know it's doable because we've got a number of electron packages, but I don't have perspective on how easy it is and whether it's working on Linux and macOS or just the former.
> Is it reasonable to use Nix in Docker (eg. I am on Mac but want to build it like it's on CI)?
Sure. There's an official Nix container image, and some of our gitlab CI jobs use it. IIRC it's a little tricky to install Nix in a container imperatively, but I think the state of that has improved since the last time I needed to fiddle with our setup.
For me, the last one is a gamechanger, as I am no longer worried about littering my system with tons of dependencies. Say I want to convert an eBook I downloaded. I can quickly use Calibre for a one-off thing, with nix run nixpkgs#calibre.
In fact, I've departed from vanilla Silverblue, as now my base image is an actual, bona-fide Docker container as well.
I feel like the immutable Linux OS is the pragmatic approach, whereas NixOS is the pure, ideal way that requires a lot of changes to tools and workflow to work. Maybe even in this case, worse is better.
it's like having your whole OS in git. it's amazing. want to try out some nightly build? no problem.
on most computers (personal, home computers) the various lib dirs get scary after a while. repoquery can help, but is kind of a hassle. if some executable `foo` depends on some specific behavior in some version of some library, it's fine. that weird library can stay in place, I can try out nightly's of foo, can always rollback to the system that used to work, and can flip to the new foo when it's ready.
I think it's a good practice to wipe and re provision from time to time to make sure provisioning and backups actually work, but like, you can screw around with _crazy_ things and not really have to worry about having to rebuild the whole thing. (well, I haven't yet) I still will, because that's how I am. but I can do it when I feel like it, not when I fucked up. it's so easy to un fuck a nix system.
I guess the big caveat is, this is a hobby. I haven't used nix for work. Back in the day, I did support docker in production when the kernel patches were coming pretty quickly. Finding the right balance of patch level, and libc version, that would support docker, and the other stuff that needed to run was a pretty complicated dance, and missing a step would be, not great. Knowing what I know now, even as a hobbyist, I probably could have made my life a lot easier back then.
Now I have 2 years old NixOS installs on personal machines and they do not get bloated with packages that the package manager can no longer handle, and is basically always at a state as if it were freshly installed. It just “feels right”. To me, that makes the occasional edge cases worthwhile to suffer through (some binary package, mostly) - especially that in most cases something like steam-run is enough to make it work. When it isn’t, systemd-nspawn can boot you into any systemd distro you would want.
[0] - https://github.com/benpye/openocd/blob/mrs-2023-08-07/defaul...
Furthermore, Docker images are monoliths that's isolated from the system. Instead of system administrators choosing the isolation boundaries that best fits their requirements, the image packagers put walls around each and every image. As a result, Docker containers are inflexible compared to traditional Unix applications. Piecing together multiple Docker containers is a clunky experience.
In contrast, Nix is an integrated build/configuration/deployment system. Entire systems down to the package level can be reproduced from Nix expressions.
It also gives system administrators power over the whole process. Administrators have control over things like how each components are pieced together, and where isolation boundaries are placed. They can even make tweaks to the package build steps.
Docker can't substitute Nix. It's not a matter of elegance. Their features and capabilities are wildly different.
> Dockerfiles used to create Docker images don't have any mechanism to ensure reproducibility whatsoever
simply generate it once and keep it as an immutable artefact
Sure, I think the biggest benefit is being able to get a temporary environment for a project. You can have all of your dependencies pinned, even for languages that don't have package managers, and for mixed-language projects. You can get easily get a shell that contains the exact compiler versions, libraries, etc. from a single config file, and it's isolated from the rest of the system. I guess it's kind of like having a lighter-weight container.
With nix instead of a tarball being the root filesystem it's a bunch of symlinks to stuff in the nix store directory--the process you run is exactly the same as whatever you're running in docker.
One crucial difference is that nix does nothing for network isolation, you have to do all that on your own (if you need it).
That's a feature. I don't normally want any of those.
Just to be clear, Cygwin is a great project. But it's not something I'd actively choose over a native Linux system if I had the choice of operating systems.
Yeah, for sure! I don't use Nix this way very much on Linux because I mostly just use NixOS. But I use Nix this way on macOS at work every day, because that's the Unix desktop that's available to engineers there.
By not going full NixOS, the tradeoff is that you suffer some minor integration issues, and in return you get some more escape hatches. This is a pretty good option for some people imo.
Declarative system config on non-NixOS is a bit more work, because you either have to write your own sort of metapackages with buildEnv¹ or use a third-party module system like Home Manager, System Manager (that one's in very early stages, so pronably skip it if you're a newbie), or Nix-Darwin. The latter are really nice, but you might have to think about how they plug into things like your OS's bashrc file sometimes, and sometimes their install procedures are kinda wonky.
The other main integration quirk to watch out for on non-NixOS is 3D graphics acceleration. Check out nixGL and you'll be covered for that.
Using Nix for nifty per-project development environments, though, is pretty smooth and basically identical whether you're on NixOS or not. If that's an interesting use case for you, Nix on non-NixOS is worth checking out.
Nix as a package manager for CLI tools that don't depend on the base system, or trying out software without permanently installing it, is also kinda nice on other distros. It's especially nice for running newer software on top of stale distros, but it can also take you in the other direction, which might sometimes be useful on Arch.
I think the head first approach of just committing to daily driving NixOS can be good for some people, and it certainly was for me. But learning your way around the Nix language and developer tools by focusing on developer environments and oddball packages on a foreign OS can definitely make for an easier, lower stakes learning process.
> What good is it if half my system is installed by pacman, half by asdf-per-project and the rest by nix?
If you're not using Nix for any per-project stuff, you're definitely missing some value, as that's likely among the first low-hanging fruit you'll encounter on your Nix journey.
That said, stacking Nix with other package managers and only using it where it makes most sense is totally viable. On macOS, I use it in combination with light use of pkgsrc and Homebrew. You can definitely do the same kind of thing on Linux.
targets.genericLinux.enable = true;
xdg.mime.enable = true;
xdg.systemDirs.data =
[ "${config.home.homeDirectory}/.nix-profile/share/applications" ];My config/flake is cleaved in two: one for NixOS (which isn't used for Ubuntu) and one for "Generic Linux" (which is used everywhere). I used to have nix-darwin in there too, but Intune on Ubuntu has allowed me to retire the work Mac.
The second-half is where the real value lies (my NixOS config is actually really minimal, just enough to get into sway). I use home-manager to manage all my dotfiles. I switched work laptops recently and nix+home-manager brought up the environment in less time than it took to install Ubuntu - and that's a full desktop replica (Gnome plugins, hotkeys, installed apps, nvim config, and everything).
The next big benefit is direnv integration. Direnv itself is bloody fantastic, but add nix to the mix and you have a whole new level of magic. If there is an in-tree flake I can just echo `use flake` to establish a dev environment with all dependencies, at the right version, through the mere act of cd'ing into that dir. If there's no in-tree flake (as is the case with my work repos), I can use one of my dev-shells with e.g. `use flake $HOME/Nix#cmg-js` (assuming I've cloned my config to ~/Nix).
There's no real reason for me to use NixOS on my own machine, but I do love the boot speed and the ability to roll back. I'm pretty fearless when changing my OS config. The nature of Nix means very little beyond what you explicitly demand is installed, and it keeps snapshots of your previous configs. It's not mandatory at all, though. Home-manager is where it shines.
100%, I recommend people to start with home-manager before NixOS as it's easier to escape the Nix when you're out of depth than if you're on NixOS. It'll manage your user systemd units and everything else within your user, it's great.
1) There is more to learn now.
2) Split ecosystem.
I'm looking to be educated, I've made contributions to nixpkgs (two "mid-sized" changes) but for some reason I couldn't bring myself to learn flakes ... Although that could have been because they were in limbo.
I have to say though, nix is increasingly becoming a nerd thing philosophizing over minute issues rather than "getting shit done" to benefit the maximum number of users ... :-(
I'd point to the home-manager installation instructions for an example of Nix flakes having a nicer UX. Running flake-enabled codebases is just much nicer than non-flake codebases.
https://nix-community.github.io/home-manager/index.html#sec-...
https://nix-community.github.io/home-manager/index.html#sec-...
The former involves running `nix-channel --add <...> home-manager && nix-channel --update` in order for `<home-manager>` to mean anything. With the latter, it's simply `nix run home-manager/release-23.05 -- init --switch`. (Albeit, `home-manager` is already registered as an alias for `github:nix-community/home-manager`).
One key detail is that `nix-channel --add` modifies the 'system state', whereas running flakes essentially does not.
-- Regarding "you can pull in arbitrary Nix code by reference". With flakes, you have a standard entry point where you know the interface it has. Whereas, having a 'default.nix' is a convention, and there's no way to know how a `default.nix` is to be used without reading its source.
Speaking of vim, I've got neovim configured completely with home manager. It references plugins as first-class nix packages, and also any external dependencies like language servers, node packages, parsing tools etc are also managed by Nix. You can reference the deps in the vim config and it will automatically download and point to them in the nix store without even installing them.
It's amazing. Everytime I upgrade and building my system, my nvim setup gets upgraded automatically too. No need for a package manager or manual updates.
Nix is perfectly capable of referencing external text files (e.g. `./my-config-file.ini`), which you can write with vim or whatever if you like.
It's trivial to put such files in place with Home Manager (or NixOS config) if you want, e.g.
home.file.foo.source = ./my-config-file.ini;
I wonder what we can do about this, or what is the best way to annotate long strings in Nix code that are actually source snippets. Something like how code snippets are handled in Org files would be awesome.
I consider lock files to be labour-intensive work-arounds for fundamental design problems. One of the things I like about Nix is that its design doesn't need them, and even allows me to automate-away their need in other tools (e.g. defining a project's "main" derivations using import-from-derivation on the result of a lock-file-generating derivation; this works well with legacy tools like Maven, for example)
But the community does seem to be still tinkering focused.
I miss the early 2000s when even the hackers seemed to want to make their stuff easy to use(although my source for that is I saw a screenshot of low orbit ion cannon one time, I don't actually know much about hacker history).
Because Nix does NOT make packaging easier. Maintainers are the ones who create distros, and they can live a happy life with something as crude as PKGBUILD, since build instructions are almost always stable.
Even for users, most of them just don't need to juggle with package versions. Bleeding edge is mostly covered by Arch and package pinning works ridiculously well for daily uses.
So "make their stuff easy to use" doesn't apply here. Rather, Nix is harder to use in reality.
A few months ago Steam stopped working. Nobody seemed to know why. It's back now, but git-cola is gone and I'm not even sure how I broke it. Some nonsense about type errors in the hotkey module, which I don't have time to debug right now so I'm just ignoring and using the flatpak.
Most everything I use on a daily basis has 100s of MBs of dependencies in several layers... Forgetting the virtualeng when pip installing something that happens to have a PyQT dependency and trashing my git cola is... not great and Linux is full of stuff like this.
The amount of effort needed to avoid or fix this kind trouble is more that I imagine Nix would need if it was fully polished.
On Nix I was able to package a Python app in a matter of hours, and
How do you come to that conclusion?
Those who switch to using flakes are enthusiastic about the improved UX.
> flakes is that it is too much added complexity for tiny incremental gains
A 'flake' is more/less a set of inputs and a set of outputs. I'd say it's less complicated than a package.json file in an NPM project.
The benefits are hard to get otherwise.
With flakes, `flake.nix` provides a standard entrypoint to a Nix codebase. Without flakes, you might see a `default.nix`, but would have to read the `default.nix` to know how it's supposed to be called.
With flakes, the exact version of Nixpkgs used is locked. Without flakes, I'm not aware of an easy way to do this with channels; and there are a variety of ways (read: no standard way) to do this in Nix code.
- flake-specific state is fully captured and locked
- this locking applies for all platforms the flake targets
- extreme ability to be composed
The first two solve the same issue as Gemfile.lock, its the PLATFORMS section, and the platform annotations on each of its gem line, which allows `bundle lock` to lock not just for the platform it's being run on but all platforms that are targeted.
It also canonicalizes the way to declare targets to build, whereas previously it was kind of TIMTOWTDI (e.g default.nix evaluation is the default target and calling nix-build sans arguments builds that to `result`, and if you want more you write alternative nix files, the whole organisation of that is on you)
These are very useful for project-oriented nix stuff.
For system (nixos, nix-darwin)/environment/profile/home management I still prefer classic channels, which gives me a single, consistent nixpkgs source, kind of like BSD's base which is handled as a whole (except here it applies to the whole of packages). I am still able to locally override this or that by selectively pulling another nixpkgs (or other source) and use that to pick a specific package version (either forward or backward) but otherwise I stick to the single consistent one.
It feels to me that using flakes for the whole system would be like using npm and having a whole recursive tree of `node_modules`. I certainly don't want to have a thousand different openssl around!
It's conceptually comparably complicated, but the actual practical experience of writing flakes is much more complicated. This is not the fault of flakes specifically, but rather due to the complexity of nixpkgs. Although on second thoughts the fact that there are a bunch of libraries like flake-utils and flake-parts out there does seem to point at a verbosity UX issue.
FWIW I like Nixlang, but I agree that static typing would improve it. Maybe Nickel some day, blah blah blah :)
This problem seems to be solved commonly in two ways:
- People thoughtlessly and senselessly listing only x86;
- People using the popular flake utility repository.
Having to rely on an unendorsed third party just to write architecture-independent code isn't very appealing to me. This seems like a serious deficiency in the design of Flakes, though it would be tolerable at least if this functionality was at least in Nixpkgs.
allSystems = [
"x86_64-linux"
"aarch64-linux"
"x86_64-darwin"
"aarch64-darwin"
];
forAllSystems = f: nixpkgs.lib.genAttrs allSystems (system: f {
pkgs = import nixpkgs { inherit system; };
});
Well, we're talking about evaluating a pure function (literally the entire philosophy of Nix is treating a build like a pure function). Those things are variable state as an input which affects the output, and therefore must be specified as inputs. I don't see the problem here (if you have already accepted that you want to treat builds as pure functions with total encapsulation of the state required to run them, and thus having deterministic outcomes).
Perhaps you haven't dealt with enough Python packages that make assumptions about the availability and state/version of underlying OS-level libraries (and Python itself: See the transition from versions 2 to 3 as an example of this chaos) to understand how important it is to control for assumptions.
What you gain in convenience today, you pay tenfold for as "interest" later (this is thus definitely one aspect of the "tech debt" problem).
One reason for doing this would be for when $DISTRO declares that they will now support a new architecture. What packages would then be expected to require patches? If existing packages were marked as 'all', those can reasonably be expected to succeed and should cause noisy errors if they don't. Conversely, packages which were explicitly marked for specific architectures (even if that is all the architectures that the distribution supported when they were created) can be expected not to work on the new architecture without modification. This does of course require the packager to actually know the difference between using the keywords and just marking every known architecture. And to be fair, what I was originally advocating for in my comment was more like Debian's 'any' wildcard, which is not quite as strong as the assertion that the package is completely architecture-independent.
I would say that a specific tag for architecture-independent builds actually improves the situation: explicitly acknowledging the assumption that exotic architectures will never be an issue. Of course, as your sibling comment mentions, the interpreter is still part of the closure in Nix, so maybe this is incompatible with the existing choices made in the Nix ecosystem.
Nothing is ultimately pure since you're running on top of a kernel which can affect computation. It's unclear to me what practical gain you get from trying to obsessively get "maximum purity", which is in any case illusory because you're still running on a kernel from outside the closure.
Making it impossible to write portable code isn't a good design.
> obsessively get "maximum purity", which is in any case illusory
I don't understand this argument. Should I stop using functional languages because a cosmic ray could potentially mutate a variable that I expect to be immutable? Could it be that expecting perfect purity is in fact the enemy of "good" in this case? The fact that I've almost universally had better experiences in languages that emphasize "purity" (despite running on "uncertain" underpinnings) seems to undermine this reasoning.
Or in JavaScript, you need left-pad to align text to the right, which everyone and their uncle depends on.
Nix is a programming language, where there are holes people will plug them with libraries, what's weird?
Can someone with more background give a quick explanation about why is this a source of tension in the nix community in the first place? Is this a situation of too many cooks in the kitchen or something else?
Some aspects of Flakes are a bit contentious. For example, official adoption of Flakes will mean Nix will no longer be a mono-repository. Plus, Flakes are tightly coupled to Git.
However, Flakes bring welcome improvements to longstanding issues. Most notably, package pinning and explicit versioning, which are problematic when using channels.
> Flakes are tightly coupled to Git.
How is it tightly coupled to Git if it supports Mercurial out of the box?
Other than using version control to determine what are the sources of the derivation, what does flakes lean on version control?
> Other than using version control to determine what are the sources of the derivation
I don't think that defaulting to using git to list the sources when inside a git repo means that it is tightly coupled to git. If one doesn't want to commit/stage the flake files in the repo one can do `nix develop .` and it works . It will take a little longer because it will copy the entire directory to the store iiuc.
Do you mean nixpkgs? I don't think stabilizing flakes means nixpkgs won't remain a monorepo.
I'm still not convinced by flakes; however, I agree that channels are problematic (and I still don't fully understand them, after a decade!). The first things I do after installing Nix are to delete all the channels and unset `NIX_PATH`. I used to use `<nixpkgs>` for bootstrapping via `fetchgit`, but even that's unnecessary now that `fetchGit` is part of `builtins` :)
You're right that `builtins.fetchTarball` is faster than `builtins.fetchGit` (due to the ridiculous amount of commits in the Nixpkgs repo). I like to keep such definitions in a single, company-wide/project-agnostic git repo (what the Nix Pills series calls the "repository pattern"), and have individual projects import them via `builtins.fetchGit`.
Many years ago we didn't have `builtins.fetchGit`, so had to use the 'fetchgit' function from Nixpkgs instead. That created a chicken-and-egg situation if we wanted to take the Nixpkgs version from some other git repo; hence needing to "bootstrap" via `(import <nixpkgs> { config = {}; }).fetchgit`, and cross our fingers that `NIX_PATH` wasn't set to some crazy value (which, of course, I would inevitably do... https://github.com/Warbo/haskell-te/blob/24475a229908caa3447... )
Note that we need `config = {};` when importing Nixpkgs to avoid an impurity which tries to read files in $HOME. More recent versions of Nixpkgs also need `overlays = [];` to avoid another impurity (looks like this changed at Nixpkgs 17.03, according to https://github.com/Warbo/nix-helpers/blob/master/nixpkgs.nix )
Flakes were a RFC then merged as an experimental feature and shilled too much to the community to the point they are now a quasi standardized feature even though they didn't go through RFC and therefore ignored all the valuable feedback.
This whole debacle made a lot of invested people tired on both sides.
I don't agree at all. I use flakes with cross compilation literally all of the time. Having a concept of "cross-compilation" at the CLI level, instead of being part of the nix evaluation itself would be a travesty IMO. I love being able to do a single nix eval that includes 3 different architecture of native builds for my host toplevels, along with "cross" variants that set hostPlatform to x86 and produce cross-builds.
I love it, only thing I would love if there was a proper way for passing arguments to flakes for build /dev/run only dependencies and other ways to specialize the scope of the user using the flake and infrastructure attached to it :)
A friend of mine called it a "rabbit hole" but TBH it's words of power -- I can speak new instances of my environment into existence, not only on newly-acquired compute devices, but on other OSes as well (I put some effort into segmenting the user environment and system config between home-manager and standard /etc/nixos/*)
Nix flakes will only make me more powerful
You're not kidding. I used nixos-anywhere [1] for the first time the other day. You can point it to any linux host and it will partition, format, and supplant the target's existing OS with your nixOS config, all with a single command. Incredible.
The plan itself seems very well thought out. I had a vague desire for an incremental stabilization of flakes some time ago, but this is so much better.
The flakes divide is one of the biggest remaining problemd facing the community, now that the issues with the release cycle have improved so much and continue to improve. Each step taken in executing this plan will be a real boon for the Nix community.
Onward!
It's fine, Nix things are conceptually different from anything before.
You could say flakes are like Rakefile, but that would be a lie. You could say they're like Gemfile and Gemfile.lock, but that would be a lie too.
You could say that derivations are like packages, but that would be a lie, you could say they're like container images but that would be a lie too.
Now, you can try to understand what these concepts are by mapping to known knowledge of other things in terms of use cases they solve, that's fine, but at their core these concepts are different and quite unique, so it only makes sense for them to have different names.
Naming is hard, Nix folks had to pick one for each concept, and they chose not to lie by picking existing names that would not map and create bias in the understanding and expectations. It makes for a steeper learning curve, but I'd argue long term in the learning process it's the right move.
Let me try to give some synthesised definitions (which may be incomplete and lies in the Feynman sense):
- a derivation is the reification (or realisation, as in, "make it real") of a function call on the filesystem. IOW the concrete result is derived from the function call, memoized using content addressing.
- a flake is a central definition of a number of target derivations in the context of a specific source state of dependencies. If you look at an actual (snow)flake there's this recursive/fractal design coming from a central point, which kind of visually represent the conceptual structure underpinning a nix flake.
So for me the names totally make sense! But I can understand that they don't out of the blue (and they did not for me at first!)
It turns out (not really by accident) that these cover use cases that were implemented by this or that tool, and instinctively through a partial understanding of the nix concepts we're in conflict: the other tools give us a (biased) understanding which both increases and is in conflict with our understanding of nix concepts. This is a normal process, but at some point by letting go of what we previously know and trying to understand new things for what they are and not what we project onto them, we open ourselves to fully grok, and both fully leverage the new concepts and realise their limitations.
Never thought about it that way, but it kind of makes sense. I thought the name was inspired by the Nix logo.
The nix logo is indeed a snowflake (made of lambdas!) I'd say it perfectly captures the core nature of nix.
The docs are much better at describing this as "a derivation is a build task." I don't care that it's the return value of a function call, I don't care that because the language is purely functional it may be called lazy and memoized and so on. They are interesting technologically, but the abstraction is bad if it is so leaky you need to understand the internals to use it.
> You could say that derivations are like packages, but that would be a lie,
> a flake is a central definition of a number of target derivations in the context of a specific source state of dependencies.
It's a package. More specifically, a package definition or description. People understand that the result of package installation depends on the total resolution of dependencies and this may be fairly fine grained or a difficult to parse graph (they just trust that the package manager does the "right" thing). Most modern package managers do this and it's not confusing to call it a package (although, it's in good company because every package manager likes to call their packages something cute and different).
I find the need to include additional libraries to make flakes usable for some reason a weird hurdle I don't personally enjoy (i.e. flake-parts or systems)
It's even more of a plus that it is aware of Git and makes rather surprisingly intelligent decisions when automatically upgrading the system on a schedule (as one example, it seems to be able to deal with when the Git repository has staged or unstaged changes just fine, producing commits with just the lockfile changes.)
You can also just look at the niv sources.json file pretty easily if you knew the URL. Rather trivial command to run.
I don't find the above compelling (or correct?)
What's the Niv equivalent of this?
# flake $ nix run github:fzakaria/mvn2nix#mvn2nix
At this point I recommend we take this through another medium like email or feel free to open up a discourse to help fill in the knowledge gap. Feel free to tag me on it.
But niv has been great, I wish we can have patching natively so all my tons of patches can be bolted on faster.
It also can't do transient overrides which is quite literally the essential reason that Flakes is killer to me.
For example, I can cross-compile to riscv64-linux, including a bunch of 3rd party flakes repos *specifically* because I refer to them and then override their nixpkgs to point to my nixpkgs with various cross-compilation fixes.
Definitely not possible to do this with niv.
It also does nothing to solve the countless issues with channels that I've watched afflict countless new Nix users, every week, for 3+ years. [insert the usual 8-stop troubleshoot list when a user has any issue possibly related to channels]
Flakes to me introduced a lot of overhead beyond what niv can provide.
I get 99% value just pinning my top level dependencies since most of the time I'm referring nixpkgs + 1-2 other packages.
Linode was nice enough to have an official "how to set up NixOS on Linode" guide, but it requires fiddling with partitions by hand. At that point, I feel like all of the repeatability gains are gone. If I want to spin up a fresh server, I have to read a guide and set stuff up by hand? Could spend that same time setting up an Ubuntu by hand each time followed by git pull or scp to deploy. Ended up using Ansible in the end, as it really does let me just get a new VPS and deploy to it very fast when the time comes.
I wonder why the big providers once supported NixOS and then decided to stop!
(see `virtualisation` folder for other providers)
Then you can do something like:
nix build .#nixosConfigurations.doImage.system.build.digitalOceanImage
Its probably best to also create separate configuration only for the image. As a bonus, its already configured, so no need to run `nixos-rebuild switch` after droplet creation. Partitioning configuration is also handled by the builded, however, filesystem choice (and partitioning schema) is limited
(I'm not saying that this is easier than out of the box support for NixOS: IMO if you are even aware how to create custom NixOS image then you are probably somewhat experienced NixOS user)
Sibling comment mentions that nix has ways to build NixOS VM images (e.g. https://github.com/nix-community/nixos-generators has a generator specifically for Linode).
Another option is to use nixos-infect, which will replace a Linux distribution with a NixOS distribution. https://github.com/elitak/nixos-infect -- e.g. this blogpost discusses using nixos-infect to deploy NixOS for a cloud provider which doesn't have an official NixOS image. https://xeiaso.net/blog/nix-flakes-terraform
> At that point, I feel like all of the repeatability gains are gone. If I want to spin up a fresh server, I have to read a guide and set stuff up by hand?
Just as with ansible, "git pull to deploy". You'd keep a copy of your NixOS configuration.nix somewhere else, and would be able to apply it later.
> it really does let me just get a new VPS and deploy to it very fast
Using a tool you're familiar with to get the job done is going to be faster than learning to use a tool you're unfamiliar with.
I believe much of the use of NixOS is for hobby stuff, and for personal machines. (c.f. "In what environments do you use Nix", Development (1242) vs Home Server (845) vs Production (386) https://discourse.nixos.org/t/2022-nix-survey-results/18983).
Never underestimate hobby usage ;)
> Using a tool you're familiar with to get the job done is going to be faster than learning to use a tool you're unfamiliar with.
I'm well aware of this and had the time to do some upfront investment. During that investment, it seemed hard to boil a fresh deploy down to a couple of clicks. Given the great responses I'm getting, that was probably wrong, but wasn't easy for me to figure out through simple web searches!
Nowadays you can declare partitioning instructions with Nix just like the rest of the system: https://github.com/nix-community/disko
And you can use something like nixos-anywhere with it for a alternative process, if your provider isn't as cool as Linode and won't let you boot arbitrary ISOs: https://github.com/numtide/nixos-anywhere
Building images directly for cloud vendors' platforms on your local system and then pushing them up and assigning them to VMs is probably nicer than actually installing NixOS in the cloud, though. See this comment's siblings for that :)
The log entries could also include a flag if the flake was installed "dirty" (having uncomitted changes at the time).
For example if you have a shell.nix that you were running with `nix-shell` which defaults to using channels to obtain nixpkgs
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
# ...
devShells.${system}.default = import ./shell.nix { inherit pkgs; };