Hacker News Reader

Comment by Kednicma | Hacker News Reader

I wish I understood how to bring new things to Nix. Like, if I have some language or framework with a bunch of plugins and modules, and I have the freedom to bend the framework a bit in order to get it working with Nix, then what is the best path for doing that? All of the documentation I've read seems like it's been written by folks who deeply understand not just each different language, but also some sort of central Nix philosophy.

View on HN

I think one way to look at nix is that it want to act(in good way) the central source of truth for deterministic package, and one way it goes about it is by doing the step for both system level packages, and language level packages(for example, if you approach a nodejs app, you will lock both the nodejs version, the underlying openssl or other headers lib used by node-gyp, and the node_module packages itself in nix) and that is one of the main flow of nix, take a existing project/app/software, and convert it to usable nix packages, and that work even in edge cases like:

- app that is binary , so you can't build it from source, but is using dynamic libraries, then you can use autoPatchelfHook to tell nix to patch the binary to swap out the libraries to nix's version

- app that uses a lot of hard coded file system path that you would prefer not to patch them all out, then you can use buildFHSUserEnv to package/run the application/package in full FHS-compatible scenario

but the main point is that, nix is extreme in its approach, and the best approach in my opinion, is to do swapping step by step, ex:

you want to convert a existing project to nix, lets say nodejs

- you add the main deps in the default.nix (nodejs, yarn, high level utility needed to build)

- use sandBox false to allow building the app with yarn/npm in internet accessible(by default nix uses sandBox which disallow internet inside build step, which causes issue like npm install not working, etc)

- confirm working, then start using tool likes yarn2nix, other similar, and convert the packages itself to a usable set of nix packages

though I do agree that nix have the knowledge bias issue, and alot of my learning of nix involve me looking at the nixpkg sources itself and sometime even the nix's code

The trouble is that it's not just that nix is opinionated, it's that much of the complexity is in patterns and conventions that are implemented in nix. Learning everything about everything about the Nix toolchain is only one third of the battle; the rest is learning all of the conventions and patterns for writing Nix expressions, and these conventions and patterns are subject to variance between target languages. This is made terribly difficult because the language is dynamically typed and it's almost impossible to know about the "shape" of any given parameter because there is no way to know where it is defined in the enormous nixpkgs repo except to grep around and try to find the place where the caller is invoked, what is passed into it, and trace that back to an original import statement (and from there to a file on disk and the corresponding definition). That long tedious process is the hot path for developing in Nix.

Very much agreed; nix is its own world, which makes it really hard to get into. I wish someone had written "nix but in python" (or bash, or whatever) that still used a nix-store equivalent, kept the overall design and the immutable and reproducible packaging, still connected everything by hash, did all the hydra-style build infrastructure.. but didn't use nix-the-language, and made an explicit effort to use more conventional language wherever reasonably possible. Ideally, it'd even let you write expressions in arbitrary languages; it should be perfectly possible to say, "here is a directory containing 'inputs' and 'output' subdirectories based on your declared dependencies; place any build steps you want in build.sh, so long as they deterministically populate 'output' from the package directories in 'inputs'" (and then run the build a few times, without network access, to make sure). I don't see anything in nix that actually needs nix-the-language, or even a functional language at all, and I think nix-the-package-manager would be far more accessible if they'd not tied nix-the-language to everything.

I think functional languages are a good fit for the problem though. Given some input, compute a static build plan. It's very rare that you need to do imperative-style computation for that. What I think Nix really needs is some kind of static typing, because right now if you make a type error the error might appear in a completely random location, making it difficult to debug.

I don't think this is any more true for this problem space than other problem spaces. The "pure functional package manager" property doesn't come from the functional expression language, but from the thing that takes the static build plan and executes it. You could make these build plans in Python (or Starlark https://go.starlark.net) very elegantly and with the massive added bonus that they are intelligible to a much broader audience, and Python still allows for a very functional style. The only caveat I would add is that the Nix expression language has really nice support for multiline strings and Python still hasn't figured that out. Lastly, Python also supports type annotations and even a type checker (although the type checker leaves a lot to be desired, such as basic types or callbacks that take kwargs so it might be better off to implement one's own type checker).

But 100% agreement that Nix needs a type checker!

Take this example build file in Mozilla's custom, Python-based build system: https://searchfox.org/mozilla-central/source/gfx/skia/moz.bu...

The language is imperative, and the result is fully declarative. A pure functional description of this logic would worsen understanding instead of improving it, I believe, because when your conditionals affect multiple output variables, you have to duplicate the condition in multiple places.

Looking at the code, I don't see any advantage of using imperative language here. All the conditions used there are already available and don't change during the execution of that code. So the conditionals would be nearly identical in functional language.

Nix advantage here would be that is lazily evaluated, so there would be an additional benefit that only conditionals that are used are evaluated.

Functional benefit of Nix would be that Nix would know that a package doesn't need to be rebuilt if no inputs changed.

Side note Mozilla actually uses Nix internally, it appears that it is used for Rust and Firefox and most likely other things[1][2]

[1] https://github.com/mozilla/nixpkgs-mozilla

[2] https://github.com/mozilla/release-services

> Looking at the code, I don't see any advantage of using imperative language here. All the conditions used there are already available and don't change during the execution of that code. So the conditionals would be nearly identical in functional language.

Yes, except that I can read this and I can't read nix. That's probably based on background, but more people are familiar with imperative than functional languages.

> Nix advantage here would be that is lazily evaluated, so there would be an additional benefit that only conditionals that are used are evaluated.

Agreed, although that's unlikely to be unique to nix?

> Functional benefit of Nix would be that Nix would know that a package doesn't need to be rebuilt if no inputs changed.

As would anything that can compare hashes. Or heck, make(1) can do as much.

> As would anything that can compare hashes. Or heck, make(1) can do as much.

Of course, language like Python is Turing Complete so you could implement the same functionality . You could also add functions that behave and perform operations lazily.

But you're missing the point though, those are properties that Nix provides natively that are tuned to what is required for this domain. Because the only side effect in nix is a derivation, and because language is lazily evaluated it is made for this purpose.

If you would use for example Python, you would need to write wrappers that would provide that behavior and it no longer would be Python, it would be some kind of DSL written in Python at that point and the risk then would be that it would be very easy to introduce iterative code that's not functional and not lazily evaluated (kind of like it happens with Chef recipes).

The Nix Language actually is not the problem here, the language is fairly simple, the biggest difficulty is actually nixpkgs which is huge and poorly documented. You often have to look at its code to understand what's going on. If it was written in Python you would have the same issues.

There is one thing that Nix Language could do better, a lot of people wish it was a typed language, because if it was typed it would be much easier to find a definitions, and wouldn't require as much of greping the code. But python again wouldn't help here.

Nix is also Turing complete, and Python (to my knowledge) can also run in a side-effect-free context through various methods (it’s easy enough to enforce during code review but there are technical solutions as well). There’s also Starlark which is a sealed, embedded Python-like which is NOT Turing complete and which can also avoid side effects (evaluating to a derivation, if you like).

Nix may be simple but it’s unfamiliar, and you’re already asking people to understand a novel package management system and everything about everything about their entire dependency tree, and these unknowns are collectively more confounding than the sum of their parts.

Python would help because it is familiar and gradually typed. That said, I’m open to any language that satisfies those properties (maybe also “composes nearly from expressions”).

I don't believe Nix is Turing complete, unless you include ability of running python or bash script.

Regarding the rest of your first paragraph, that's essentially what I said, you can write side effect free code in python, but it's so easy to introduce side effects if you're not paying attention.

I'm saying though that nix language isn't the hard part about nix, but perhaps you're right and it is one extra difficulty that just makes things harder.

But take a look at most o Nix derivations, majority of them end up being just an attribute set (aka dictionary, map etc in other languages)

Take look at https://nixos.org page (they changed it after this submission was posted, and it is better than the old page at showing what Nix can do)

> I wish someone had written "nix but in python" (or bash, or whatever) that still used a nix-store equivalent, kept the overall design and the immutable and reproducible packaging, still connected everything by hash, did all the hydra-style build infrastructure.. but didn't use nix-the-language, and made an explicit effort to use more conventional language wherever reasonably possible

Guix?

I haven't dug in far enough to be confident in this, but I think they basically just did a straight one-to-one replacement nix->guile? Which is certainly better than creating a whole one-off language just for the purpose, but is still tied to the one language, requires you to wrap your head around a hole different mental model and programming environment, and importantly to me, uses a language that I'm not familiar with in a paradigm that I don't frequently touch. I mean, I'm barely a programmer, let alone a LISPer; I can write Arch PKGBUILDs and Alpine's APKBUILDs and BSD port/package Makefiles, but the moment that creating a package requires using any significant amount of a real, non-trivial programming language, complete with local idioms, I'm going to get stuck really quickly. A lot of my frustration with nix is that everything is a special case. There's no way to "configure && make install"; you have to use some special nix module that knows how to handle autoools/cmake/whatever. And I understand why this is really great as long as you're doing something expected, but the first package I tried to write didn't fit into any of the obvious patterns so I was stuck in the deep end with no obvious way to just write the three shell commands that I needed to actually build the package.

Typically getting languages to talk with each other is very difficult and onerous for the programmer unless the languages were built to interface with each other, and in this case, that probably means "different syntax frontends for a common semantic backend".

Even if you build Python-like and JS-like and etc syntax frontends for a hypothetical common backend, they will only be syntactically similar, but the semantics and standard libraries will be different than Python, JS, etc so there's really very little to be gained here.

And maybe it's even worse, since the whole goal is to allow a Python-like frontend to reference functions and variables defined by other frontends, the authors will probably have to understand certain aspects of all frontends. if those frontends are sufficiently similar and familiar (e.g., Python and JS) then there isn't much of a problem, but if you throw in an obscure syntax like Nix this becomes harder.

I'm not saying that I want library/module integration; I want nix to be able to build a package by creating a working directory with a set of input directories and an output directory, and then running an arbitrary script to convert the inputs into the output.

If I understand correctly you can do that today (nix builds with bash scripts by convention but there's no reason you can't ask it to use Python or whatever or call into Python/etc from bash). But that's all orthogonal to the expression language.

Nix-the-language is the easiest and most pleasant part of Nix. It's basically JSON with recursion and variables.

Grafting Python on top of the idiosyncrasies of how Nix does things would be a nightmare of impedance mismatch.

I have a different subjective experience.

The mixing between parenthesis and braces and the lack of standard indentation makes some derivations really hard to read.

I can't give you an example right now, but I've been looking at the nixos modules that deal with creating NixOS images lately. So you can get an idea of the part of the codebase that gave me this experience.

> The mixing between parenthesis and braces and the lack of standard indentation makes some derivations really hard to read.

For sure, you're absolutely right. I'm just saying that doing that in Python would have exacerbated the problem, not made it better. (Imagine someone taking the full power of Python and unleashing it on the derivation writing process, yuck.)

Well, if I'd have it my way, I would write the derivations in Haskell. That way, I would have a clearer picture of the different data types that flow through the build process.

I don't think Python would be a problem. I've used it for lots of other similar configuration-esque tasks (which is what the Nix expression language is doing--producing a configuration payload that describes how to build a target) and it works well. If you really want to constrain it, there is Starlark (https://go.starlark.net) for exactly this purpose, but in practice I haven't really needed these constraints.

Aside from the official docs and wikis, I find the nixpkgs repository[1] extremely useful as a reference and is definitely one of the things that make me love Nix. It helped me a lot to get started.

Any official package from Nix belongs to this single repository, making it easy to see what other packages are doing. If I want to see how python packaging is done, I simply search "python" in the nixpkgs repository.

It's also easier to grasp what is going on in nixpkgs than other packaging systems. The Nix expression language offers better readability than say, RPM spec files. Meanwhile the nixpkgs repository includes support for numerous build systems and frameworks. This combined allows you to create packages with minimal effort most of the time.

[1]: https://github.com/NixOS/nixpkgs

If you want to know what the result could look like I would recommend the following blog post by Christine Dodrill where she shows off her setup for working with Go code in Nix: https://christine.website/blog/i-was-wrong-about-nix-2020-02...

It's more focused on the result then on how you get there, but that result really impressed me.

Are you stuck with the interface for the user? Based on what I saw with python and postgres the latest and greatest seems to be:

yourthing_custom = yourthing.withPackages(ps: [ps.plugin1 ps.plugin2 ] )

Or are you stuck with how to implement it?

mkDerivation is a nice hammer to try

Reply on news.ycombinator.com