sdboyer

I'd say all of these problems have answers in CUE.

> I want to be able to write a "generic environment" module for each application and then parameterize it accordingly for each environment.

This is pretty solidly in the target use case range, i'd say - managing variations of the same "object type" over some dimension is a lot of what's targeted by the way that CUE treats directory hierarchies when loading files: https://cuelang.org/docs/concepts/packages/#instances

The main thing you have to consider in designing a layout is that you have to take a compositional approach to how you define individual config instances. That is, you can't start from prod's config, then override a value or two for staging.

If i were to do it - i have not, this is not how i currently use CUE - my first approach would probably be by defining defaults at the "policy" level (per the above link), which effectively allows you to get exactly one "override"-ish behavior.

Lots of possible approaches to this, though.

> but you can't quite emulate functions as far as I can tell

Function-like capability is present, just in a form that's less familiar. I think of them as "function structs." This post has a bunch of examples https://github.com/cuelang/cue/issues/139#issuecomment-55677.... It seems there's a plan to add a more comfortable notation (https://github.com/cuelang/cue/issues/943), but it's fundamentally possible now.

Cannot +1 this hard enough. It is the kernel from which all other useful things flow.

If this is the sense in which the comment is intended - CUE is capable of some kinds of computation. It's just not Turing complete.

My 2c - thus far, i've found the language features enabled by this constraint much more useful than the expressiveness lost - at least, for the purposes i've chosen.

It's not necessarily easy to see, just from general descriptions of CUE, what's possible.

Dagger was demo'd at the recent Dockercon - gives a bit more of a sense of the possible with CUE: https://docker.events.cube365.net/dockercon-live/2021/conten...

> real language

In what sense is CUE not a real language?

> you'll be able to serialize dashboards to Cue schema

It'll be possible to serialize/represent the dashboards in CUE. Here's a handwavy, pseudocode-y example i use in the talk: https://gist.github.com/sdboyer/d76196f94ca78d1e84c739e95e64...

That said, it's not like we're planning on replacing all the "export JSON" buttons in the Grafana UI with "export CUE." One of the interesting properties of defining schemas in CUE is how it allows us to remove schema-defined default values from a dashboard's JSON. The JSON representation can actually look a lot more like a concise CUE representation.

> versioning

Versioning of the Grafana schema is the essential design goal of the "scuemata" system that is under discussion in that epic issue

Versioning of artifacts that are instances of the schema is a key goal with Polly https://docs.google.com/document/d/1GU0DGy-X6z4FVwbJYPsBKRdq...

> non-visual editing

Like, editing something other than raw code in an editor? Yes, this is also something directly enabled by the schema (again, see the Polly doc, the "Produce" heading). For data-intensive tasks, such editing experiences are the only way to see your logic in the context of data, and therefore IMO prerequisite for confidence

> reproducibility

Yup. This already isn't "hard" to do today, but reproducibility gets more complicated at scale - those questions about how to map what's on disk to what's in your Grafana (or whatever app) in my parent comment become more complicated, leading to friction, leading to staleness.

Very excited to see someone doing this! Right now, Grafana is [planned to] relying on an anemic CUE->Typescript translator for getting its schema to the frontend - https://github.com/sdboyer/cuetsy. (Somebody also pointed me to Project Cambria recently, which could be an interesting compilation target for what we have https://www.inkandswitch.com/cambria.html)

Being able to work with CUE natively in TS, though, would be a huge gamechanger for what we can do with CUE in Grafana

CUE is currently centered around its CLI, but AIUI, that's not the long-term goal. i read in some CUE issue somewhere that the goal is shifting towards enabling frameworks rather than driving people to the CLI, though i don't have a link.

Our use of CUE in Grafana is an example of framework-style usage. It is a hard requirement that users never have to install the CUE CLI to perform any of our planned CUE-related tasks; rather, the needed tooling is baked into Go packages we export, and things like grafana-cli. (Avoiding a dependency on the CUE CLI also gives us a defense mechanism against breaking changes)

re: Grafana (i'm the author of the linked issue) - i'm quite excited, i do think there's a world of possibilities here.

Two-way sync with a git repo is one possible path, and we've talked a lot internally at GL about how to best support it. My sense is that we can do it with relatively little friction and likely will - but if you're just syncing with a git repo, there's still a lot of arbitrary, opaque repo layout decisions that still have to be made (how do you map a filesystem position for a dashboard to a position in Grafana? In a way that places the dashboards next to the systems they're intended to observe? With many teams? With many Grafana instances?) which induce new kinds of friction at scale.

Fortunately - and not mutually exclusively with the above - by building the system for schema in CUE, we've made a composable thing that we can make into larger systems. That's what we're starting to do with Polly: https://github.com/pollypkg/polly

Conveniently, my parts of a Grafanaconline talk tomorrow discusses both of these https://grafana.com/go/grafanaconline/2021/dashboards-as-cod... :D

in a post-dep world, this won't matter anymore. dep strips vendor directories out of any dependencies it pulls in.

sadly, gb is one of the extant systems i haven't had a ton of time to explore. but...

i guess the analogue to what you're describing would be

1. <add an import path> 2. `dep ensure` 3. `git commit -am "all of your vendor'd source`

a bit more detail, starting at a high level: gb is a replacement toolchain. dep is focused strictly on dependency management. there is no notion of a `dep build`.

with dep, there's no explicit command to actually fetch a dep; the import graph is still queen, as is customary in go. so there's no direct analogue to `gb vendor fetch github.com/some/pkg`. if you want to add a dep, import it in your source code, and run `dep ensure`. (there's some flux in exactly how that works right now, but what i'm describing is the state we're moving towards)

`dep ensure` is really the workhorse command, and pretty much the only thing you'll ever need to run. (in fact, the only three subcommands we currently plan on having are `init`, `ensure`, and `status`).

shit, if there were an actual company with real use for Merovingian numismatics, i would work there in a heartbeat

Force pushes - by far more common than straight-up repository removal - are handled without problem; we let you stick with your old version. (At least, that's how it should be - there might be a couple more test cases to write. I know I designed for this problem early on).

Repo removal, renaming, or whatever, are still problems, for sure.

Today, dep populates vendor/ with dependencies, and works equally well whether you decide to commit them or not.

yeah, this is one of the main pain points there. the reason it's so difficult to solve in glide is intrinsically tied to the engine glide uses for version selection - otherwise we'd have dealt with it long ago.

indeed, we've talked with them quite a bit :)

nix/guix was not ignored.

There are a host of other reasons that we could happily debate for years (no thanks), but bottom line: you can't use it everywhere you can write/compile Go code.

it is, and works well enough for individuals, but doesn't make for the friendliest ecosystem.

dep mgmt, at least, is poised to improve a lot in the coming months. we've got growing consensus around some metadata files, and i'm nearly done with my SAT solver (github.com/sdboyer/vsolver), which will be in glide, and maybe others, soon.

> and then be able to compose the things you’ve proved separately

This has always struck me as the really challenging, problematic part about correctness proofs. I've no formal training in this area (sadly), but how does one go about being able to just compose your proof components? Often as not, that seems like a pipe dream - http://queue.acm.org/detail.cfm?id=2889274

> Language package managers are good for quickly sharing code written purely in that language with other developers, but unacceptable for anything else.

Right, so, what almost all of the FLOSS world is doing, almost all of the time. Miiiight be worth optimizing for that case. Especially if a proper PDM would compose well with larger systems...which it would!

A community, gaslit by years of poor tooling, IMO

> What Nix/Guix are not doing is reducing the burden of specification. You write down all the details of a very specific, reproducible software stack, but you do not make that software much more composable or extensible than it already was.

Yep, this. Glad someone else is saying it, because I'm starting to wonder how people can read literally thousands of words rooted on a foundation of making it easier for developers, and reducing their uncertainty, and then expect them to write a book-length spec doc.

(note that requiring people to write down tons of spec information reduces, at most, one kind of uncertainty, and at high cost)

> Spack stores the generated build provenance after doing some constraint solving [3], but the tool helps fill in the missing details of the dependency graph, not the human.

Sounds a lot like what's I described, no?

(I haven't heard of Spack, will look into it)

> isn’t fully reproducible because it’s not a full system package manager

I didn't touch on shared libs in the article, and probably should've, at least to point out that, for the purposes of the discussion of a PDM, it's mostly out of scope.

Things are really just so, so much nicer when we actually let there be layers and scopes of responsibility.

Nix/Guix have been brought up a bunch of times in follow-ups. I was aware of Nix (though not Guix) before writing the piece, and I do need to check it out. But...

The primary requirement for a PDM is that developers will actually use it. If the interaction is much more complicated than the command diagrams I laid out towards the end of the general section, it's really just not gonna happen. Maybe someone could build a guix frontend that focuses in on just those things. I don't know. New territory for me.

My guess, though, is that SPMs and PDMs are really only superficially similar.

fwiw, folks on twitter suggested to me that npm may be moving in something like my "sync-based PDM" direction:

https://twitter.com/rebeccaorg/status/698370989386694657

Well, spoiler then: I don't advocate for generic package management. That's someone else's problem. Maybe, if the ideas I've put forth in there lay useful foundations, it's something that could happen, but I very much doubt it's practicable.

Even if it does happen, though, I suspect that end-user package management still ends up looking different from developer package management.

I considered including some mention of crypto assurances of packages in the article. I did not, because it would take someone who knows the constraints dictated by such systems far better than I to come up with a means by which such mechanisms could be integrated into a PDM...and have people still use it.

Most of the integrity of packaging systems (at the PDM level), derive from the assurances provided by the underlying VCS - e.g., Git's tamper-proofing assurances by virtue of how its commit DAG is built. If your system has a registry in the middle, that does create an SPOF; if that registry intermediates the VCS with tarballs, that takes away the clients' ability to rely on the VCS for verification. So, in my naive view, that's the point at which signing becomes more crucial.

Glad you like glide, and thanks! Though I'm at the very most a co-author - I've contributed only ideas so far, and maybe a comment fix or two.

Now that I'm finally done writing this, maybe I can write some code.

As long as I briefly have HN's attention - how did y'all find the repo-as-universe, and universe alignment, metaphors? Useful?

(This was the diagram leading up to it: http://imgur.com/bzy22DA)

Yep - an LPM (which can work as the sort of self-contained installer you're asking for) is a reasonable thing to present to users. As I argue in the article, though, you can't have a sane LPM without a PDM undergirding it.

i agree, this overlap is a problem. and i agree that there's a not-obviously-wrong argument to be made that ignoring SPMs is the wrong approach. to this end, someone pointed me to rubinius' wiki yesterday, which had a nice breakdown of the PM hierarchy: https://github.com/rubinius/mkrb/blob/master/README.md#the-m...

OTOH: part of the reason that i included the notion of "compiler, phase zero" is because i think that's one thing that, scope-wise, differentiates SPMs from PDMs. an SPM's output is executables. i think this difference is significant. maybe i'm wrong.

(If anything, I think there's more overlap between an SPM and an LPM).