Build systems inevitably evolve into something turing complete. It makes much more sense to implement build functionality as a library or set of libraries and piggyback off a well designed scripting language.
CI systems are also generally distributed. You want to build and test on all target environments before landing a change or cutting a release!
What Turing complete language cleanly models some bits of code running on one environment and then transitions to other code running on an entirely different environment?
Folks tend to go declarative to force environment-portable configuration. Arguably that's impossible and/or inflexible, but the pain that drives them there is real.
If there is a framework or library in a popular scripting language that does this well, I haven't seen it yet. A lot of the hate for Jenkinsfile (allegedly a groovy-based framework!) is fallout from not abstracting the heterogeneous environment problem.
Any language that runs in both environments with an environment abstraction that spans both?
>Folks tend to go declarative to force environment-portable configuration.
Declarative is always better if you can get away with it. However, it inevitably hamstrings what you can do. In most declarative build systems some dirty turing complete hack will inevitably need to be shoehorned in to get the system to do what it's supposed to. A lot of build systems have tried to pretend that this won't happen but it always does eventually once a project grows complex enough.
Do you have examples? This is harder to do than it would seem.
You would need an on demand environment setup (a virtualenv and a lockfile?) or a homogeneous environment and some sort of RPC mechanism (transmit a jar and execute). I expect either to be possible, though I expect the required verbosity and rigor to impede significant adoption.
Basically, I think folks are unrealistic about the ability to be pithy, readable, and robust at the same time.
Examples of cross platform code? There are millions.
>You would need an on demand environment setup (a virtualenv and a lockfile?) or a homogeneous environment and some sort of RPC mechanism (transmit a jar and execute). I expect either to be possible, though I expect the required verbosity and rigor to impede significant adoption.
Why need it be verbose? A high level rewuorements, a lock file and one or two code files ought to be sufficient for most purposes.
This is why I wanted to see some specific examples. I haven't seen much success in this space that is general purpose. The closest I have seen is "each step in the workflow is a black box implemented by a container", which is often pretty good, though it isn't a procedural script written in a well known language. And it does make assumptions about environment (i.e. usually Linux).
I used jenkins pipeline for a while, with groovy scripts. I wish it had been a type checked language to avoid failing a build after 5minutes because of a typo, but, it was working.
Then, somehow, the powers that be decided we had to rewrite everything in a declarative pipeline. I still fail to see the improvement ; but doing "build X, build Y, then if Z build W" is now hard to do.
I think a CI system using JSON configured via TypeScript would be neat to see. Basically the same thing as Gradle via Kotlin, but for a modern container (ie: Docker) based CI system.
I can still go back to Gradle builds I wrote 7-8 years ago, check them out, run them, understand them, etc.. That's a good build system IMO. The only thing it could have done better was pull down an appropriate JDK, but I think that was more down to licensing / legal issues than technical and I bet they could do it today since the Intellij IDEs do that now.
I'll be absolutely shocked if current CI builds still work in 10 years.
You're switching from rpms->k8s? Actually nothing has to change per repo for this.
Also it creates a nice standard that is easily enforced by your deployment pipelines: no ./run? Then it's undeployable. kthxbye etc..
This becomes important when you have >50 services.
I much prefer GitLab + k8s to the nightmare of groovy I provided over the last decade anyway..
It seems that zig [1] already does it. Hoping to try that someday...
I agree 100%. Every time I see "nindent" in yaml code, a part of my soul turns to dust.
Yup. For this reason it's a real shame to me that Helm won and became the lingua franca of composable/configurable k8s manifests.
The one benefit of writing in static YAML instead of dynamic <insert-DSL / language>, is that regardless of primary programming language, everyone can contribute; more complex systems like KSonnet start exploding in first-use complexity.
"""
apiVersion: v1
appName: "blah"
""".FromYaml().Execute()
I do like kustomize, but the programming model is pretty alien, and they only recently added Components to let you template a commonly-used block (say, you have a complex Container spec that you want to stamp onto a number of different Deployments).
Plus last I looked, kustomize was awkward when you actually do need dynamic templating, e.g. "put a dynamic annotation onto this Service to specify the DNS name for this review app". Ended up having to use another program to do templating which felt awkward. Maybe mutators have come along enough since I last looked into this though.
I haven't found time to take a look at Helm 3 yet though, it might be worth switching to.
[0]: https://www.vadosware.io/post/using-makefiles-and-envsubst-a...
[1]: https://www.vadosware.io/post/setting-up-mailtrain-on-k8s/#s...
(Or use a structural templating system like jsonnet or ytt)
Then again, grunt/gulp was a horrible, horrible shitshow, so it's not a silver bullet either...
Now what makes this incredibly complex is that the configuration step itself is semi-declarative. I may be able to reduce the configuration to "I need these dependencies", but the list of dependencies may be platform-dependent (again with recursion!). Given that configuration is intertwined with the build system, it makes some amount of sense to combine the two concepts into one system, but they are two distinct steps and separating those steps is probably saner.
To me, it makes the most sense to have the core of the build system be an existing scripting language in a pure environment that computes the build database: the only accessible input is the result of the configuration step, no ability to run other programs or read files during this process, but the full control flow of the scripting language is available (Mozilla's take uses Python, which isn't a bad choice here). Instead, the arbitrary shell execution is shoved into the actual build actions and the configuration process (but don't actually use shell scripts here, just equivalent in power to shell scripts). Also, make the computed build database accessible both for other tools (like compilation-database.json is) and for build actions to use in their implementations.
GNU make actually has this, but it's done poorly. It has build STEPS in the shell language, but the build GRAPH can be done in the Make language, or even Guile scheme. [1]
----
I hope to add the "missing declarative part" to shell with https://www.oilshell.org.
So the build GRAPH should be described as you say. It's declarative, but you need metaprogramming. You can think of it like generating a Ninja file, but using reflection/metaprogramming rather than textual code generation.
And then the build STEPS are literally shell. Shell is a lot better than Python for this use case! e.g. for invoking cmopilers and other tools.
I hinted at this a bit in a previous thread: https://news.ycombinator.com/item?id=25343716
And this current comment https://lobste.rs/s/k0qhfw/modern_ci_is_too_complex_misdirec...
Comments welcome!
[1] aside: Tensorflow has the same staged execution model. The (serial) Python language is used for metaprogramming the graph, while the the highly parallel graph language is called "Tensorflow".
I do agree it'd be nice with a more general purpose language and a lib like you say, but should this lib be implemented in rust/c so that people can easily integrate it into their own language?
Many unknowns but great idea.
Lua would be a good candidate for the latter, but its standard library is minimal on purpose, and that means a lot of the functionality would have to be provided by the build system. Interaction with the shell from Python is needlessly cumbersome (especially capturing stdout/stderr), so of those options my preference would be Ruby. Heck, even standard shell with a system-specific binary to call back to the build system would work.
Easy shelling - check.
Easily embeddable - check
Easily sandboxable - check.
Reasonably rich standard library - check.
High level abstractions - check.
If you're still guessing what language it is, it's Tcl. Good old Tcl.
It's just that is syntax is moderately weird and the documentation available for it is so ancient and creaky that you can sometimes see mummies through the cracks.
Tcl would pretty much solve all these pipeline issues, but it's not a cool language.
I really wish someone with a ton of money and backing would create "TypeTcl" on top of Tcl (à la Typescript and Javascript) and market it to hell and back, create brand new documentation for it, etc.
That said, you're right, at least at first blush, tcl is an attractive, if easy to forget, option.
They did. Larry McVoy of BitMover created Little, a typed extension of Tcl [0]. Didn't do the marketing bit, though.
In comparison, if I learned Github Actions syntax, the only thing I know is... Github actions syntax. Useless and isolated knowledge, which doesn't even transfer to other YAML-based systems because each has its own quirks.
This is so true. That's why I hate and love Jenkins at the same time.
Also, don't forget that CDK for Terraform now exists[0] as well.
[0]: https://www.hashicorp.com/blog/cdk-for-terraform-enabling-py...
It seems to work pretty well, though it feels a little constraining when I'm trying to figure something out and I can't do the standard `import pdb; pdb.set_trace()` thing. There's probably a way around that, but I've never bothered to figure it out.
I think the rationale for making it syntactically similar to Python was: we want to add a macro processing language to our build system, and we want to support common features like integers, strings, arithmetic expressions, if-statements, for-loops, lists, tuples and dictionaries, so why not base our DSL off of Python that has all that stuff, so that people familiar with Python will have an easy time reading and writing it?
Then they implemented a very limited but very fast interpreter that supports just the non-Turing-complete DSL.