Hacker News Reader

Comment by q3k | Hacker News Reader

This delusion is especially visible in the DevOps space. For some reason we have decided as an industry that instead of writing some code in whatever 'real' language we will base operational work on YAML with ad-hoc templating and stringly-typed programming constructs.

The main culprits are Ansible/Salt and all the string-templating based tools for Kubernetes (Helm/Kustomize/...).

Especially with tools like Helm I believe we reached peak insanity levels. Instead of using a general purpose or configuration-specific (like Jsonnet/Cue/Dhall) programming language to build and then emit in-memory objects to YAML manifests, the industry is using YAML to define templated YAML and then parametrize this templating with even more YAML.

Now we just write complex declarative configuration with no debugging facilities, a non-existent type system, inversion of control that usually ends up working against us and no way to interact with external data apart from stringing everything together with fragile shell scripts. But hey, there's no need to write a single line of Go/Python/Java/Ruby!

View on HN

Well, I would have to disagree (but agree somewhat, as well :)). These insanity levels you describe, and I agree with you here, are actually pushed through projects by coders for coders.

Here in lie much of the ”devops” problem, imo, as coders seem to want to allow easy configuration management through “simple“ declarations, inventing almost a new language in the process.

Having worked for 20 years with systems/configuration management, mainly as a developer I can tell many coders have not. Not at scale, and not supporting 100s of different services simultaneously. Hence square wheels gets re-invented.

To be rid of all yaml templating and loosely coupled technical integrations you will have to treat config/service delivery/ci/cd/what-have-you as a business domain of it’s own, and develop it the same way.

I guess we agree, completely perhaps?! :)

I’ve been fortunate to have been able to work everything from small upstart to big corporate enterprises and to me this is more true now than ever.

As a developer I build my own version of a CMDB, I do not use crazy templating - I write an API, that leverages the CMDB and produces sane output. It’s still configuration as code, just something that is easier to scale and adapt.

Guess I’m only disagreeing on whom to blame for this mess.

YAML is okay for representing a simple declarative configuration. It starts to break down when you try to encode too much operational logic into it.

If we step back, there are a few different strategies for dealing with operational logic. From least scalable to most (roughly):

1. Human processes (ex: checklist)

2. In the config itself (ex: Kubernetes' YAML)

3. A config generator (ex: Helm, Kustomize)

4. Software (ex: Kubernetes Operator)

I see a lot of people get stuck doing one for too long.

When doing one becomes unwieldy, it's time to consider the next. That or find a way to reduce the need for all that operational logic.

> simple declarative configuration

The point of all this is that software configuration is never a simple declaration. It is always a mess of behaviors and derived values.

I am not sure I totally agree, but it at least correct enough to always break something. And adding abstraction layers just because you choose to use an underpowered notation at the bottom isn't a good practice.

To play devils advocate, isn't the idea with Ansible at least that it's idempotent. The YAML should describe the final state and should be concerned with branching or lower level features like that.

(It's the same with SQL, you describe what you want back rather than how its achieved. A declarative approach works fairly well there.)

There are plenty of declarative programming language families. Lisps, MLs, SQLs. Configuration is code. Use a programming language.

So how do you differentiate a declarative language like SQL with a declarative YAML file for Ansible?

What would be the benefit of adding a declaritive language into the mix over YAML?

If you want to build a DSL, build a DSL. DSLs are easily embeddable in the mentioned programming languages.

YAML is not typed and you can basically do whatever. The worst of all worlds.

Tooling. Data transformation. Libraries.

Emacs is configured using Lisp. Because of that it is amazingly configurable. XMonad uses Haskell, which gives types to avoid lots of error cases.

The thing is providing an actually programming language doesn't remove anything specially if the same config can be written as cleanly.

And you would avoid "YAML templates for creating YAML", when you actually need to process some data (even if it's just for pre/suffix creation in names). Secrets retrieval is another thing that you also need to do and template.

Also, YAML is a terrible to parse language, with can give out weird error cases. Other languages compilers/interpreters are more mature.

And ofc, if you provide SDKs for your IaC tool, you should be able to use the language that your developers are more familiar with. Taking advantage of the good practices their are used to. Don't limit with "declarative languages". Use a programing language that make more sense, and leave data languages for data.

> What would be the benefit of adding a declaritive language into the mix over YAML?

The capability to readily define, store in libraries, and use reusable abstractions that apply within a configuration or across multiple individual configurations.

Here's an example of a config system using prolog: https://github.com/larsyencken/marelle

It's a hard to tell where marelle ends and the config begins, since it's all just prolog.

However, that final YAML-defined state is not guaranteed to define the entire functionality of the system. The typical counterexample is this: removing a 'create file' clause does not cause that file to be removed on subsequent runs, as the tool has no concept of ownership nor diffing against previous states. The emitted YAML does not represent the indended full state of the machine, just what actions to take to bring it from some unspecified historical state to an underspecified target state. There is no guarantee of consistency.

Thus, it is very easy to get in a situation where an Ansible playbook applied against two machines with slightly different production history will result in very different behaviour.

If you want real declarative configuration management, try Nix/NixOS.

> removing a 'create file' clause does not cause that file to be removed on subsequent runs, as the tool has no concept of ownership

Terraform on the other hand does have a concept of ownership, diff-ing, and applying changes. It takes some work as you now need to track state, but I've been very happy with Terraform.

While (most) actions in Ansible modules are idempotent the entire playbook is not. So how your system is going to look at the end of a run is highly dependent on the order of everything in your YAML files and the current state of the system. You can ensure a file exists in and part and remove the directory containing it in another, and you are non the wiser unless you run the playbook multiple times and pay attention to the changes.

What's needed is an internal data-structure that defines the state needed -- YAML declares the structure directly, but an alternative is to use either a DSL or even a general-purpose language to build that data-structure.

Of course, you could write code to generate your YAML, but the tooling is _not_ going to help you with that today.

Maven and Gradle are examples of each of these methods -- both build up an internal model of the build, but while Maven uses XML, Gradle uses a Groozy DSL.

Of course, I meant Groovy. The keys (at least on my Dvorak keyboard) are right next to each other :P.

It's fascinating reading such a strong argument for using Lisp...and then considering anything else than Lisp.

> isn't the idea with Ansible at least that it's idempotent.

Sure, that's true with many of these, so what?

The problem isn't that the languages are declarative—functional code is declarative.

The problem is that they are extremely limited in their expressive capabilities, so that it is much more complex and error prone to describe the final state in them than it would be—even in a declarative style specifying the final configuration—in a more complete language than YAML (or, in many cases, a language essentially limited to JSON’s expressive capability even if it also supports a YAML serialization.)

> It's the same with SQL, you describe what you want back rather than how its achieved.

YAML would be an inadequate alternative for SQL’s role, too.

I'd argue somewhat that the approach works well for SQL. On a superficial level it allows you to hide the implementation details of your query execution. In practice not so much. Hence keywords like "explain" were invented and statements like "CREATE INDEX". And table partitioning. And hints. And myriad of other practices where the SQL abstraction "leaks" details from the layer below.

You can implement LUA in less lines of code than YAML, lua has been around longer, easier syntax, loads of support, proper community around it. Yet people still use YAML on hype alone, it's awful.

So, those two things are quite different. One is a programming language, the other a markup specification.

And LUA has its own hype, just not in configuration scenarios.

The ironic thing is that—specifically in the cloud space—we call this no-code abomination “infrastructure as code”.

But at least there is the AWS CDK and Pulumi now to enable “infrastructure as code” as code.

> Especially with tools like Helm I believe we reached peak insanity levels. Instead of using a general purpose or configuration-specific (like Jsonnet/Cue/Dhall) programming language to build and then emit in-memory objects to YAML manifests, the industry is using YAML to define templated YAML and then parametrize this templating with even more YAML.

As a big proponent of Dhall, I gotta say, probably the primary reason for that at this point is due to a lack of language bindings. Most tools in the Kubernetes ecosystem are written in Go. Currently, there is no official Go language binding for Dhall - one implementation[1] is pretty close, but the marshalling features need some work, and it's not the author's day job. The only way to get Dhall configuration into a Go program today is to either shell out and call the Haskell-based reference implementation directly or to script your Go binary such that the Haskell-based reference implementation translates the Dhall configuration to JSON and then feeds that into the Go program. Even that's not ideal - you end up maintaining your schema in Go code and in Dhall simultaneously.

I very much believe that Dhall will solve the stringly-typed insanity which you refer to, but the language is not quite there yet. While I'd love for some of the Kubernetes core developers to step up, I mean, I can understand people's unwillingness to adopt something that isn't 100% perfectly well-supported and handed to them on a silver platter.

[1]: https://github.com/philandstuff/dhall-golang

There is a levels of indirection problem with that explanation.

Using Dhall to create the YML you will use isn't great, but it's quite possible today and is much better than using YAML to script a YAML generator that will read some more YAML to create the YAML you will use.

I guess some people simply like YAML.

I agree, which is why I use Dhall at work. But the fact of the matter is that `kubectl apply -f config.dhall` isn't realistic yet, and there's a big segment of the market that's just not willing to accept anything but that.

This sounds a lot like the "We do everything with XML now"-phase that the Java world has gone through.

Agreed, this is a very strong parallel.

Powershell shines here.

DSC or pure powershell has bunch of useful tools and a real programming language to use in between. Yaml like configuration can be achieved with nested HashTables that have almost equivalent readability without loosing any of the devlopment capabilities.

I'm reminded a bit of systemd. Whatever their problems, when things went south, it was usually fairly easy for someone who could code to walk through the init scripts involved and figure it out. The config files of systemd at first glance look quite simple and declarative, but if things don't work, one almost immediately slams into a wall of complexity.

Sometimes one doesn't "need" a coder because the system is such that it wouldn't do you any good anyway.

Exactly, the fix is also obvious but not quite happening just yet. Several languages now support internal domain specific languages. Several of those are strongly typed. And several of those can compile to native code and integrate with other stuff around them and also come typically with nice tool support in the form of syntax highlighting, auto complete, etc. Basically all the stuff that yaml sucks at can be fixed by having a richer language model than free form strings in free form objects and lists with no semantics whatsoever; no auto complete; and nothing but run time errors to tell you you copy pasted it wrong (because that's what happens when there is no tool support).

IMHO a big part of the problem is that the companies involved are actively incentivized to justify their existence by making things more complicated instead of less complicated. If things are too simple, nobody needs their consulting, training, support, etc. Worse most of these companies are competing for attention and typically only provide part of the solution instead of the whole solution. So there's a lot of money in cross integrating multiple complicated solutions from different companies. Indeed many of the companies involved are serial offenders when it comes to this. As soon as you get companies like Red Hat, Amazon, Oracle, etc. get interested in stuff, brace yourself for some convoluted vendor lockin. That's basically what happened to Kubernetes.

When it comes to languages, I'd love something that has a compiler and auto complete. I'd also like something that has a notion of internal domain specific languages. IMHO Typescript, Kotlin or Rust could work for this. The point of this would be leveraging a type system for static checks and code completion. That kind of rules out dynamic languages like javascript, python, or ruby (all of which have been used in devops in the past). They are nice but not nice enough.

However, typescript has enough static typing that you get some meaningful autocomplete. Also, it's already popular in the node.js world (i.e. lots of stuff available to integrate). Kotlin would also be able to fit in that world since there's a transpiler to javascript, it can utilize typescript type definitions and it integrates with node.js and npm. Rust would be a bit of an outlier here and probably a bit too much in terms of language complexity for the average yaml loving devops type with a mere few years of full stack experience. With Kotlin, the transition of gradle from groovy to Kotlin has been interesting. Groovy basically is a dynamically typed language and they've slowly been replacing it with the more strictly type Kotlin. That sort of stabilized with v5 and with v6 it got usable for a lot of projects and recent updates to Intellij have improved how it is able to support developers with e.g. code completion and other features. Something like that aimed at devops could work.

Thank god trends like Pulumi and the new AWS sdk is emerging.

General purpose programing languages are getting more expressive by the day, why do we use data serialization languages instead for configs? it doesn't make any sense.

Configuration is code not data.

The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation.

These are people that by and large don't want to code, not saying that they can't or won't.

To be fair this has been in Windows shops, where scripting has only recently (last 5-10 years) taken off, so you've got a lot of windows admins that the closest they've been to code is Batch scripting with a bit of Powershell. This is a big change for them

As it happens i read about pulumi recently and I've put it on mt list of todo things, but I can't see that I'll be able to sell it to our team and our team is blessed (cursed?) with three former developers

> The main problem is that, in my experience, the majority of DevOps teams are Ops teams that have been renamed and refocused towards automation.

I'm on a relatively new team kind of like that (it's not a former Ops team but most of the team was pulled from Ops/DBA/analysis teams, and fits the description, and it's not a really a DevOps but more of a Dev+Ops team) and the general reaction of the team to being introduceed to the AWS CDK has been “we need to move to that as soon as we can”, even from people who very vocally never wanted to be programmers. And that's with, in many cases, a couple months experience with both programming and YAML IAC in the form of CloudFormation.

> These are people that by and large don't want to code

I disagree. That's the stereotype that tool builders have of such people. Good ops people have always loved coding, or we wouldn't be living on the mountains of Bash scripts also known as "Linux distributions".

(Besides, people who don't like to code won't like writing tons of declarative markup either. So there is little point in the current approach either way.)

It might be a stereotype but it's also my experience, which granted is limited and Windows based, which as I pointed out in the previous comment hasn't been really onboard with the scripting experience until relatively recently

As to the declarative markup, for instance Azure Devops still doesn't have feature parity between YAML pipelines and classic pipelines, so while you could well be right about the same resistance to yaml, it's not, necessarily an issue yet.

Most devops tools are built on Linux for Linux, then ported to Windows later as an afterthought, so I don’t think platforms are much of a factor. It’s a defect in the production pipeline somewhere, something like feedback from potential users not reaching developers until it’s too late. It doesn’t help that, in some cases, vendors just impose what is going to happen, and the community is simply forced to put up with it.

I still have my reservations against Pulumi in particular, though.

I've only really skimmed their documentation, but the idea that calling 'new' against a VM class instantiates a VM in production seems to be to magical for my taste, and might be an indication that it's yet another product that targets the happy path more than what we tend to experience in production: failures.

> General purpose programing languages are getting more expressive by the day

You know, once upon a time, we understood that declarative approaches to software engineering were superior to imperative approaches, when declarative approaches are feasible. Declarative approaches are much safer and easier to test, at a cost of only being able to express what the tool accepting the declarative approach can understand. Imperative approaches are strictly worse for any problem set where a declarative approach solves the problem within performance requirements. The additional expressiveness of languages like Pulumi is the last thing I want.

YAML is a horrible language for declarative system configuration because a) any sufficiently complex system will require you to generate your declarative codebase in the name of maintainability, b) generating code for any language where whitespace is significant will lead you to an early death, and c) stringly-typed languages are fundamentally unmaintainable at sufficient scale. But this is not an indictment of a declarative approach! It is an indictment of YAML.

> Configuration is code not data.

Data > code. Data does not need to be debugged. The best code you can have is deleted code - deleted code does not need to be maintained, updated, or patched. Code is a necessary evil we write in order to build operable systems, not a virtue in and of itself.

There are declarative general purpose programing languages.

That data you are talking about does need to be debugged, like Helm charts and pipeline definitions. Sure data is better, but config is code, not data.

Generators need to be debugged, not data. It's very easy to test a generator - a few unit tests checking whether, for a given input, the generator produced the expected output, and you're set. Data sometimes needs to be cleaned, but there's no such thing as a bug in data.

Whether the generated declarative output produces the expected behavior on the part of the tool interpreting the declarative output is part of the tool's contract, not the generator or the declarative output. If you need to check the tool's behavior then either a) you wrote the tool or b) you're writing an acceptance test for the tool, which is an entirely different endeavor.

Things like pipeline definitions and helm charts are generators.

No, Helm uses charts (data) to generate object definitions (in YAML). Helm is the generator.

There's nothing that prevents you from writing a unit test that runs `helm template` directly to check whether a given chart with given values will produce a given set of YAML files.

>but config is code, not data.

Config is both. Config variables are data. The code that accesses and uses those variables is...well...code. they should be kept separate. Like any other code and data. Config isn't a separate special entity, it's just another part of the program. The data part should be represented as such and the code part should be code. Trying to combine them and create a special 'config' language is mistake.

That's a bit of a no-true-scotsman there. If the problem is just the markup of choice, we should see an alternative markup emerging any time now. If we see imperative-focused tools instead, maybe it's not just about the markup.

We do see alternative "markups", if you want to call them that, emerging that solve the generative issues - the two that come to mind are Dhall and CUE. They bring forth JSON to help them interoperate and be relevant in a world that predominately expects JSON/YAML, but they can also be read directly.

I think "declarative" is a bit of a red herring here. Deterministic/reproducible/pure is a more appropriate distinction: configuration languages like JSON/YAML/XML/s-expressions/etc. are trivially deterministic, but not very expressive, leading to boilerplate, repetition, external pre/post-processing scripts, etc.

Allowing computation can alleviate some of those problems, whether it's done "declaratively" (e.g. prolog-like, as in cue) or not (e.g. like idealised algol with memory cells).

The main reason to avoid jumping to something like Python isn't that it's "not declarative"; it's that Python is impure, and hence may give different results on each run (depending on external state, random number generators, etc.). Python can also perform arbitrary external effects, like deleting files, which is another manifestation of impurity that we'd generally like to avoid in config.

tl;dr The problem isn't the style of computation, it's the available primitives. Don't add non-deterministic or externally-visible effects to the language, and it wouldn't really matter to me whether it's "declarative" or not.

I use Lua for configuration files. It's easy to restrict what you can do in Lua (I load configuration data into its own global state with nothing it can reference but itself). Plus, I can define local data to help ease the configuration:

    local webdir = "/www/site/htdocs"

    templates = 
    {
      {
        template = "html/regular",
        output   = webdir .. "/index.html",
        items    = "7d",
        reverse  = true
      },
      
      {
        template = "rss",
        output   = webdir .. "/index.rss",
        items    = 15,
        reverse  = true
      },
      
      {
        template = "atom",
        output   = webdir .. "/index.atom",
        items    = 15,
        reverse  = true
      },
    }

When I reference the configuration state, templates[1].output will be "/www/site/htdocs/index.html". And if the base directory changes, I only have to change it in one location, and not three.

> Thank god trends like Pulumi and the new AWS sdk is emerging.

The more things change, the more they stay the same.

The pendulum is swinging back towards scripting languages. But give it a few years, and we'll be railing against scripting languages for not being idempotent enough, and moving back to the latest version of YAML and XML (perhaps we'll go with TOML this time) with custom interpreters.

We have seen this half a dozen times already, between turing complete DSLs in ruby, to plain bash, perl, and python scripts. The other side are the DSLs written using YAML, JSON, and XML (and every other config language written in history).

> new AWS SDK

Could you share some details on that? A quick Google search didn't reveal anything about major changes to the SDK.

Probably referring to the CDK (Cloud Development Kit), a newish SDK for developing Cloudformation infrastructure-as-code in general-purpose programming languages.

https://aws.amazon.com/cdk/

I believe the OP is referring to the AWS CDK (Cloud Development Kit) - https://docs.aws.amazon.com/cdk/latest/guide/home.html

> Configuration is code not data.

As any Lisper knows, code is data is code.

Maybe because we've chosen such rigid, inexpressive and DSL-hostile programming languages for devops... I mean, Python, Go... ofc you'd rather write your DSL in YAML instead and forego any checking the language might provide since everything else would be too awkward. Something like a "statically typed Ruby" would probably shine here.

TL;DR: "your language sucks, my language would shine"

Language doesn't matter. What matters is the attitude of tool builders that everything should "simply" be described as data. When that turns out to be insufficient, as it inevitably does, hacks are introduced and sooner or later you end up with crippled and idiosyncratic pseudo-languages.

Reply on news.ycombinator.com