Why are we templating YAML?

Also see `webpack` as a successful example of code-as-configuration in the wild.

Hi, is Pulumi an generalized AWS CDK(https://github.com/awslabs/aws-cdk/blob/master/examples/cdk-...)? Looks pretty similar :D

Are you going to add C# support?

> My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years.

I think that you’re right, and I think it’s great, because we have a programming model in which code is data and data is code: Lisp & S-expressions.

It’d be downright awesome to have a Lisp-based system which used dynamic scoping to meld geographical & environmental (e.g. production/development) configuration items. But then, it’d be downright awesome if the world had seriously picked up Lisp in the 80s & 90s, and had spent the last twenty years innovating, rather than reïnventing the wheel, only this time square-shaped. But then, the same thing could be said about Plan 9 …

I’ve not yet had the time to take a look at Pulumi, but I hope to have time soon.

I think what you're doing with pulumi is the right answer and it's only a matter of time before this becomes the norm. The author's examples could easily be done with plain ol' JS/ES/TS with more far more extensibility and customization when the need arises.

I also feel this is where JSX got it right. Instead of creating yet-another-templating-language (looking at you Angular!), they used JavaScript and did a great job of outlining how interpolation works. Any new templating language is always going to be missing some key feature you expect out of a general programming language and your customers will continue to ask for more features.

Take for example Terraform and HCL, they're continually adding more and more [templating features](https://github.com/hashicorp/terraform/blob/master/website/d...) and [functions](https://www.terraform.io/docs/configuration/interpolation.ht...) because there's so many different ways to skin configuration/infrastructure as code. What if TF just expect a "computed" JSON object and it's left up to the developer to figure out how to put it together?

I'm gonna keep an eye on Pulumi and hope to be able to use in a real project soon.

This looks absolutely great! I’ll give this a thorough look over for our coming API development/deployment.

Thanks for plugging!

In ROS we have these XML launch files that are just awful. They have enough features to be a really bad programming language for configuring and launching (often conditionally) numerous robot software nodes.

In ROS2 the launchfile can now just be a Python script. Very much learned all this the hard way and the solution was to just support Python. I think it's brilliant.

I have been using ksonnet but that is now officially dead. Working with jsonnet seemed unnecessarily painful when coming from coding typescript. This information is quite timely and welcome, I'll look further at the ts example.

As a happy pulumi user, I have to say I am very impressed with the experience. An order of magnitude improvement on maintainability over our old terraform code base. Highly recommended.

Build files (e.g. makefiles are their various descendants like SCons, rake, etc) seem to be in the same general boat except very early on mixing "real languages" (or at least shell scripting) was obviously allowed so they've always leaned far more towards the "yes, it is a general purpose language" end of the spectrum.

I owned the majority of the configuration system and ecosystem for Borg, Google's internal cluster management and application platform.

Unfortunately, what described here is good in many level, but not excellent in any.

If you are OK to describe the complexity of your infrastructure in a programming language similar to the general purpose language, then a well abstracted API built on original APIs from cloud providers are more familiar to devs. And it will be more reliable performance and flexible.

If you want a config experience, something like kustomize is leaner and more compatible with the text config model.

I also cannot see how this interoperate with other tools, which will seriously limit it's appeals to people using other tools.

For what it's worth--I've been using Pulumi on a couple of different projects and, today, I couldn't imagine starting a cloud-based project on anything else. The Pulumi team has spent more time than almost anybody I know on understanding how to attack these problems; I guess I have a bit of an understanding of just how much work that is, as I've tried to do the same thing and their solution is better.

I appreciate that their revenue model doesn't require making the open-source version frustrating or stupid and I appreciate that they're incredibly responsive. And some of the stuff you'll see around cloud functions/Lambdas and the deployment thereof will fucking blow your mind.

It's good. You should strongly consider it.

To go with your general line of thought, see how many JS-based projects are increasingly moving towards a JS file with a default export as a config file.

This is a great analysis, but it's missing a fundamental point: why do we have a problem with these approximations of a programming language or just using a programming language to template stuff?

Because your build then becomes an actual program (i.e. Turing complete) and you have to refactor and maintain it! This is the common problem of using a "programming language as configuration" (e.g. gulp?)

Dhall solves exactly this problem: https://dhall-lang.org

It has the same premises of Pulumi, but without the Turing completeness (I don't know if/how Pulumi avoids that, but if it does it should be part of the pitch), so you cannot shoot yourself in the foot by building an abstraction castle in your build system/infrastructure config.

We use it at work to generate all the Infra-as-Code configurations from a single Dhall config: Terraform, Kubernetes, SQL, etc.

And there is already an integration with Kubernetes: https://github.com/dhall-lang/dhall-kubernetes

The problem with code as configuration is that the config file is indeterministic and it takes longer to extract information from the file.

This has long been a problem in the python/pip community, as its basically impossible for the build tools to determine the dependencies of a package without fully downloading and running the setup.py file.

Static config files are static for a reason!

There are several possible situations:

- the django like situation: the configuration is pure code, and it's a mistake. It was not necessary, it brought plenty of problems. I wish they went with a templated toml file.

- the ansible like situation: the configuration is templated static text. But with something as complex as deployment, they ended up adding more and more constructs, until they created a monstrous DSL on top of a their implementation language, with zero benefits compared to it and plenty of pitfalls. In that case, they should have made a library, with an API and documentation making an emphasis on best practices.

- and of course a big spectrum between those

The thing is, we see configuration as one big problem, but it's not. Not every configuration scenario has the same constraints and goals. Maybe you need to accept several sources of data. Maybe you need validation. Maybe you need generation. Maybe you to be able to change settings live. Maybe you need to enforce immutable settings. Maybe you need to pub sub your settings. Maybe you need to share them in a central place. Maybe they are just for you. Maybe you want them to be distributed. Maybe you need logic. Maybe you want to be protected from logic. Maybe the user can input settings. Maybe you just read conf. Maybe you generate it.

So many possibilities. And that's why there is not a single configuration tool.

What you would need, is a configuration framework, dealing with things like marging conf, parsing file, getting conf from the network, expressing constraints, etc.

But if you recreate a DSL for your config, it's probably wrong.

> My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.

This is the why I prefer to use a JS file for configuration instead of native JSON or YAML file if those options are available.

Having dealt with puppet, cloudformation, ansible and other solutions that have gone in and out of fashion and also dealing regularly with Kotlin, Java, Javascript, and recently typescript, my view is that configuration files are essentially DSLs.

DSLs ought to be type safe and type checked since getting things wrong means all kinds of trouble. E.g. with cloudformation I've wasted countless hours googling for all sort of arcane weirdness that amazon people managed to come up with in terms of property names and their values. Getting that wrong means having to dig through tons of obscure errors and output. Debugging broken cloudformation templates is a great argument against how that particular system was designed. It basically requires you know everything listed ever in the vastness of its documentation hell and somehow be able to produce thousands of lines of json/yaml without making a single mistake, which is about as likely as it sounds. Don't get me started on puppet. Very pleased to not have that in my life anymore.

On a positive note, kotlin recently became a supported language for defining gradle build files in. Awesome stuff. Those used to be written in Groovy. The main difference: kotlin is statically compiled and tools like intellij can now tell you when your build file is obviously wrong and autocomplete both the standard stuff as well as any custom things you hooked up. Makes the whole thing much easier to customise and it just removes a whole lot of uncertainty around the "why doesn't this work" kind of stuff that I regularly experience with groovy based gradle files.

Not that I'm arguing using Kotlin in place of Json/yaml. But typescript seems like a sane choice. Json is actually valid javascript, which in turn is valid typescript. Add some interfaces and boom you suddenly have type safety. Now using a number instead of a boolean or string is obviously wrong. Also typescript can do multi line strings, comments, etc. and it supports embedding expressions in strings. No need to reinvent all of that and template JSON when you could just be writing type script.

I recently moved a yaml based localization file to typescript. Only took a few minutes. This resulted in zero extra verbosity (all the types are inferred) but I gained type safety. Any missing language strings are now errors that vs code will tell me about and I can now autocomplete language strings all over the code base which saves me from having to look them up and copy paste them around. So no pain, plenty of gain.

And yes, people are ahead of me and there are actually several projects out there offering typescript support for cloudformation as well.

This is my experience and it's clearly biased from maybe one bad example but ... Scons is an example of code over configuration and from what I could tell I never met someone that truly understood it. Because it was code over configuration every programmer added their own interpretation of what was supposed to happen, no programmer truly understood what was really going one and it turned into one giant mess of trying to understand different programmers hacks and code to get the build to work. I'm sure some Scons expert will tell me how I'm full of crap but I'm just saying, that's my experience.

So, what's my point? My point is configuration languages help in that they push "the one true way" and help to enforce it. Sure there are times you end up having to work around the one true way but given very powerful tools of a full language for configuration leads to chaos or at least that's my experience. Instead of being able to glance at the configuration and understand what's happening because it follows the one true way you instead end up with configuration language per programmer since every programmer will code up stuff a different way.

>> I really dislike YAML

I wasn’t aware anyone liked it

I generally dislike languages like YAML or Python where whitespace matters, and can break your code, however, YAML is way more easily human readable than JSON, so I started to appreciate it for readability purposes.

I guess YMMV, but after you've used both YAML and JSON for a while, you might appreciate YAML a little bit more.

The simple parts of YAML are great for simple applications. The complicated parts of YAML are not great for complicated applications.

YAML is a bit bonkers in that it's a superset of JSON (all valid JSON is valid YAML), so if you don't like the whitespace sensitivity, you can write your YAML like this:

  {
    a: 42,
    # But you can have comments!
    b: "hello world",
    c: "and
      multi-line
      strings!",  # and trailing commas!
  }

I find YAML to be almost unusable. IMO it's just not intuitive. If I get to choose a format for my config files I would only use TOML, it's just better (again IMO).

I don't mind YAML. I dislike that something things become strings and sometimes they become other types:

   foo: bar   # {"foo":"bar"}
   foo: "bar" # {"foo":"bar"}
   foo: 42    # {"foo":42}
   foo: "42"  # {"foo":"42"} 

Other than that, no major complaints. My editor understands YAML and shows the indentation level in the background (highlight-indentation-mode) and auto-formats files so they all have consistent indentation (prettier-mode). As a result, it is not much of a nightmare to edit, despite the fact that semantic whitespace COULD cause you a lot of problems.

Writing YAML is easily the part I hate most about writing/deploying software. It's unstructured, feedback cycles tend to be slow (e.g. when deploying k8s configs into prod), and you can't possibly write something useful without documentation pulled up. It's definitely easier to implement than purpose built DSLs, but it's not a good experience.

> Syntactic whitespace kills me.

Okay, I'm gonna be the asshole in the room, but how hard is it to just use consistent indentation? I can't count how many times I've heard people complain about significant whitespace in languages.

Not only is it not difficult to begin with, but every code editor and IDE will show you where there's a syntax error in your YAML. People are free to dislike YAML, even for its significant whitespace, but how does it "kill you"?

Look at this example from the article:

```

something: nothing

  hello: goodbye

```

This is pure sloppiness, and anyone who has trouble carelessly adding pointless bytes to code, no matter the language, is sloppy. I don't understand why people criticize YAML and Python because "whitespace is hard".

P.S.: There's a similar configuration language called ArchieML, which is similar to YAML but doesn't have significant whitespace.

http://archieml.org

I know you only said Python is better, not great, but you might want to check out OpsMop: https://medium.com/@michaeldehaan/opsmop-building-the-next-g... . By the creator of Ansible, in pure Python, including the config.

I like looking at YAML when it doesn't use any of the insane YAML features. Even so I'm not convinced it should be used (at least not as widely as it is) for one big reason: it can be truncated almost anywhere and still be valid. This causes way more issues than you might think. JSON has no such issue - the only case I can think of where you can truncate JSON into valid JSON is if your JSON is just a number.

YAML is useless because it replaces JSON (tree structure that is minimally verbose to not be confusing) with something worse (a tree structure that is just less verbose than JSON to be slightly confusing)

Python has better IDE support and, maybe more importantly, python does not completely disallow any indentation style so it handles unsupported IDEs better. But it's essentially an IDE support problem.

My life improved a lot since I got an YAML mode for emacs. Now things would be just perfect if haskell's cabal migrated to dhall...

god those examples are ugly - commas at the beginning of a line? mismatched brace styles?

Came here to say similar. In particular dhall does allow scripting (functions etc.) but is non-Turing-complete as a feature. This seems like a particular sweet spot to me as it allows for more dynamism than data formats like json/yaml while constraining the scope sensibly.

It also has very nice bindings with haskell and nix

For the sake of this comment, let's define "templating" to be attempts to solve the problem "I need $FORMAT due to an existing constraint, but $FORMAT does not entirely meet my needs on its own" (in this article, $FORMAT is YAML). Additionally let's say that in order to be a "template" something must be a text file (e.g. exporting a database table as $FORMAT does not count as "templating" for the purposes of this comment).

I think there are three very different kinds of tools that people use for this:

1. Interpolation/preprocessor languages: This is what the author is talking about. There are delimiters/tags/sigils to distinguish "the templated parts" from "the rest" and the primary operation done by the template engine is substitution. "The rest" is literal content that's already in $FORMAT and it remains mostly/entirely unchanged during template rendering. Languages of this type are basically glorified `sed`. This can be nice because they're agnostic as to their embedding (any string will do) so they're very portable/flexible (you don't have to create "handlebars for YAML", "handlebars for HTML", "handlebars for CSV", etc; one implementation does it all). Languages of this kind can work in the small but don't scale well for all the reasons mentioned in the article/comments. The language doesn't know anything about the semantics of $FORMAT and that can cause all kinds of pain. Examples include golang templates, PHP, ERB, handlebars, the C preprocessor, Jinja, etc.

2. Compilers/code generators: These are "complete" languages that compile to $FORMAT. The difference between these and and interpolation/preprocessor languages is that the entire input is the language, not just specific chunks/tags. This kind of language can be nice because you have complete control and can therefore guarantee valid output and do tricks like supporting multiple different output formats for the same input, but the downside is that you're working with an entirely new language so there's a learning curve, you need specialized syntax highlighters and other tools to work with templates, etc. Examples include HAML, Jsonnet, Dhall, etc.

3. Embedded DSLs: Templates of this kind are valid $FORMAT from the beginning, but have embedded ways to specify transformations to be applied to the parsed AST. These languages are homoiconic with respect to $FORMAT. First $FORMAT is parsed, then the template engine iterates through the AST to perform evaluations, then the result can either be used as-is in memory or serialized back to (a possibly different) $FORMAT. This is sort of like an interpolation/preprocessor language with the evaluation order swapped: preprocessing is "run the template engine, then parse $FORMAT" while this is "parse $FORMAT, then run the template engine". A downside of this approach is that it is less general, e.g. it only really makes sense when $FORMAT has a well-defined structure (you probably can't template plain english sentences with this approach), but these days most "data languages" have converged towards being semantically equivalent to JSON (lists, dictionaries, and primitives) and this approach works well for any of them. An upside is that like compilers/code generators you can guarantee that the output will be valid $FORMAT no matter what the template looks like. Examples include JSON-e, Lisp macros, CloudFormation templates, etc.

It's unfortunate that all of these get called "templating languages" because they're very different beasts from one another, and usually when I see conversations about this stuff these distinctions get blurred and you end up with apples-vs-oranges comparisons. If I had my druthers we'd reserve the word "templating" for the first one and use different terminology for the others, but that ship has sailed.

> If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world.

I actually really like the idea behind XSLT: machine-friendly, human-tolerable, structured data + declarative rules for turning that data into a display, or a report, or whatever else.

The execution was horrible though: incredibly verbose, lots of overcomplication due to XML weirdness/asymmetries (e.g. attributes vs elements vs text, namespaces, ...); mixtures of different languages hidden inside each other (e.g. XPath hidden in attributes); etc.

I would really like to see what this could look like if done in a more minimalist, lispy fashion (normal code-is-data stuff in Lisp is similar, but I think term-rewriting is a more appropriate evaluation mechanism for such rules)

I actually like XML templating, it's the only one that supports it in the file.

I export custom queries for data to xml, Json, CSV and HTML.

Where HTML is XML + XSLT. It works great. And clients can even theme it

What you say you want is XSLT though, just wity a better syntax and for YAML. XSLT was just fine, had a lot of very good ideas, just horrible syntax.

Horrible syntax is kind of a forgivable offense, lots of things have horrible syntax and work fine.

Generating YAML with go templates though.. it is just horrible on so many levels.

I definitely feel the 90s can take the higher ground over 2010s on this one.

I kinda miss it, actually. XML had many warts, but at least everybody spoke it, and it was the same everywhere. Occasionally you still had some overlapping but different things, like XSD and RELAX NG schemas (though even there, there was a big difference - one is a language for describing data types, and the other is a language for describing grammars). But it's better than several dialects of JSON, YAML, TOML etc.

I also rather liked thorough extensibility. Namespaces were the right idea, despite clunky syntax. Today you can see Clojure doing something similar in Spec.

And while we're on the subject of XML, XSLT and Clojure; I feel like this is the best solution for readable serialization of tree-like data, and an associated ecosystem of tools (to validate, transform etc). Note some nice features for humans, like the ability to comment out a specific node, in addition to the usual line-oriented comments.

https://github.com/edn-format/edn

> I don't think anyone wants to use Helm.

Here is why everyone should use Helm:

Helm 2.0 introduced package as a first-class concept for Kubernetes and created the standard to distribute applications, thanks to Helm thousands of people could discover and collaborate on cloud-native deployments of the open source software https://github.com/helm/charts/tree/master/stable published and managed by organizations and contributors all over the world.

Helm 3.0 keeps innovating, it adopts the most forward-thinking approach to package management and Kubernetes config management by using higher level domain specific language based on Lua to create expressive package management system:

https://sweetcode.io/a-first-look-at-the-helm-3-plan/

Helm is also backed by CNCF[1] and is the best choice so far for organizations to create a reproducible CI/CD pipeline in a Kubernetes cluster.

[1] https://www.cncf.io/blog/2018/06/01/cncf-to-host-helm/

I’m in agreement; it seems lots of projects use helm charts for hello world / standard deployment demos, but considerably fewer run Helm charts exclusively in production clusters.

>it's as bad or worse than XML for config files

XML works very well for config files. It's schema-optional (but is there), well-specified, human-readable, has plethora of supporting technologies (making things like templating easy), and is well supported by every language.

At the very least it is way better than JSON.

If you think the whitespace is bad, just wait till you see the implicit casting: https://hitchdev.com/strictyaml/why/implicit-typing-removed/

> it's as bad or worse than XML for config files

Let's agree to disagree here. No human should ever write XML. No human should ever be forced to read it.

YAML is very readable and writable if you stay away from the corners. Templating allows you to stay clear of the corners (the 1 char operators that concatenate stuff, b64 stuff and so on).

This is my favorite collections of reasons that yaml is a bad config file format: https://github.com/cblp/yaml-sucks

SQL as a configuration format is not that bad of an idea

It’s DRY run amok. People don’t want to see the same bit of anything in two places, forgetting that code will be read, so they remove a “redundancy” but create duplicated effort for everyone, every time they have to decipher the thing later.

I also don’t think we have a workable definition of “configuration”. 70% of the config at my work is hard coded service discovery. If we moved the service discovery anywhere else (say, consul, kubernetes, hell - Docker swarm), we’d need far fewer sets of config than we have deployment environments. When there are only two or three you don’t need templating.

How often do you really have the same service deployed twice in prod and legitimately want it to work differently? I can count the scenarios I know on one hand and none of them have occurred for me in almost ten years, except read replicas and that shouldn’t be more than a few lines of config.

The article goes on to talk about Jsonnet, which takes the exact approach you describe - it generates JSON by aiming to be a "templated JSON" where the templating involves generating semantic objects, not strings.

Here's an example (adapted from some real-world code) where I specify the k8s cpu limit in one place, and then look up that info in several other places to avoid needing to change multiple values later:

    {
      local container = self,
      requests: {cpu: 5.5, memory: "2G"},
      limits: container.requests + {memory: "4G"},
      environment: [
        {
          name: "NUM_THREADS",
          value: std.toString(std.ceil(container.requests.cpu)),
        },
      ],
    }

Note how I can patch the container.requests object with an alternate memory limit, and how I can calculate an expression for the NUM_THREADS value in order to automatically set it to ceil() of the requested cpu.

(edited for nicer formatting of the code)

we're working on this: https://www.youtube.com/watch?v=4yepPOznakk

http://codesolvent.com/config-node/

I prefer https://github.com/taskcluster/json-e

Has the advantage/disadvantage that it's still valid json/yaml

I wrote a scary command line wrapper for it:

https://wryun.github.io/rjsone/

Has libs for Python, Go, and JS, and there's a bazel interface.

Baffles me. I don't like any language or file format where whitespace matters. Even Haskell bothers me in this regard. To me white space shouldn't add cognitive load. I want to look at the symbols not the formatting of the antisymbols to understand what is going on.

Write JSON and use your editor tools to format it with nice indentation, and you are sweet!

That said Yaml makes an excellent format for reading, but not for writing.

I've taken the strategy of emitting JSON, but accepting YAML input (maybe a restricted subset in cases where it's untrusted data). YAML can function as a super-set of JSON, so you can have comments in hand-written/modified data this way while emitting a simpler to parse data format.

It's a lot easier for end-users to read and write by hand. If that is not a requirement, a simpler machine-friendly format is likely better.

I think it is partly momentum - at this point most CI systems use YAML in spite of its insanity - even Azure Pipelines!

And partly it's because the syntax for multiline string literals is very minimal, which is kind of a nice feature for CI since you tend to have a lot of them.

It's still insane though. TOML is much more reasonable.

The best thing that ever happened to Cloud Formation was the new YAML syntax.

Writing YAML is easy with a good editor like VSCode. Install the YAML code outline extension for sugar; works great for OpenAPI specs. YAML flow style offers some good options for keeping the file compact.

Anyone working with cloudformation would wonder why we ended up writing json?

I really like TOML as a compromise between "looks pretty" and "parses well".

That being said, I find it funny that the author is crying afoul, saying that YAML isn't as well suited to be templated as JSON is, when XML had schemas sorted out ages ago.

People ran away from XML, saying it was this verbose, ugly, bandwidth hungry (back when we were sending XML over AJAX) behemoth (which it totally is)... but I think when your usecase is complicated enough to worry about templating, you should take a hard look at XML and ask whether it might suit your better.

Easier to read and write than JSON.

JSON requires constant quoting, can't support multiline strings, has no comments, has no/little typing (e.g., no datetime type). It's not good if a human needs to encode data.

For configs, I think TOML beats YAML hands down; I think YAML's spot is at encoding data structures that humans need to read/write.

I do agree that YAML, the spec, is fairly complicated. But YAML, as used in most projects, by most people, is not, and can be picked up fairly quickly. It is easier to visually read as it removes much of the clutter that would exist in the comparable JSON. It isn't typically necessary to know the entirety of the YAML spec to be useful with YAML, and most of the parts you won't know will get introduced by an obvious-looking sigil, which can be used to figure out what you're dealing with.

When I've actually sat with folks struggling with YAML, it's almost always in configuration tools, and it's also always around the templating bits. Ansible, in particular, has a bizarre templating: it happens after YAML parsing, which is not the mental model most people use when approaching it. I've also found that most of the people I've spoken to intertwine YAML and Ansible's templating functions, thinking they're one in the same.

I do not think Ansible makes good use of YAML: I would rather write task files in an actual programming language, since they are — at their core — a program. (The tasks do have some metadata attached to them, but the core task itself is a program. A function in some real language can get metadata attached to it in a number of ways, and that would be a better solution.)

Because json is poorly suited for config files, and it is very convenient to have config files that closely mirror apis, like kubernetes does. I've seen too many teams who try to use json for config end up developing yet another custom superset of json to allow comments or multiline strings or such. If you are going to use a superset of json might as well use an existing one like yaml or json5.

That said I agree with all the criticisms of templating yaml. We have to do the same (with helm and other tools), and I have pushed hard to adopt conventions that we only use flow-style and not block-style to avoid all the white space problems when splicing together chucks of yaml. And on the plus side we get trailing commas and other such niceties which don't exist in json and make it harder template.

Except that you cannot encode pretty valid Double.NaN or Date(), which is showstopper for many.

I remember mongodb started with json, but switched to in-house bson pretty early because of the json limitations.

> real selling point of JSON

Is that really the selling point? I interact with numerous services that speak JSON...and most of them aren't written in JS. YAML and JSON both have to be parsed to be used, even by JS. Otherwise it's just a string.

To me the real selling point in JSON is the dead-simplicity for humans to read and CHANGE. YAML is human-READABLE, but frankly I often screw up changing it because the formatting is a little too magical and I'm a little too unfamiliar. JSON is downright picky and obnoxious...but that makes it really easy to make a change. Screw up the quotes? It'll complain about the quote. Dangling comma? It will complain. Did I cut-and-paste from an HTML display that screwed up my whitespace by compressing everything down to a single space? Nothing cares.

When do you ever need to read YAML through vanilla JS? Install YAML/TOML/whatever parser and you're good.

I believe people are back to doing that for web stuff. Open up any modern web stuff and it looks like "const widget = <div>foo</div>".

Not making this up! https://reactjs.org/docs/introducing-jsx.html

I agree that simple YAML can be nice as a quick and clean tool, but Ansible is an example of everything wrong with how YAML is used. Layering program flow constructs like loops, variables, templates, references etc. are exactly the sort of abuses that make YAML feel awkward.

And you end up with program compiled to a config anyway, so with a proper toolchain it means your real config is the program.

People keep increasing complexity unintentionally precisely because they don't realize that code = data. There's no real distinction. Code is data is code.

You will end up having Turing completeness somewhere, it's just a matter of choosing (or blindly selecting, like most people do) where. For a popular product, it eventually gets embedded in the configuration language, turning it into half-assed programming language (see most web-related templating). For less popular products / more enterprise'y settings, you can probably get away with embedding the Turing-complete part in your bureaucracy. That is, I can't code my config to make it do what I want, but I can pay you to get developers to write some code and export it to the config language as a keyword. There's a spectrum to this, and tradeoffs galore.

But ultimately, YAML is nothing but a tree notation. Tree notation is enough to represent high-level programming languages. Lisp without parenthesis, if you might, or Python, if you squint your eyes.

If you are generating it, why use it at all?

> It's a solved problem

For who, exactly?

Based on the appearance of Helm, this is probably referring to Kubernetes, which really does have per-environment things that can't be represented in any form other than namespaces. For example, you have an app running (that's a deployment in k8speak). You want network traffic to get to this app, so you set up a load balancer. The load balancer config needs to know the name of your app (or more precisely, a selector for "pods" that are created by the deployment), which will change between environments, so now there is that common variable that has to be updated in both places. That's a simple example but is why people are templating their k8s configs. Yes, you could work around it by giving every environment its own namespace and using the same set of objects in every environment (differing only by namespace, which should probably be in the file... but since you can't edit the files, you can pass it in to kubectl probably), but there are other cases where even that doesn't work.

You do need some way of saying "this is the base configuration and this is what we change for staging and production". Helm is a way to do that, and a popular one, but it's pretty ugly. Hence this article.

I think his main argument is why are we using a text based templating when we can go up one step further and have language based templating.

like why have:

{"foo": "<%= bar %>"}

when you could just have

{"foo": bar}

the problem with text based templating is the templating language has to make a decision about escaping and it is sometimes the wrong one. for example rebar used an erlang haml [at some point... maybe they fixed it :)] which meant it escaped html special characters by default. but this makes almost zero sense when generating erlang configuration files.

i guess the reason that it is like this is because it is just easy and for most YAML/JSON/etc configuration files it is not a problem because you are basically doing static substitution or 'dynamic' substitution but with a safe range of characters. so the reason things are 'bad' is because the current solution works for 99% of the use cases and no-one wants to spend time fixing it when they could spend that time fixing a real problem. heh :/

Same here... just using the same language as the rest application, usually a file with constants or a class/struct/module/whatever.

the only reason config files in another language might be required is when you require configuring the application after it is compiled to native binary code.

Obviously this doesn't apply to all these applications written in scripting languages.

I do wonder how you create type-safe config files though. I currently have a `development.ts` and a `production.ts`, where the `production.ts` is only loaded when `node_env === production`. `production.ts` contains blank/default values and the file is overwritten on the server with production secrets.