The perfect configuration format? Try TypeScript
reflect.run
reflect.run
One of the arguments for TS in the post is that it can encourage DRY. It was DRY that made this TS-based kubernetes config so hard to follow, because simple repetitive statements got abstracted into parameterised functions, or classes to inherit from, or other things that are resolved dynamically at runtime. Not to mention that you're also going to start mixing in random dependencies from NPM to help abstract things further. This is great for application development but not so great for pure configuration.
A language designed for config, like Dhall or CUE, can give you much of what a typed language offers without letting you go crazy with a full-blown language runtime.
http://mikehadlow.blogspot.com/2012/05/configuration-complex...
Likely controversial, but I really like how Django does its settings. I think because Python code can start off just looking like a .ini file with variables and assignments, and get more complex only as required.
On the other end of my work, I'm a fan of having embedded firmware apply to as many devices as possible, and then specialise with configuration settings from EEPROM.
And from some years of experience with configuration management systems, configuration creation can accrue a surprising amount of complexity over time. Mostly because unlike the code they support, a config management system has to support many versions of the code and multiple versions of a configuration at the same time.
Essentially you'll generate the config from the code, but you can always view the raw output side by side.
(...No one asked, but here are my $0.02 ;-)
You need typescript in your environment to use typescript!
The four formats the OP listed can be read by anything from Ada to EXCEL. But typescript can only be ready by a TS build flow.
In fact, I think JSON was a bridge too far. A config file shouldn't need a compiler.[1]
FOR GENERAL PURPOSE: big no.
BUT: If this is purely for TS projects where the tooling already exists, then OK. I'd use it.
EDIT [1] I'm referring to the article's early suggestion to run JSON through JSMin to support comments. So transpiling is more apt, not compiling.
JSON just needs a parser, like most other configuration formats. It's purely declarative.
Does JSON need a compiler?
Edit: It’s not that accurate. It’s just math used in a way to compose parsers using parsers.
I'm dense: what's a combinator?
edit: Forgot partial application. Not strictly required, but super handy.
Recently I took what learnt in the last 2 years using jkcfg/typescript and taken it to Deno in form of an opinionated port of jkcfg called: dxcfg[2]. Its early days, but I would bet on Deno/typescript for future ops configuration.
[1] https://jkcfg.github.io/#/ [2] dxcfg: https://github.com/dxcfg/dxcfg
You can get many of the benefits but also run custom validations (dictate how two keys in a map relate to eachother, or force a be of a certain shape instead of just a string).
Here's the example from the script: [0]
[0] - https://malli.io/?value=%7B%20%3Alocale%20%22en-US%22%0A%20%...
Please use a language designed for configuration, but still sane. There are several if we leave the JSON-inspired family. Starlark, pystachio, Dhall. All are restricted, but allow functions and code reuse. All are simple enough to write a new interpreter for.
And they absolutely use TS for the std library, just not in the runtime itself
What exactly do you mean by this?
> And they absolutely use TS for the std library, just not in the runtime itself
Yes, internally. The code that actually makes up deno itself is just plain Javascript.
And JSON parsing is a minefield. http://seriot.ch/projects/parsing_json.html
I don't know of any notable S-expr based config languages, probably because they are so easy to parse, people just roll their own, leading to balkanization.
I personally find it silly to need more than that. YAML, JSON, etc. What are we even doing here, besides trying to make life more difficult? Are we doing configuration or are we doing plugins and calling it configuration?
We'd all be better off if we just admitted we really want an extension language and went with embedded Lua or Scheme. With Dockerfile and others you have to run a separate lint program, assuming anyone even bothers doing that. At least with a proper language you would have built-in robustness. Most of these configuration formats/DSLs are half-assed and unnecessary anyway.
The easier a config language (syntactically, but also in terms of capabilities), the easier it is to write a parser in Go, and Rust and Java etc. All of this particularly matters for libraries or in larger codebases that are multi-lingual.
Using starlark here because I'm most familiar with it.
Starlark's `load()` statement is "find the file at this exact literal path from the workspace root" (no relative paths allowed either). Typescript/Javascript's import is much more complicated. If you encounter an error in Starlark, you fail right there, no exception handling to implement.
Starlark doesn't allow recursion, has an extremely limited set of built-in types and functions, and that means it is very easy to reason about, and even a naive interpreter is pretty fast at evaluating it. I'm also reasonably certain about writing a simple interpreter for it in a week if I had to port it to a new language.
Of course, I view configuration languages on a continuum.
I'd definitely want a pomodoro timer to just use a .ini file. TOML/YAML is great for where you are just going to have a bunch of plain type assignments. No JSON, because comments.
Once you want to write larger configurations, and you need to provide APIs to the user in the language, I'd pick Starlark/Pystachio because they are still intentionally restrictive and Python is one of the most popular languages out there. At this point, you are probably trying to configure something like a large build system (Bazel) or some kind of deployment cluster (say Kubernetes) and you can start providing all sorts of utility libraries (either via bindings to your programming language, or as Starlark libraries). But due to their restricted nature you can still do all sorts of bounded analysis of the evaluation. You can remove things like random() because it's a bad idea.
Yes there are use cases where you want a full blown language to configure something, but those are extremely rare and at that point I think you are no longer "configuring" but just "programming". I very much prefer to keep these 2 things separate.
On their homepage they call it a "programmable configuration language that you can think of as: JSON + functions + types + imports"
Pros: not Turing complete but quite powerful, nice type system and functions. For example one can write a type safe config migration from v1 to v2 (and check statically that it works), exports to JSON, YAML etc.
Cons: uses Haskell like syntax, no bindings to various languages. You basically need to ship the dhall binary to convert the config to something else that your target application will consume.
There are bindings to some languages: Haskell, Clojure, Go, Rust, Ruby and Purescript. https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...
Depends on what language runtime can do with those responses. Arbitrary code execution in a config (a la Django settings and indeed OP’s idea) can do much more damage, but even that doesn’t scare many people as this kind of configuration is considered an inherently trusted source.
It also runs Typescript directly.
My understanding of the situation right now is that by default, Deno runs TS files by transpiling using SWC, which strips types without checking them, and then sends the result to V8. Optionally you can enable type checking which invokes TSC (with no emission) before SWC. All this is done "magically" from the user's point of view.
It would be cool if tsc had a flag —sandboxed or similar that does not allow any sideeffects (fs access, output, forking, net requests, etc)
// this returns in 0 ms
export const config: any = {
get value() {
while (true) {}
}
} as const;
// accessing config.value runs forever
console.log(config.value);
The point of having a config file in the first place was so that it could be replaced, at start time or better yet at runtime, without touching the rest of the code, such that the same code can be parametrized and instantiated as you wish. When you use a fully-powered language, this becomes a matter of discipline instead of something guaranteed by the format.
ALL of the other competing config file formats are attempts to walk this line between too little expressive power and too much expressive power, such that people naturally separate their code and config. Typescript-as-config just abandons this goal; it's not a new idea, just one that has been rejected over and over again by each new format because everyone else saw the benefit of limiting expressive power.
1. A constrained specification where its difficult(but not impossible) to shoot yourself in the foot.
2. Devops: as in equitable participation of operators and developers in maintaining configuration on a build it, run it basis.
Languages like skylark/starlark, dhall, cuelang, jsonnet promise a constrained spec(for operators) while being expressive enough(for developers). But after using skylark, jsonnet and dhall in production, personally I haven't seen enough adoption in terms of developer contribution so the promise of expressiveness which is attractive enough for developers hasn't been true for me.
At my current workplace, we have taken a middle approach. We use typescript to generate/emit yaml/json and use "official" validators to validate this configuration. The developers get a familiar and "blessed" frontend to the configuration while the operators can still enforce schema validation, resources and security policy on the generated config.
But yes, if one could import this base typescript package directly in their application, and generate the configuration which can be then picked up by the operator pipeline and isapplied to the infrastructure. That would be bad. Really bad.
I would like to see an option to whitelist syntax/types/keywords in Typescript. Allowing only const, dicts, arrays, assignments (but not while). It could just abort if something other than the whitelisted appears.
It could get rid of most of this problems, and not allow network or infinite looping (or file access, and others). Of course it would not be perfect, but could be good enough.
As I understand it it's essentially the Google Configuration Language (GCL) author responding to Hashicorp Configuration Language (HCL), with language development modeled after Golang. Still in early stages but very promising.
It also becomes trivial to parameterise config so I can setup the same general Cloudfront distribution once and tweak it based on the individual application. It's open to abuse but it's better than interpolating values into Cloudformation templates.
The only problem with AWS CDK is that sometimes the type hints aren't clear enough in Python, that part could have been a lot smoother imo.
For me, one of the most important features of a configuration language is the ability to directly refer to other values of the configuration, and more importantly, in the case of a configuration split in separate modules, to refer to values that could be provided by the other modules.
Essentially, this means that the final configuration should be the fixed point of a function, that the user would provide the configuration as a definition of this function (or in the case of multiple modules, as multiple definitions that would be merged using function composition), and that the evaluator would be able to determine the fixed point of this function.
I feel like this kind of configuration (used notably in NixOS) provides a lot of what's missing in traditional configuration languages, makes it easier to compose configuration modules, without requiring to write the configuration in a Turing-complete language.
Under some circumstances, configuration needs to be generated dynamically so you need a programming language — and TypeScript would be fine for that. But a "perfect" configuration format isn't perfect if it can run arbitrary code.
One day one of the other developers introduced a change to the configuration that had a side effect. It made sense; after all, the configuration is read at startup, so surely it makes sense to put code there that you want to have run at startup? You can imagine my surprise when I made change that read the configuration again after startup.
Configuration should be purely functional and idempotent, with no side effects, or as you say, inert.
* They can be non-deterministic (do a different thing each time you run them).
* They can be non-hermetic (access stuff in the environment you don't know about).
* They can do naughty security things.
* You can't present GUIs of them because they aren't declarative.
All but the last one don't exclude programming languages. Here's an interesting project to make hermetic deterministic Javascript (Typescript support is planned):
For the sorts of places where you don't have a GUI for the settings anyway (which is the common case) I think it makes loads of sense. It beats making the kind of declarative programming languages you see in YAML files.
And what do we do when we want the Cartesian product of two sets? Ahem Terraform.
[plugin "foo"]
binary = "C:\\Program Files\\Foo\\Foo.DLL"
- types if you want them
- IDEs/tooling/a kind of UI-out-of-the box
- often other useful toolchain programs for packaging, management, etc
- a well defined syntax
- and the big bertha for REALLY COMPLICATED config: autocomplete in your tool/ide/etc
- and yeah, turing complete dropouts, meta configuration frameworks for generating config
I'd rather choose a bad idea than having to reimplement a Turing-complete programming language in YAML a few years down the line.
The reason you don't want to use a programming language for configuration is that configuration shouldn't be programming - configuration should be idempotent, it should describe a state, and it should contain only data, and structures of data (arrays, maps, sets, what have you.) No functions, no classes, no mutability.
If you use a programming language as configuration, the temptation to make your configuration more and more complex, and add features and "shortcuts" to integrate it more deeply with your application will be irresistible, and eventually your config will need its own config, which will eventually grow to need it's own config, ad infinitum.
And I say this as someone who advocates using Lua tables for config in gamedev, if you're not willing to be disciplined with avoiding Turing creep at all costs, don't do it.
I also found that the user documentation was just a skosh above useless. It felt like it was more concerned with not being wrong than being helpful. Reminiscent of older javadocs in the java collections library.
The same could be said of many build systems, I suppose.
Not just for configuration, but for things like test fixture data, and even data assets like i18n translations, or basically anything that projects might commonly use JSON for, internally. It's just so much easier to create, edit, manage, and deal with in TypeScript.
But it is hard to imagine TypeScript-as-config taking root in projects that don't already use TypeScript.
It's basically Json with comments, trailing commas, and unquoted object keys (and some other minor things).
Much nicer than real json as a config format, but still simple and declarative, not requiring an interpreter or being turing-complete.
Here's an example: https://github.com/jkcfg/jk
That beats having to learn an entirely new language.
I know: It's a binary format. And you need to add a library to read it.
BUT
- Your configuration _has_ data types.
- You can have a copy with an old config within the file easily (create table backup_config as select * from actual_config; ).
- The file is small.
- Encryption is built-in if you need it.
- And it's programable
I’m not sure I’d recommend node-config (it was chosen by a past team). Its magic undermines the confidence in type safety provided by choosing TypeScript in the first place. But it does give some good baseline conventions one can apply without all the magic.
The point of having configuration is to gain the ability to configure software without rebuilding in my mind. Maybe I'm missing something.
I think the whole approach is just fundamentally misguided. Each of these systems should probably use 2-3 different languages & formats, not just one. And especially not YAML. Ugh.
So you're right, config files should not be code. Unfortunately, much of "modern" config management involves writing programs in config file formats, which is also extremely bad in a whole different way.
Yes.
> So what use is a configuration file if I need to run it through the build pipeline?
Compilation doesn't necessarily happen in a “build pipeline”; typescript is available as a library.
> The point of having configuration is to gain the ability to configure software without rebuilding in my mind.
There are many different purposes of configuration, and they are sometimes in tension, which is why there are so many configuration formats (spanning from limited languages like TOML through general purpose programming languages, whether interpreted or compiled) with an ebb and flow as to which approaches are more and less popular over time.
You can even get by without including TypeScript, if you are willing to just strip out the type information (there are various libraries that do this).
You're right, the summary presents this configuration as a stop-gap, a possible pre-cursor to a better, future JSON.
This might not even be warranted when inter-process communication is a primary concern. After all, there's nothing inherently wrong with using whatever data transfer mechanism your standard lib provides.
It's not just protobuf though. Almost every data format for serializing structured data suffers from pushing a jungle on you when all you asked for was a banana.
The least invasive format I know of is FlatBuffers[0]. Doesn't push RPC on you. Allows you to parse a message even if you do not know this specific message's data structure in advance (like JSON).
In my experience, a lot of the validation needs to be done at runtime anyway as type checking alone won't allow you to e.g. validate if a string is a valid regular expression. Also, I think using TypeScript for configuration requires you to compile & interpret the configuration file in order to check it and obtain the actual configuration values. Not sure if I like that as it requires bundling the entire TypeScript compiler with your program.
Also, it’s Turing-complete, so you are enabling future developers or users to shoot themselves in the foot and write something that does not halt already at configuration stage.
To me the perfect language for configuration seems to be Dhall, but it doesn’t seem to have gained any traction.
[number, number, number]
Is that a feature of other languages?
As to Dhall, for example, I could validate such a constraint (and more) using a function I can be reasonably sure is safe and won’t halt (i.e. without all the drawbacks of TS and other general-purpose languages).
Where execute is to sequence a series of effects that are described in a file.
That's not arbitrary code. That's my compiled, tested and verified code.
TOML cleans up most of that, though.
This is the proposal of TFA:
export const config: MyAppConfig = {
locale: 'en-US',
timezone: 'America/New_York',
logLevel: 'info',
environment: 'local',
}
LOCALE en-US
TIMEZONE America/New_York
LOGLEVEL info
ENVIRONMENT local
It is beyond me how could any serious engineer honestly find any advantage in the first version.
A.B value
C value
D.E.F value
D.E.G valueThat problem doesn't apply to arbitrary configuration, but only to configuration files that represent package build and installation metadata. You do see this issue with things like Gradle, though, where your build.gradle file can represent the dependencies of the package being built, but the build file itself may have dependencies since you can run arbitrary Groovy code and import any package that can be contained in a jar file. It gets around this because Gradle itself can just download those dependencies before trying to execute the build script, which would be perfectly possible for a real Python build system, but not when setup.py is just being executed by the Python interpreter, which doesn't do dependency management and expects imports to already be available in the system import path.
Also reinvent the wheel. They can list four other configuration formats du jour, but conveniently leave out s-expressions, which have solved most problems people bring up over half a century ago.
... but as a rule of thumb, if your config language is Turing complete, you're setting yourself up for future pain as complexity of config grows with time and someone makes the bad decision to take advantage of that Turing completeness...
But json backed by a typescript interface definition could be a useful middle ground.
You're still writing json but you get nice code completion and realtime validation of the config file.
For broader use, I would like to see native support or an official extension for creating dependent types or specific constraints.
Things like number ranges, or the basics from JSON Schema, etc.
[0] - https://jsonnet.org/
In practice I think it works fairly well.
As a bonus, you might not need validation and schemas, too.