YAML: It's Time to Move On
nestedtext.org
nestedtext.org
I think it's crazy that when I add a string to an inline list, I may need to convert that inline list to a list because this string needs different handling. I think it's crazy that "convert an inline list to a list" is a coherent statement, but that is the nomenclature that they chose.
I don't like that a truncated document is a complete and valid document.
But what is most unappealing is their whitespace handling. I couldn't even figure out how to encode a string with CR line endings. So, I downloaded their python client to see how it did it. Turns out, they couldn't figure it out either:
>>> nt.loads(nt.dumps("\r"),top="str") '\n'
name: Install dependencies
run:
> python -m pip install --upgrade pip
> pip install pytest
> if [ -f 'requirements.txt' ]; then pip install -r requirements.txt; fi
We all have to deal with this, but it is beyond stupid.
[Install Dependencies]
run=/path/to/install-script
In fact, if I really wanted to test my tests, I'd say that directly testing the corresponding clause is the more comprehensive approach. For example, what if someone accidentally changes the line to read:
run=/path/to/install-scriptq
So let's keep inline scripts, they are very reasonable methods for just a few commands.
I don't have to re-read manuals describing how to do if/else in Ruby or Java or Lisp, but as soon as yaml and some 'devops' tooling is involved, I have to constantly jump back and forth between the reference and my config.
The main point being that the problem isn't the file format but the products that continue to push it, presumably because hacking stuff on top of `YAML.parse` is less effort than designing something that fits the purpose.
For build systems, I always liked the idea of Gradle where the core functionality was simple and declarative, but with the option to use a real programming language for things that weren't simple. For example, integrating installers or form builders (pre-processing) into a build are things I would consider non-trivial if there aren't official plugins, but it was still relatively easy to do with Gradle.
The biggest problem I always had with Gradle was that I didn't like Groovy and I always though there was a missed opportunity to have a statically typed build system with a solid API/contract and all the fancy tooling like auto-complete that you get with statically typed languages.
I see JSON5 mentioned a lot in the comments. In terms of CI / build systems, I feel like something built with JSON5/TypeScript could be really good. I'd be really happy using TypeScript for configuring things like build systems where there shouldn't really be an argument for needing it to be usable by non-programmers.
Personally I feel like I've spent way to much of my life debugging YAML syntax issues.
I do find it curious that a lot of these tools get seen as basic task runners despite offering much more potential.
* https://ilya-sher.org/2018/06/30/terraform-becomes-a-program...
* https://ilya-sher.org/2018/09/15/aws-cloudformation-became-a...
(There are workarounds, and off-hand I can't think of a great example, but bear with me.)
In the end I wrote a simple console-based mail-client, which used a Lua configuration file. That would generally default to using hashes, and key=value settings, but over the time I used it things got really quite configurable via user-defined callbacks, and functions to return various settings.
For example I wrote a hook called `on_reply_to`, and if you defined that function in your configuration file it would be invoked when you triggered the Reply function. This kind of flexibility was very self-consistent, and easy to add using an embedded real language.
Later I added some hacks to a local fork of GNU Screen, there I just said:
* If the ~/.screenrc file is executable, then execute it, and parse the output.
That let me say "If hostname == foo; do this ; otherwise do this .." and get conditionals and some other things easily. Another example was unbinding all keys, and then only allowing some actions to be bound. (I later submitted "unbindall" upstream, to remove the need for that.)
So you write yaml by hand for trivial cases, but once it get complex, you can just drop back to shell/python/ruby/node/whatever, implement any complex logic, and serialize results to plain yaml.
I still don't know how to do arrays in yaml.
Is it new line, tab to same indent or same indent + one space, do I need a dash? Does a dash make it an array or an object?
It's just simply not that big of a deal to add a few quotes and braces to make everything make sense.
The only real issue with Json is the lack of comments and strictness about extra commas.
Me either. If your documents have this property you're likely to tempt people to start trying to process partial documents.
When they do that, they violate Full Recognition Before Processing and likely there's a latent security bug as a result.
Who are you suggesting the python client belongs to, who is 'they' in 'their'?
Stepping back a bit, YAML is good enough, and this problem has been incrementally bikeshedded since at least the 1970s, it is time to move on. Human-convenient interfaces (like YAML, bash, perl) are fundamentally messy because we are messy. They're prone to opinion and style, as if replacing some part or other will make the high level problem (that's us) go away. Fretting over perfection in UI is an utterly pointless waste of time.
I don't know what NestedText is and find it very difficulty to care, there are far more important problems in life to be concerned with than yet another incremental retake on serialization. I find it hard to consider contributions like this to be helpful or represent progress in any way.
If you can write a bad YAML document because of those mis-features/edge cases, I'd say you've already lost.
Humans are messy, but at the end of the day the data has to go to a program, so a concise and super simple interface has a lot of power to it for humans.
Working at a typical software company with average skill level engineers (including myself), no one likes writing YAML. But everyone is fine with JSON.
I think it's a case of conceptual purity vs what an average engineer would actually want to use. And JSON wins that. If YAML was really better than JSON, we'd all be using that right now.
So does it really matter if YAML is superior if >80% of engineers pick JSON instead?
Conversely, if a markup language is strict enough to prevent every inconsistency, then it's not powerful enough or too cumbersome to use to be generally useful.
If you want conceptual purity (and far fewer footguns), take Dhall.
Nah, in the 1970s we had Lisp S-expressions that completely solved the problem, and everything since then has been regressions on S-expressions due to parenthesis phobia.
After hearing that thing about the country code for Norway, I became convinced that YAML has to just die. Become an ex-markup language. Pine for the fjords. Be a syntax that wouldn't VOOM if you put 4 million volts through it. Join the choir invisible, etc.
This is good: https://noyaml.com/
Erik Naggum had a notoriously NSFW rant about XML (over the top even for him) that I better not link to here, but lots of it applies to YAML as well.
(Sarcasm, just in case.)
Stuff like escape chars were well specified in Lisps of the 1970s (at least the late 1970s), including in Scheme (1975). Floating point conversion is a different matter (it was even messier in the pre-IEEE 754 era than now) but I think the alternatives don't handle it well either. You probably have to use hexadecimal representation for binary floats. Maybe decimal floats will become more widely supported on future hardware.
A type-checked approach can be seen in XMonad, whose config files use Haskell's Read typeclass for the equivalent of typed S-expressions.
1. Use only alists for maps because they prevent off-by-one errors.
2. Allow plists because they're less verbose than alists and use reader macros to distinguish them, and allow the reader macro definitions to be in the same file.
Most of the time I use option 1 because it's simpler.
There's no canonical YAML implementation, and YAML spec is enormous (doubly so if you need to work with stuff like non-quoted strings etc. )
The formal grammar counts as canonical and several implementations are derived from it: https://github.com/yaml/yaml-reference-parser
YAML is patently unsuitable for any use case where the resulting output may require hand editing.
YAML ain't markup language.
I don't know what to make of this statment, it has so much handwaving built-in. The most charitable interpretation I can find is that by 'Human-convenient' you simply meant the quick-and-dirty ideology expressed in Worse Is Better: Does job, makes users contemplate suicide only once per month, isn't too boat-rocking for current infrastructure and tooling.
Taken at face value (without special charitable parsing), this statement is trivially false. Python is often used as a paragon of 'Human-convenience', I sometimes find this trope tiring but whatever Python's merits and vices its _definitely_ NOT messy in design.
Perl is the C++ of scripting languages, it's a very [badly|un] designed language widely mocked by both language designers and users. Lua and tcl instead are languages literally created for the sole exact purpose of (non-) programmers expressing configuration inside of a fixed kernel of code created by other programmers, and look at their design : the whole of tcl's syntax and semantics is a single human-readable sentence, while lua thought it would be funny if 70% of the language involved dictionaries for some reason. These are extremely elegant and minimal designs, and they are brutally efficient and successful at their niches : tcl is EDA's and Network Administration's darling, and lua is used by game artists utterly uninterested in programming to express level design.
'Humans are messy' isn't a satisfactory way to put it. 'Humans love simple rules that get the job done' is more like it. But because the world is very complex and exception-laden, though, simple rules don't hug its contours well. There are two responses to this:
- you can declare it a free-for-all and just have people make up simple rules on the fly as situations come up, that's the Worse Is Better approach. It doesn't work for long because very soon the sheer mountain of simple rules interact and create lovecraftian horrors more complex than anything the world would have thrown at you. Remember that the world itself is animated by extremely simple rules (Maxwell's equations, Evolution by Natural Selection, etc...), it's the multitude and interaction of those simple rules that give it its gargantuan complexity and variety.
- you stop and think about The One Simple Rule To Rule All Rules, a kernel of order that can be extended and added to gradually, consistently and beautifully.
The first approach can be called the 'raster ideology', it's a way of approximating reality by dividing it into a huge part of small, simple 'pixels' and describing each one seperately by simple rules. I'm not sure it's 'easy' or 'convenient', maybe seductive. It promises you can always come up with more rules to describe new patterns and situations, and never ever throw away the old rules. This doesn't work if your problem is the sheer multitude and inconsistency of rules. The second approach is the 'vector ideology', it promises you that there is a small basis of simple rules that will describe your pattern in entirety, and can always be tweaked or added to (consistently!) when new patterns arise, the only catch is that you have to think hard about it first.
Rather short sighted and dismissive to a successful programming language that's evolved over 20+ years. Lua is a great general purpose programming language that specializes not in "game making for non-programmers" but in ease of embedding, extension/extensability, and data description (like a config language). There's a whole section in Programming in Lua[1] to that effect. The fact that it's frequently used in games is credit to it's speed, size and great C API for embedding, not because of any particular catering to game designers.
JSON schemas exist and they’re ok for relatively simple things. For more complex cases I find myself wishing I could just turn Typescript into some kind of schema validation for JSON.
Not sure if this is what you're looking for, and whether it's powerful and expressive enough for your use case, but you can use typescript-json-schema¹ for this, and validate with eg ajv.
As an alternative, the encoding/decoding roundtrip using protobuf seems reasonable to me, catches the footgun of using floating-point version numbers (it becomes a parse error), whitespace/multiline concatenation being more obvious, and allowing comments (compared to JSON):
( cat << EOF
# yes, comments are allowed
name: "Python package"
on: "push"
build {
python_version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
steps: [
{
name: "Install dependencies"
run:
"python -m pip install --upgrade pip\n"
"pip install pytest\n"
"if [ -f 'requirements.txt' ]; then pip install -r requirements.txt; fi\n"
},
{
name: "Test with pytest"
run: "pytest\n"
}
]
}
EOF
) | protoc --encode=Config config.proto | protoc --decode=Config config.proto
name: "Python package"
on: "push"
build {
python_version: "3.6"
python_version: "3.7"
python_version: "3.8"
python_version: "3.9"
python_version: "3.10"
steps {
name: "Install dependencies"
run: "python -m pip install --upgrade pip\npip install pytest\nif [ -f \'requirements.txt\' ]; then pip install -r requirements.txt; fi\n"
}
steps {
name: "Test with pytest"
run: "pytest\n"
}
}JSON schemas exist and can be applied to yaml and this is supported by many editors. For example this vscode extension: https://marketplace.visualstudio.com/items?itemName=redhat.v...
It's strange to see so many complains about "missing tooling" that actually exists and is well supported.
Unfortunately YAML already got type support, which made it easier to roundtrip, but also insecure. Creating a type calls constructors with possible insecure side effects. Which was eg used to hack Movable Type.
(I agree with you about comments though)
JSON5 has many advantages:
* Superset of JSON without being wildly different. I know YAML is a superset of JSON but it's completely different too. Insane.
* Unambiguous grammar. YAML has way too many big structure decisions that are made by unclear and minor formatting differences. My work's YAML data is full of single-element lists that shouldn't be lists for example.
* Comments, trailing commas
* It's a subset of Javascript so basically nothing new to learn.
* It has an unambiguous extension (.json5). I think JSONC would be a reasonable option but everyone uses the same extension as JSON (.json) so you can never be sure which you are using. E.g. `tsconfig.json` is JSONC but `package.json` is just JSON (to everyone's annoyance).
* Doesn't add too much of Javascript. I wouldn't recommend JSON6 because it's just making the format too complicated for little benefit.
- it has good editor support (VsCode) - has comments support - Support jsonschema
Only thing missing is trailing commas, but i would rather live without trailing commas than tooling support
{
"name": "my-tool",
"#comment-1": "Don’t change the version!",
"version": "42.1337.0"
}Also, this doesn't let you put comments at the top of the file, or before a list item, or at the end of a line.
If you're going to change your JSON tooling to handle comments of some kind, you might as well go all the way to JSONC.
This situation makes me feel rather silly
(and it doesn't have to be comment-less... JSON with comments is a thing and VSCode has syntax highlighting for it - just strip out the comments before parsing).
Aren't we past basic false dichotomies?
Sure, why not? XML rocks. I'll take it over JSON for many purposes.
Yaml and toml are fine if you keep it simple. JSON direly needs comments support (but of course wasn’t designed to be used as a human config file format so that’s kind of on us). And not just “Jsonc that sometimes might work in places.”
Beyond that, I think we generally have all the things we need and I don’t personally think we need yet another yaml. =)
* JSON streaming is a bit of a mess. You can either do JSONL, or keep the entire document in memory at once. I usually end up going with JSONL.
* JSON itself doesn't permit trailing commas. I can measure the amount of time that I've wasted re-opening JSON files after accidentally adding a comma in days, not hours.
* JSON has weakly specified numbers. The specification itself defines the number type symbolically, as (essentially) `[0-9]+`. It's consequently possible (and common) for different parsers to behave differently on large numbers. YAML also, unfortunately, has this problem.
* Similarly: JSON doesn't clearly specify how parsers should behave in the presence of duplicate keys. More opportunity for confusion and bugs.
Arrays with hole are a JS-only feature
I also started advocating in-language configuration files (Python for Python, but also Lua for Lua, etc) a number of years ago because it lets you do really useful things (like functionally generating values, importing shared subsets of data, storing executable references, and ensuring that two keys return the same values without manual copy/paste) all without needing to spec and use Yet Another Thing™ that does only a fraction of what the programming language you're already using already does.
Things that get compiled can't really use it without recompilation.
While true, parsing it is still a minefield because it's very underspecified: http://seriot.ch/projects/parsing_json.html
Clean, of course to them means some bizarre aesthetic notion of removing as much as possible. Only it’s taken to an extreme. I wonder if the same people also think books would be better with all punctuation be removed to make it look “clean”?
It’s unhealthy minimalism, causes more problems than it solves. As soon as I see a project using YAML I cringe and try to find an alternative because god knows what other poor choices the developer has made. In that sense, YAML can be considered a red herring and I’m usually right. The last project I used that adopted an overly complex and build-breaking YAML configuration syntax had other problems hiding under the covers, and in some cases couldn’t parse it’s own syntax due to YAML’s overly broad but at the same time opinionated syntax.
Just say no to YAML.
Only AFTER YAML had been around and in use for a few years did those geniuses actually realize that they had made a mistake in naming it something that it's not, and retroactively changed the name "YAML" to mean "YAML Ain't Markup Language", which was a too clever by half way of whitewashing the fact that they originally CLAIMED it was "Yet Another Markup Language", since they had no idea what a markup language actually was.
I prefer to use markup languages and data definition languages that were designed by people who are situationally aware enough to know what the difference between a markup language and a data definition language is, please.
Hard pass on YAML, whatever it stands for this week.
Here were my thoughts on the options.
JSON - No comments -> impossible
XML - Unreadable
YAML - 2nd place. Meaningful indentation also made me worried someone was going to not understand why their file didn't work. The lack of quotes around strings was frustrating.
TOML - 1st place. Simpler than YAML to read & parse. It truly seems 'obvious' like the name says.
I haven't encountered any situations where I wish I had more than TOML offers.
Check this thread:
https://news.ycombinator.com/item?id=17523194
I don't see Kubernetes switching to TOML anytime soon!
More people should give it a try. Very reminiscent of old Windows INI files and Java properties.
All we need is to revise official JSON standard(ECMA 404) to include comments.
That would be a step back for GitLab CI, GitHub Actions, Kubernetes, Google App Engine, and a bunch of other projects which use YAML and seldom encounter the Norway problem. https://hitchdev.com/strictyaml/why/implicit-typing-removed/
It is pretty niche, but attempts to improve upon JSON in a multitude of ways, one of which is the support for comments: https://spec.json5.org/#comments
http://hyperrealm.github.io/libconfig/libconfig_manual.html#...
It does need an XPath traversal and search query format for application use and data references.
I recently read a piece by Joel Spolsky that resonated with me (even though my career is not nearly as long as his).
> I took a few stupid years trying to be the CEO of a growing company during which I didn’t have time to code, and when I came back to web programming, after a break of about 10 years, I found Node, React, and other goodies, which are, don’t get me wrong, amazing? Really really great? But I also found that it took approximately the same amount of work to make a CRUD web app as it always has, and that there were some things (like handing a file upload, or centering) that were, shockingly, still just as randomly difficult as they were in VBScript twenty years ago. [0]
It makes me wonder if we're really focusing on the right stuff. Maybe there's lower hanging fruit somewhere that's more valuable than focusing on fundamentally subjective things like syntax.
[0]: https://www.joelonsoftware.com/2021/06/02/kinda-a-big-announ...
It’s a deterministic subset of Python. This means that if you have complex or repetitive configurations, you can use loops and functions to structure them. But it’s impossible to write an infinite loop or recursion.
As a sysadmin, YAML seems nice until you have actually done anything more advanced with it. See Julien Pivotto's presentation about some of its pitfalls: https://www.slideshare.net/roidelapluie/yaml-magic?next_slid... Btw. Jsonnet doesn't seem too bad either: https://www.youtube.com/watch?v=LiQnSZ4SOnw and here some examples: https://jsonnet.org/ but in my book, EDN still wins.
Or if you really must, then simply interrupt processes that loop infinitely, and fix the bugs that caused it.
You know, like you already do when you have an infinite loop.
Infinite loops are not the end of the world, you know. Processes can be interrupted, and computers have reset buttons.
The underlying conundrum is:
- systems need to be configured,
- human-readability is obviously necessary at some level,
- configuration is often very "compressible" (needs loops, needs variables to be maintainable), but
- system-writers don't know the structure of your data, the axes on which you'd want to compress things, the best abstractions for you.
Templating languages are an obvious direction, but they're uniformly bad. If they have limited expressiveness you'll run into the limits. Maybe there are templating languages with good unit testing frameworks, but I haven't seen them. "Look at the expanded diff" doesn't scale. And generating gobs of human-readable "data" (in a format that supports comments!) is very wasteful.
It's still resources to detect loops, it often involves introspection or privileged views; it's simply easier to prevent loops.
If it has first-class functions, how can you avoid infinite recursion? Like, what stops me from running the omega combinator in it? This is why Meson (a similar language) does not allow those kinds of shenanigans, to keep the language non-Turing-complete.
It's a cool idea to do configuration in a subset of Python but now you have to go implement that subset in every language.
I will look into this further.
Starlark is indeed deterministic and guaranteed to terminate (the Go implementation has a flag that allows recursion, but it's off by default), but these are two orthogonal properties.
I like CSV (and similar delimited files) it's less verbose than anything else for tabular data.
I like JSON for data transfer, you know the data types, it's succinct, and readable.
I personally don't need anything else.
I'm more likely to Yacc my own config format than use YAML or JSON personally.
JSON is great as an output format for data though.
Schema validation, code completion on IDEs, endless amount of tooling including graphical visualisation, a language for data transformation and queries, and.... wait for it... comments!
XSDs is something I think people need to pay more attention to when dealing with XML; the type system that the W3C XSD standard lays out (when used effectively) really does relieve much of the pain that people experience with XML.
My benchmark for yaml/JSON alternatives is "how would I feel explaining it to a busy, sceptical client?"
If the intended audience is purely developers, then sure. JSON (with the addition of comments and trailing commas) is just fine.
White space has the additional advantage of agreeing with itself. Other demarcations can have issues where the indentation and the structure contradict each other.
Please stop making code easy for non-coders. They don’t want to read it. They never did. They just want this damn box to work.
My benchmark is this: can an autoformatter do its job every time without breaking something that's technically working right now but possibly formatted wrong?
Every data format that cannot comply with this contains in it a huge waste of time. Even as a python programmer, I extend the same rule to programming languages.
I'm not clear exactly what the "it" here refers to but as I mentioned in other comments it's fairly self-evident that indentation is easier to visually parse than braces. A simple thought experiment - would you find it easier to skim read code where the indentation was consistent with the bracing or where it was inconsistent? Your brain registers the indentation first and you only resort to counting braces if there's a reason to doubt the former.
If it was code, an indentation error would often not compile or show errors or fail tests.
Configuration in YAML is much worse: most of the time an indentation error goes undetected until an application starts misbehaving.
Significant whitespace is perfectly ok for code but a huge footgun for YAML
But I find it incredibly annoying to estimate indentation when lines are wrapped in an editor (or webpage). Or, to a lesser extent but still throws me off, when multiple blocks end at the same line. Or when pasting blocks into another block, and having to double check to make sure the indentation was carried over correctly. I like editors that visually show indentation characters.
Or, NODE <name> .. END .
it makes a difference on how it is parsed, so it's more than a devs preference
Truncate YAML and in most cases you still have valid YAML.
For example, to set the maximum message size to by 10Mb and to set the timeout to be 30 seconds:
echo 10000000 > /var/qmail/control/databytes
echo 30 > /var/qmail/control/timeoutsmtpd
Dictionaries are just subdirectories with one file per entry, for example this is how aliases are defined to qmail:
echo fred > /var/qmail/alias/.qmail-postmaster
echo fred > /var/qmail/alias/.qmail-mailer-daemon
DJB also created a simple, portable encoding for serializing data called netstrings, see [2]. XML, YAML, JSON, TOML, and INI files all have some advantages over netstrings, but netstrings are simple to understand and simple to parse correctly.
[1] https://www.oreilly.com/library/view/qmail/1565926285/ch04.h...
For one of my art projects I make YAML files that describe the front side, back side, and web side of a "three sided card". I generate these out of several templates, currently using ordinary string templating.
I'd love to be able to load a YAML file and add something programatically to the list and have the list stay in the same format that it was in, so if it was a
[1,2,3]
[1,2,3,4]
- 1
- 2
- 3
- 1
- 2
- 3
- 4
import assert from 'assert'
import { parseDocument } from 'yaml'
const flowDoc = parseDocument(`[1,2,3]`)
flowDoc.add(4)
assert(flowDoc.toString(), '[ 1, 2, 3, 4 ]\n')
const blockDoc = parseDocument(`\
- 1
- 2
- 3`)
blockDoc.add(4)
assert(
blockDoc.toString(),
`\
- 1
- 2
- 3
- 4
`
)E.g. I tried exactly the same thing, and it was quiet difficult with Rust. Because the way to parse it usually is with serde and it just removes the AST of course.
In the end I gave up, and just used JSON for my use case.
"<em>This</em> is a markup language" since there is text which is marked up.
YAML/JSON is a way to serialise fairly common data structures (arrays/lists, hashes/dictionaries, numbers, strings, bools, etc.)
Incidentally, if you can seamlessly replace XML with something like JSON, then you probably aren't using the 'markup' bit of XML.
YAML: Probably not so great after all (arp242.net)
https://news.ycombinator.com/item?id=20731160
https://www.arp242.net/yaml-config.html
To which I posted:
https://news.ycombinator.com/item?id=20735231
I was suspicious of YAML from day one, when they announced "Yet Another Markup Language (YAML) 1.0", because it obviously WASN'T a markup language. Who did they think they were fooling?
https://yaml.org/spec/history/2001-08-01.html
XML and HTML are markup languages. JSON and YAML are not markup languages. So when they finally realized their mistake, they had to retroactively do an about-face and rename it "YAML Ain’t Markup Language". That didn't inspire my confidence or look to me like they did their research and learned the lessons (and definitions) of other previous markup and non-markup languages, to avoid repeating old mistakes.
If YAML is defined by what it Ain't, instead of what it Is, then why is it so specifically obsessed with not being a Markup Language, when there are so many other more terrible kinds of languages it could focus on not being, like YATL Ain't Templating Language or YAPL Ain't Programming Language?
https://en.wikipedia.org/wiki/YAML#History_and_name
>YAML (/ˈjæməl/, rhymes with camel) was first proposed by Clark Evans in 2001, who designed it together with Ingy döt Net and Oren Ben-Kiki. Originally YAML was said to mean Yet Another Markup Language, referencing its purpose as a markup language with the yet another construct, but it was then repurposed as YAML Ain't Markup Language, a recursive acronym, to distinguish its purpose as data-oriented, rather than document markup.
https://en.wikipedia.org/wiki/Markup_language
>In computer text processing, a markup language is a system for annotating a document in a way that is syntactically distinguishable from the text. The idea and terminology evolved from the "marking up" of paper manuscripts (i.e., the revision instructions by editors), which is traditionally written with a red or blue pencil on authors' manuscripts. In digital media, this "blue pencil instruction text" was replaced by tags, which indicate what the parts of the document are, rather than details of how they might be shown on some display. This lets authors avoid formatting every instance of the same kind of thing redundantly (and possibly inconsistently). It also avoids the specification of fonts and dimensions which may not apply to many users (such as those with varying-size displays, impaired vision and screen-reading software).
I'm not disagreeing with the author here, but as someone old enough to remember the rise of XML as a data transmission format (and Erik Naggum's masterful rant against it[0]), it's strange because historically speaking both XML and JSON were also popularized as more "human readable".
I would be curious how many HNers (and even more so newer developers outside the HN-o-sphere) have worked extensively with or even written parsers for binary (or otherwise non-human readable) file formats. Writing an MP3 metadata parser used to be a standard exercise for devs looking to level up their programming skills a bit.
It personally feels weird to me that we would keep pushing for more "human readable" data formats when the world is increasingly removed from one where non-programmer humans need to read data. Keep your data in whatever format make sense and let software handle transforming it to a more readable or more efficient format depending on the needs, even if humans can't read it (they shouldn't need to!).
On top of all that my experience has been that JSON leads to more atrocities than XML (while fully agreeing with all of Erik Naggum's points about that) and YAML creates even worse horrors than JSON. It seems we'll soon be approaching eldritch horrors if we continue to pursue human readable data exchange formats.
I like binary formats for things that are simple and don't change too often. However, I still love not having to waste days on studying yet another bespoke binary format & parser for things that are complex and don't work right for whatever reason. So when performance isn't a concern and you aren't working in a size-constrained environment, I do find that "human readable" formats are often worth it.
As a practical example, I recently hit a bug where KiCad moved some custom footprints' pad shapes around after saving & reloading. And I quickly discovered that the footprint files are just S-expressions and relatively self-descriptive so I fixed my issue in five minutes with vim without ever needing to look at docs or code. That kind of thing is super convenient. Later I discovered that other users are likewise working around the program's limitations using a text editor or custom scripts to manipulate things KiCad won't do for you; for example, to create a repetitive pattern of components in a layout more complicated than a grid.
Your (or blogger's) claim is that GitHub misparses the YAML actions config when it comes specifically to Python? Or that YAML is inherently inadequate to the task of representing Python's necessary actions?
Python_versions:
- 3.8
- 3.9
Python_versions:
- 3.8
- 3.9
- 3.10
I'm not sure what the name for this problem is.
It's something like.. an unfaithful but unintentionally working representation. Until it doesn't work anymore. The solution in YAML is to quote the values so that they become strings as intended.
Which means anything that accepts YAML accepts JSON.
So you can write your Github action code in JSON (with a yml file extension).
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
The difference between my post and that blog, and your reply for that matter, is that I'm not telling anyone not to read it. I'm inviting you to comment on why people feel the need to tell others to move on.
Here's a good example of how to respectfully disagree: https://news.ycombinator.com/item?id=29220994
And you can't pick what config format a tool you need uses.
Its all in the name. All leaf values are strings. It is literally nested text.
- YAML
- JSON
- JSONC
- XML
- TOML
- INI
- CSV
- NestedText
- Starlark
- Python
- Dhall
- Cue
- Jsonnet
- DADL
- EDN
- HCLJSON5 exists, and is quite nice. I've picked it up for configs on a work project and haven't once had an issue due to misconfiguration, unexpected parsing, or friction with leaving a trailing comma or a comment.
The nesting in JSON5 is simple and familiar to pretty much all programmers, unlike deep nesting in TOML which is a huge pain.
And then I tried to use a tool called SBNF to write the grammar for the language at a high level and have it spit out Sublime Text syntax highlighting code. Didn't quite work yet unfortunately.
Also, the changelog is quite frightening!
> [In 3.0], `[ ]` now represents a list that contains an empty string, whereas previously it represented an empty list.
* keys containing leading spaces, newlines or colons can be represented with the multiline key form, where each line of the key starts with `: `.
* leading or trailing space is not complicated; the string values are just the rest of the line after the separator from the key, `: `. The values are not trimmed.
* a string value beginning with `[` just works in most places. This would not be confused with list values, as these would only start after a new line. Only in the compact inline list and inline dict form there are restricted characters for syntax.
It seems that their claim, no escaping required, holds. The slightly more verbose form of the language constructs may be required to represent special values though.
I'm not even a Rust person. I've never worked in Rust in my life, so there is no "preference bias" in my comparing the two. I just don't find YAML, or this "improvement" as "human-readable" as people make out to be.
The way it blend types and values makes learning it super easy, yet you can do complex things with few lines.
But the main implementation exports to yaml without quoting the strings, which kinda defeat the purpose :(
(Also, the type system is absolutely delicious, but it badly needs a paper with a complete description. I’m extremely interested in how it works, but fragmentary “notes on the formalism underlying” CUE are not enough.)
Think of a graph with lots of attributes within which paths are searched for.
The doc is also quite clear and rich, but the way it's organized means I have to read it entirely before writting my first CUE file. It also lacks IRL examples so trials and errors were my best friends.
It can be discouraging.
YAML's problem is that whitespace is significant. TOML could be superior to it if it weren't for the fact that they forgot to forbid indentation. And now indented TOML is everywhere, including its wikipedia page.
If we have to make a change, why not finally bite the bullet and go to the form that has existed for decades and is obviously superior to all of these formats? S-expressions. There's even been a standard for data notation brewing for some time: https://github.com/edn-format/edn
Then we can actually forego http://xkcd.com/927 and do something useful with our significantly saved mental energy.
edit I see that I'm not at all alone in wanting edn to replace all this crap. So some action points on how to actually make that happen, in order of preference:
- write or improve robust edn parsers for your ecosystem
- write or improve robust x => edn converters for your ecosystems (x=yaml,json,toml,whateverpoisontheyuserightnow)
- use edn in your projects
- advocate the use of edn
For example, the following code translates JSON to YAML using only the Python yaml library:
import yaml
data = yaml.load("""
{
"firstName": "John",
"lastName": "Smith",
"isAlive": true,
"age": 27,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": "10021-3100"
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "office",
"number": "646 555-4567"
},
{
"type": "mobile",
"number": "123 456-7890"
}
],
"children": [],
"spouse": null
}
""")
print(yaml.dump(data))
address:
city: New York
postalCode: 10021-3100
state: NY
streetAddress: 21 2nd Street
age: 27
children: []
firstName: John
isAlive: true
lastName: Smith
phoneNumbers:
- number: 212 555-1234
type: home
- number: 646 555-4567
type: office
- number: 123 456-7890
type: mobile
spouse: null
import yaml
data = yaml.load("""
{
"firstName": "John",
# Woah comments
"lastName": "Smith",
}
""", yaml.SafeLoader)
Whoever thought this was a good idea in 2021 has to be braindead. But the CEO was pleased. Probably because they know Excel.
Back in 2006, the testing was written in HTML tables, no idea how it manage to still be around.
That's false. http://p3rl.org/JSON::XS#JSON-and-YAML
This YAML:
user:
name: Ted
user: {"name": "Ted"}
{"user": {"name": "Ted"}}Basically take JSON, make quotes optional for strings and make curly brackets optional if the object is indented properly, and boom you've got (something like) YAML.
E.g. "on" should no longer be treated a true. Thats a 1.1 version archaism. And 3.10 is going to have the same issue in JSON.
No doubt YAML could still be improved and maybe we'll get there eventually. 2.0 is a long discussed goal, but the creators of YAML (who I have talked to extensively) are cautious, thoughtful and methodic and won't make that jump until they are sure of it.
Meanwhile 1.2 is a fairly good spec, and difficulties largely lie with implementors and users.
YAML is a human-readable data-serialization language. Note the word readable, and that it's for data serialization. It's not intended to be human-writeable. It's not intended to configure an application (unless that configuration is the result of serializing the data object created after someone configures the app).
Since programmers don't really understand the different types/classes of file formats and what they're for, they choose the wrong formats for the wrong tasks. And then the people who are forced into using those formats for those programs find it's highly problematic - but they get pissed off at the formats and not the programmer!
I'm perfectly happy for people to create new data serialization formats, new configuration formats, new markup formats. But please avoid the trap of thinking "this format or that format sucks". None of them suck for what they were created for, when used correctly. YAML, for example, has huge advantages over most common data serialization formats. But those advantages fly out the window as soon as humans start writing YAML by hand, and then bolting on weird custom logic, as if this data serialization format were a higher-level language.
I use YAML quite extensively for an internal tool, since it's pretty much the only human-useable format out there. But the weirdness of the language makes it error-prone while the quality of available parsers is simply unacceptable. I ended up writing my own parser for a subset of unambiguous YAML (not unlike StrictYAML, but even simpler) that offers nice error messages etc. and it works very well for us.
Basically just a Typescript object where types are defined on the key somehow.
Because JS/TS objects are basically JSON with a bunch of annoyances (too particular about commas, always-quoted keys, no comments) removed.
You can use `literal_eval()` (in python, there are similar constructs in basically every scripting language) to prevent prevent people from putting code in the config, but really, it's the person writing the config file's computer, let them do what they want.
For compiled languages, it's a bit harder, but there are zero cases where a separate "configuration language" should exist.
No numbers? Looks nice otherwise, but this seems like a very weird decision.
That said, JSON with its "I might overflow your number" attitude is not much better in this regard.
I think NestedText is unique in leaving all leaf values as strings, so it does not need quoting or escaping.
Everything involves a compromise. YAML provides a lack of clutter at the cost of ambiguity. JSON is unambiguous, but comes with visual clutter. In both cases there are still lots of types they cannot handle and so must be passed on to the application.
The compromise with NestedText is that it provides simplicity and a lack of clutter by not supporting any data types for leaf values other than string. Thus, all interpretation of the text is deferred to the application. But fundamentally that is the best place for it, because only the application understands the context and knows what is expected.
Seriously, the more time I spend with other configuration formats the more I start appreciating JSON. It is a simple array and object format. Nothing can go wrong with it. No indentation rules. Easy to encode and decode. Easy to turn into actual arrays and objects in your programming language. Lack of types is not great, but use a type checker in your parser that throws exceptions and you are fine.
JSON is simple and that is why it is successful.
* start with JSON
* add comments (obviously)
* allow trailing commas (obviously)
* allow left-side and right-side to be independently and explicitly typed
* extensible type support (JSON's lack of u64 is absurd)
For example the following are all equivalent:
* "foo" : f32 = 13.37
* "bar" : f32 = "13.37"
* "baz" : f32 = 0x4155eb85 : hex
And extensibility to allow:
* "ham" : u64 = 123456789123
I would also be inclined to allow explicit config file hierarchy by allowing:
* configs can specify their base
* configs can specify their child
* must be acylic
I've thought about implementing this myself. But I've never written a real lexer or parser. It's on my side-project TODO list.
There are a few gotchas that are easy to catch with validations.
The end.
It's wishful thinking because the complexity is inherent, unfortunately. That's analogue to saying "programmers should not write bugs". Humans are fallible and error prone, it's not going to happen unless a language is restricted in such a way that a category of bugs is not possible by design. However, YAML's design is sprawling, so despite best intentions people will run into the problems caused by the complexity. Possible ways out are restrictions of the design (e.g. Strict YAML) or whole replacements (e.g. NestedText).
> There are a few gotchas that are easy to catch with validations.
Does this actually exist? If not, who's writing the code for these validations? How can we make sure everyone who needs to use them is using them?
The idea sounds good on paper, but not workable in practice because "patching after the spec problems" requires global coordination.
Can Lua's interpreter be compiled without some "dangerous" APIs enabled (whichever those might be) and thus be made viable as an embeddable and isolated configuration engine?
I'm just getting sick and tired of all the half-baked configuration formats and want to look for something that's both more flexible but still strict and unambiguous. I wonder if it's possible?
As I said, naive question.
This is actually a good question. The people who are not asking and going right ahead with that plan are doing a dangerous thing. <https://www.cs.dartmouth.edu/~sergey/langsec/occupy/> It's not viable because a subset of people would like to have the following properties upheld:
• Parsing configuration should be decidable and finish in finite time.
• Parsing configuration should not be a security exploit.
• I should not have to implement the Emacs runtime just to parse its configuration file.
> something that's both more flexible but still strict and unambiguous. I wonder if it's possible?
This really depends on what you mean by these words; I'm interested to hear your idea in detail. Meanwhile, have a look at Dhall <https://dhall-lang.org/> and the other languages mentioned in <https://news.ycombinator.com/item?id=29221643> and compare.
I'd also add the author has a competing framework so maybe there's a bit of pre-existing bias.
I don't want the language to be flexible enough that simple common errors go unnoticed - I WANT the parser to tell me at parse time if I screwed something up. It's a similar dynamic to dynamic/static typing.
Thankfully it's very rare I ever have to open any config file in a text editor that isn't aware of dozens of formats so things like a missing brace are always suggested anyways and it isn't much a problem regardless of format.
"tall and skinny". Well it looks skinny, but in bytes is actually fatter than the csv. Similarly, for comparing stuff, the columnar display works better.
My go-to is ini. Simple. Everything is a string. No assumed hierarchy in section naming. Just key/value pairs under sections. It is up to the using application to parse them how it sees fit.
Stop using configuration languages as programming languages and leave YAML alone as configuration language.
If you know about dangers of unsafe YAML, why don't you just replace load() with safe_load()??
2) If your configuration file is too big, maybe you're doing it wrong.
3) TOML is by far the less ugly format for configurations that i've ever used. Far from perfect, but it's predictable and doesn't require indentation.
python-version:
- 3.8
- 3.9
- 3.10
python-version:
[3.8, 3.9, 3.10]
It's nice, simple, and allows variables and functions.
Jsonnet has a similar feature set, but I find it nigh unreadable.
"NestedText was inspired by YAML, but eschews its complexity. "
Mandatory XKCD: https://xkcd.com/927/
Fun fact: HTML was invented because SGML was "too complicated".
It has powerful types, can be extended, clean explicit syntax, whitespace independent, easy for humans and machines to read. It's like JSON done right.
EDN is less ideal for Python and other similarly somewhat high-level languages and there just aren't any libraries for C/C++ from what I have seen. In general, XML/ JSON/ YAML/ custom binary or text formats it is in that order for most software from what I can guess. Remember, .docx, .xlsx and other document formats are also basically just XML-based configurations for programs/ interchange formats.
https://stedolan.github.io/jq/
Is there a syntax for specifying yaml schema like coach?
You could have found this by searching the Web for "jq for yaml". <https://kislyuk.github.io/yq/>
> yaml schema
You could have found this by searching the Web for "yaml schema". <https://rx.codesimply.com/> <https://web.archive.org/web/2021/http://www.kuwata-lab.com/k...>
In practice, schemas designed for operating on the JSON infoset (not the serialisation) will also work.
- All the flexibility of python, plus commenting!
- json module can test or dump as a flat blob o' text or pretty-printed
#ProblemSolved
#OKjustMoved
Do people really represent software versions as floats?
3.10 being converted to 3.1 makes sense. Use a string instead.
The problem is almost no data that people want to actually represent has this form and every way people have tried to beat XML into representing other things (i.e. lists and dicts) is kludgy is awkward.