Using Lua as a Serialization Format
mkhan45.github.io
mkhan45.github.io
https://www.lua.org/history.html
The current version of Lua still has some traces of that origin. For example, to keep RAM use manageable when parsing very large files the parser directly emits bytecode in a single pass, without building an intermediate abstract syntax tree.
It seems cute to do it this way, but now you have to start talking about removing certain functions from the standard library, running the script in a sandbox, etc.
When you set up the runtime in your language most libraries make it trivial to define that a given function can call into your own code, i.e. I could define 'getUser' in my lua code to call back to a C# function with variables, etc...
[1]: https://docs.bazel.build/versions/3.3.0/skylark/language.htm...
It's true that computations could be arbitrary long (as with most other languages). Bazel has a flag to reduce this kind of problem (--max_computation_steps).
These two things combined were probably the biggest software design mistake I ever made. We still have to drag a stripped down Tcl interpreter along in our asset pipeline 20 years later because it's nearly impossible to switch away (because of the manually written 'scriptlets' lurking everywhere in asset source files - of course it would be possible, but it was never on the right side of the cost-benefit equation).
Never mix code and data, folks :)
And code is data, and has always been.
for i = 1, 1000000 do
for j = 1, 1000000 do
for k = 1, 1000000 do
for l = 1, 1000000 do
...
Once you move your publicly injectable data format from declarative to something else, you're in for a world of hurt.The article actually suggest pro doing this, i.e. using a script engine (Lua here) to store your configs/settings/data, and I agree, it can make certain things much simpler, and more clean.
If you have pure data file formats, like JSON, XML or even your own custom binary formats, it's trivial to upgrade or change them (such as switching from XML to JSON to your own binary format and back).
But once you have human-scripted parts such an automatic conversion suddenly gets much harder or impossible, since now you need to transpile from one scripting language to another.
That's pretty much the worst case, but the problems already start when APIs or data layouts change. If a data item goes away, it's trivial to automatically inspect a pure data representation like JSON and remove or ignore such obsolete data items. But with free scripting, you suddenly get an obscure scripting runtime error.
And finally, if you want to process such mixed data / code files in another tool, you always need to integrate that specific scripting engine, which is a much bigger dependency than (for instance) a simple JSON or XML reader/writer, or even just your own binary file format readers/writers.
For instance the Emscripten SDK's .emscripten config file suffers from this: at a glance this looks like a simple text file with key/value string pairs, but on closer inspection it must actually be evaluated through a python interpreter to be any useful. So what could be a simple text parsing tool to extract filesystem paths from this config file suddenly requires a full blown python environment.
So basically, by allowing "turing complete" config or data file formats, you make future changes much harder then they need to be.
Of course all of this isn't all that important for small throwaway projects. But sometimes those small projects (or quick'n'dirty solutions in general) become much more important and longer-lived than expected.
There's great value in sticking with "inert" data formats like XML or JSON, which have standard parsers in most languages, developed over years.
Once data crosses over into code, there's a whole host of potential issues to consider, including the requirement of an interpreter/compiler and a sandboxed environment, with all that entails. The Emscripten config needing Python is a perfect example.
Another example I thought of, is Markdown and MDX. The latter allows the use of JavaScript and React components mixed with markup, for dynamic content. It's brilliant in its own way, but I'm wary of using it as a document format precisely because it's no longer just data. Documents in MDX can't be consumed by other languages, and will always require the full JS/React environment.
Tcl actually feels like it could reasonable to use for data, due to its ubiquity/longevity. Lua, or better yet some kind of Lisp, also seem attractive as a serialization format. The interpreter could be small and stable, and the latter at least might even be simpler to parse than XML or JSON.
> We still have to drag a stripped down Tcl interpreter along in our asset pipeline 20 years later because it's nearly impossible to switch away
This makes me see how the decision created a long-term dependency to a language, where the data was no longer in a portable format. Good to keep this in mind next time I'm tempted to allow data to become code.
Unfortunately every configuration file format these days seems to add so many features that it inevitably becomes Turing-complete... the trick is finding the balance between expressiveness and halting decidability.
[1] http://www.lispworks.com/documentation/HyperSpec/Body/v_rd_e...
I don't care for the syntax (at all) but I am seriously impressed by the careful design. I've always wondered where you'd need to draw the line between plaintext config files and executable code, and Dhall is much further into "programming language" territory than I would've expected.
*Dhall homepage for anyone else that's interested - https://dhall-lang.org
Sounds a bit like a missing specification for the distinction of load-time and run-time configuration.
Configuration, to me, is something that happens once and not often, and is therefore "something = foo". Update is a run-time thing, and therefore "if something then bar".
Lua covers both of those cases superlatively. Case A, byte code. Case B, same. Just be sure you have "generate byte code" in the right place, C, too ..
It would be nice to be able to statically verify that their submitted configuration "behaves nicely", but the moment you introduce any kind of Turing-complete configuration language all guarantees go out the window because the problem becomes literally undecidable. [And then you try to mitigate it by imposing limits like "kill the parse process if it takes longer than X seconds or uses more than Y megabytes of memory".]
The advantage of, say, a simple .INI file for configuration purposes is that the time required to parse it (and storage space required to remember what's in it) is O(n) where n=size of config file...
EDIT: so take the example Lua file from the original post but change the loop limits from "32" to "16777216":
for i = 0, 16777216 do
for j = 0, 1677216 do
add_body({x = 20 * i, y = 20 * j, mass = -0.1, rad = 2})
end
end
Congrats! Your 128 byte long configuration file now requires 2^48 iterations of the inner loop to parse and generates 2^48 instances of the "body" object. That's an expansion factor of at least 2^41 generated bytes for every byte of input, which is a shockingly bad Denial-of-Service attack against whatever is responsible for parsing the evil config file.Like, I get that there are services that require this level of trust to the end user, but why wouldn't I solve this problem by timing the config load and immediately stopping any config process that takes longer than it should? Its the 21st century, we can still use interrupts. ;)
More troubling are loops which consume CPU without hitting memory limits. I would hope Lua provides a way to limit the number of instructions when evaluating expressions. If not then delegating reading the config to a limited subprocess could work at the cost of still more complexity.
And whichever you choose, that's great(I've gone on a huge concatenative binge myself), but then to define the application's final protocol, you have to filter it. You have to filter to turn many of these meanings into errors, which leads towards syntax, type-checking, and all the rest.
And to the extent that this is a problem, it mostly reflects the human and philosophical factors. What we believe in determines what meanings we wish to see represented in the protocol; our beliefs are many, varied and often contradictory, with entire organizations and systems of governance designed around reconciliation of these beliefs; therefore extensively structured protocols are necessary to define the "sometimes-errors" that result from our incoherence, our limited notion of reality. And that is producing the gradual ramp up towards complex languages, formats and protocols that address the impossible with the incomprehensible.
But data and protocols that are very truthful usually do not require such a complex structure, or that structure emerges from natural laws. They may be hard to compute, or hard to store - DNA sequences are very large, and modern cryptography is premised on computational difficulty - but the code that deals with these things is not large in expression, and only needs validation along some very particular axes. Regardless the task of discovering and implementing these things is hard, because evaluating the theory is hard, much harder than validating whether a stream of bytes represents a number, a string, or a list.
You're technically correct (the best kind of correct), but jandrese wrote "a complete language", not "a Turing-complete language". You don't need Turing-completeness to create security problems – filesystem access alone does that.
> The former is trivial to solve: add a timeout.
That only solves the termination problem, which is a minor concern anyway. You don't need much time to compromise a system if you have filesystem access. No "bugs in the parser implementation" needed.
It was okay. It's cool that you can import data directly into Lua scripts, since Lua usually has poor library support for any other data format.
But I hit a wall with a limit on the number of tables in a file or something, that was specific to the Lua impl I was using. (Probably LuaJIT)
Sometimes I miss Lua. I think with only a few big breaking changes it could be a much better language.
I've used it a bit for simple game development(Love2D). It's an okay language but it seems only suitable for building anything beyond small systems. No good IDE support and needing to emulate OOP was a bit annoying to me.
As for object-orientation it takes Go or Rust-like approach where there isn't an explicit class construct but conventions; you can build your own or use libraries to get there. In fact Rust or Go are younger so Lua should be appreciated for doing this a lot earlier. Having rigid object-oriented-ness will generally make the language less usable with other paradigms like functional, imperative, generic, etc.
It was a really pleasant experience but we were not working with a very large number of tables. I miss Lua all of the time, too, and play around with it whenever the opportunity arises. It holds a really special place in my programmer heart, alongside lisp, C, and Ruby, for having shown me some really interesting things at the right time in my career. It's too bad it doesn't have more widespread adoption, I think a lot of people could benefit from using it for the things it's good at.
When I ask around, I don't hear about it in use locally (edit: that's actually not entirely true, Blizzard uses it and maybe some of the other local game shops, but I don't see it anywhere else locally). I met someone at a conference a year or two back who was using Kong. That's been it. Maybe I'm just missing it but I don't really see widespread use.
Its such a simple but powerful language, has one of the best interop apis with C and I love how they approached coroutines.
It's really a shame it doesn't see wider use. We used it to ship all our game logic on the PSP, 400kb prealloc block and it worked like a charm.
Edit: And of course in Magic Lantern, for Canon EOS DSLRs https://magiclantern.fm/
Redis[2] lets you write internal scripts in Lua.
Edit: Forgot Torch[3] which is great for generating nonsense with RNNs.
I think it's definitely getting more of a foothold in the world outside of games.
[1] https://konghq.com [2] https://redis.io/commands/eval [3] http://torch.ch
I think that's because there aren't many things it's really good at. It excells at a niche – as a light-weight, easily embeddable scripting language – but is mediocre as a general-purpose programming language, because it's standard library is extremely spartan and lacks so many things you need on a day-to-day basis.
The language is well designed, with orthogonal features that complement each other. Coroutines and meta-tables are easy to understand and still powerful way to organize and control code base.
You can actually learn it in a week, and hold all of the language in your head. A very valuable thing in my opinion. The code is very readable to beginners (unlike Perl, Haskell or APL).
The interpreter is incredibly fast. Even non-JIT implementation is surprisingly fast.
The LuaJIT FFI is so convenient that the difference between C/C++ libraries and Lua libraries almost disappear.
There are cross-platform hosting frameworks that allow you to write and execute dynamic interpreted code on otherwise locked down platforms (Android, Switch, TVs...).
I would say Lua lacks tooling and better organization of community. Biggest Lua communities were built around products that embed Lua like WoW, Roblox, Corona (now Solar2D).
[1] Lua Parsing Expression Grammar http://www.inf.puc-rio.br/~roberto/lpeg/
It turned out that one of the first companies (embedded, automation) I interviewed with used it for integration between various third party subsystems. I think those conversations helped me feel like less of an imposter at the time.
I never use it anymore, but the language is nice. I wonder what web development world we'd live in now if they'd embedded lua instead of creating JavaScript.
You can't have more than N local variables, I believe. The problem goes away if you put them into a top-level table.
https://mkhan45.github.io/2020/06/12/Lua-integration.html
Edit: Ah, 'Lua' should be lower-case.
Everybody who had access to this format can take it over. It can do far too much, that's why everybody uses JSON. It can only represent hashes, arrays, and some limited primitives. No links, no logic, no lib. But lua is far too powerful. Game over.
Lisp would be even better, btw. Much easier to parse, and much easier to avoid eval. But his idea is eval.
In what format, that you then have to parse?
More powerful features can also be helpful if you have a large config file that would typically need to be generated by a separate program. One example of this is shown in the linked blog post, when they use a for loop to initialize a large array that has many similar objects.
It can also help if the config file starts gaining special "PRAGMA" fields over the years. There is a certain tendency for configuration files to slowly evolve into an ad-hoc and poorly implemented scripting language and in those cases, switching to a proper programming language cuts the problem at the root.
That said, for data exchange situations JSON has the advantage that it is more well-behaved in the case of malicious inputs.
Lua is not far too powerful, Lua is as powerful as you want it to be.
The author doesn't mention it explicitly, but I think that's his case.
For his situation, I think it's a good use of the tool.
For pure "serialization", this is a bit overkill, of course.