I started using Lua because I needed to gather logs from many different systems and build timelines around the events to identify patterns across all the aggregated data. The logs were large, and the metadata was often very complicated.
I looked at several SQL-like solutions, and at Perl extensions on CPAN. After lots of googling, it finally occurred to me that I might need a programming language that had been designed with efficient data description as a high priority. I was surprised that a streamlined "little" language like Lua fit my requirements [0]. I was expecting to have to learn something big, complicated and esoteric.
I needed my resulting datasets to be portable between various Unixes, and fairly standalone. Because of the long load times, I didn't want to use a tool that required any more processing steps on input than absolutely necessary. While serialized Lua tables are more verbose than CSV, they do compress well. They are also valid Lua programs. They can include metadata of arbitrary complexity, and even turing-complete snippets to modify expression of the data as it is loaded.
Salvatore Sanfilippo (antirez) had many languages to choose from for his "data structure server". The next major release of Redis will have a Lua interpreter. [1]
I don't know Erlang well enough to guess -- is a valid Erlang program an appropriate way to store a large amount of polymorphic data? Is Lua perhaps a better way to approach this problem?
[0] "Lua as a data description language (Dimitris Papavasiliou)" http://lua-users.org/lists/lua-l/2004-06/msg00028.html
[1] "Short term Redis plans" http://antirez.com/post/short-term-redis-plans.html
I don't know about embedding it in Erlang, but anecdotally, it works quite well as a data dump format.
1. All network I/O is non-blocking, even if you write code in a blocking manner (i.e. no callback programming when writing scalable servers)
2. Erlang's processes are green threads, so you get massive concurrency for free (i.e. no cooperative thread programming when doing large-scale multi-tasking)
3. In an OTP environment, code will be executed as part of a supervision tree, so it will be very difficult to crash the interpreter (i.e. no strict need for error- and exception-handling code)
So while this interpreter will probably be quite slow for single-threaded tasks compared to the standard interpreter, there will be benefits if you want to run a lot of Lua code concurrently in a networked environment.