Trials and Tribulations of Native Lua Compilation
samyeyo.github.io
samyeyo.github.io
This is already true of upstream Lua implementations. It is very easy to write thin Lua binding layers to C libraries.
That is the entire point of Lua. It’s easy to call Lua from native code, and it’s easy to call native code back from Lua, so you can have call stacks that seamlessly interweave Lua and native code.
I’m very skeptical of the idea that this can unlock improvements in performance or size. We already have LuaJIT which is plenty fast, and PUC-Lua which is so small, it runs on MCUs.
My point is about deploying standalone applications.
Tools like srlua, luastatic, and many others, package Lua scripts with the interpreter and required libraries. These bundled executables can sometimes trigger antivirus false positives, just like tools such as py2exe for Python. They use packaging techniques that can resemble those used by malware, which is why I called this approach "hacky".
If you have the source code for these modules, you can statically link to them too. I've done this myself for Windows a few times, and it works pretty well in my experience.
Static linking itself is not a problem for Lua.
If all you want is a tiny script that checks the weather or downloads a file, having to manually link or bundle all those pieces just to ship a standalone executable is a lot of work. That's one of the problems clx aims to solve.
When your application depends on several Lua and binary modules, you may have to compile each binary module, embed the Lua scripts while preserving their expected module paths, link everything against the Lua VM, and write glue/loading code to make everything work together (see the comment below regarding LuaSocket)
Of course, it's possible. But it's a lot of manual work, especially for newcomers.
With clx, the goal is simple: compile your Lua program into a native executable. No stock Lua VM dependency, no glue code, no module-loading code to write.
> no glue code, no module-loading code to write.
How does that look like in practice? Your write up only shows the case where you’re writing your own custom clx glue code to interface with a binary module, which is strictly worse.
Yes it's in another thread.
clx provides a simpler, stackless C++ API because it doesn't use the Lua VM.
The Lua C API compatibility layer lets you reuse existing binary modules without rewriting their source code. The wrapper is there for compatibility, not because modules must be written that way.
For new modules, you can use clx's native C++ API directly.
Correct, but I'd argue that you need to do this anyways (well, I do this anyways), because I don't like the alternative of redistributing binaries compiled by random third parties from the internet.
> link everything against the Lua VM, and write glue/loading code to make everything work together (see the comment below regarding LuaSocket)
> Of course, it's possible. But it's a lot of manual work, especially for newcomers.
To be clear, I think that your project is great, but this really isn't that much work:
# FILE: .gitignore
program
*.luac
*.o
# FILE: Makefile
CFLAGS := -std=c23 -Wall -Wextra -Werror
ifeq "${mode}" "debug"
CFLAGS += -ggdb3 -fanalyzer -fsanitize=address,undefined,leak
else
CFLAGS += -O2 -flto -s
endif
%.luac: %.lua
luac -o $@ $^
%.o: %.c
${CC} -c -o $@ $< ${CFLAGS}
c_libraries := library.o
lua_libraries := main.luac
program: main.c ${c_libraries} ${lua_libraries}
${CC} -o $@ $< ${c_libraries} ${CFLAGS} -llua
// FILE: library.c
#include <lauxlib.h>
#include <lua.h>
#include <lualib.h>
#include <stdcountof.h>
static int test(lua_State *L) {
lua_pushstring(L, "Hello from C!");
return 1;
}
int luaopen_library(lua_State *L) {
luaL_Reg library[] = {
{"test", test},
{nullptr, nullptr},
};
luaL_newlib(L, library);
return 1;
}
// FILE: main.c
#include <lauxlib.h>
#include <lua.h>
#include <lualib.h>
#include <stdcountof.h>
#define preload_c_lib(name) \
do { \
extern int luaopen_##name(lua_State * L); \
lua_pushcfunction(L, luaopen_##name); \
lua_setfield(L, -2, #name); \
} while (0)
#define preload_lua_lib(name) \
do { \
luaL_loadbufferx(L, &name[0], countof(name) - 1, #name, "b"); \
lua_setfield(L, -2, #name); \
} while (0)
static void initialize_modules(lua_State *L) {
// Load standard modules.
luaL_openlibs(L);
// Load external modules.
lua_getglobal(L, "package");
lua_getfield(L, -1, "preload");
// "main.lua"
static char const main[] = {
#embed "main.luac" suffix(, '\0')
};
preload_lua_lib(main);
// "library.o"
preload_c_lib(library);
lua_settop(L, 0);
}
int main() {
lua_State *L = luaL_newstate();
// Initialize the modules.
initialize_modules(L);
// Run main script
if (luaL_dostring(L, "require('main')()") != LUA_OK) {
printf("Error: %s", lua_tostring(L, -1));
}
lua_close(L);
return 0;
}
-- FILE: main.lua
return function()
local library = require("library")
print(library.test())
print("Hello from Lua!")
end
To run: $ make && ./program
cc -o program main.c library.o -std=c23 -Wall -Wextra -Werror -O2 -flto -s -llua
Hello from C!
Hello from Lua!2) You may like Lua as a language and want to use it for general purpose programming, not necessarily embedding into host program
I will say though that the performance aspect has to be taken with a grain of salt, "native compilation" is not a meaningful improvement if every source line translates to multiple branches or vtable lookups. From my experience plain conversion of Lua to C (assuming you faithfully follow the specification, including coroutines, metatables, etc.) produces equal or slower code with rare exceptions for numerical loops. If it wasn't all AI generated I'd maybe spend time understanding this project.
This has been a long known reality of compiling dynamically typed and functional languages to native code.
As for AI, I've spent months of my free time building clx. The design and implementation are mine. It's true that I use AI to help with documentation and to debug complex issues, but that doesn't make the project AI-generated.