Hacker News Reader

I agree with your point completely. I just want to add that throwing out raw numbers like "10.3MB" or "400 KB" is not very precise. Binaries can vary immensely based on whether they have debug info, string tables, etc. or whether these have been stripped away.

I wrote a size profiling tool that can give much more precise measurements (like size(1) on steroids, see: https://github.com/google/bloaty). Here is output for LuaJIT:

    $ bloaty src/luajit -n 5
         VM SIZE                     FILE SIZE
     --------------               --------------
      74.3%   323Ki .text           323Ki  73.8%
      12.5%  54.5Ki .eh_frame      54.5Ki  12.4%
       7.6%  33.2Ki .rodata        33.2Ki   7.6%
       2.2%  9.72Ki [Other]        12.9Ki   2.9%
       2.1%  9.03Ki .eh_frame_hdr  9.03Ki   2.1%
       1.2%  5.41Ki .dynsym        5.41Ki   1.2%
     100.0%   435Ki TOTAL           438Ki 100.0%

And for Pixie:

    $ bloaty pixie/pixie-vm -n 5
         VM SIZE               FILE SIZE
     --------------         --------------
      57.5%  4.39Mi .text    4.39Mi  44.7%
      33.7%  2.58Mi .data    2.58Mi  26.3%
       0.0%       0 .symtab  1.31Mi  13.4%
       0.0%       0 .strtab   978Ki   9.7%
       8.8%   688Ki [Other]   595Ki   5.9%
       0.0%       8 [None]        0   0.0%
     100.0%  7.64Mi TOTAL    9.82Mi 100.0%

In this case, neither binary had debug info. Pixie does appear to have a symbol table though, which LuaJIT has mostly stripped.

In general, I think "VM size" is the best general number to cite when talking about binary size, since it avoids penalizing binaries for keeping around debug info or symbol tables. Symbol tables and debug info are useful; we don't want people to feel pressured to strip them just to avoid looking bad in conversations about binary size.

Its worth noting that the design of the RPython JIT will always result in a large amount of static data in the resulting binary. The RPython translator basically generates a bytecode representation of most of your interpreter and bakes that into the binary. You can probably expect at least a 2x size increase in the size of your binary. As a reference point, after stripping [Pycket](https://github.com/pycket/pycket)'s binaries are 6.1Mi without the JIT and 16Mi with the JIT.

To compare, LuaJIT is about 400 KB, and that includes the Lua standard library, a JIT almost certainly more advanced than Pixie's current one, an incremental GC, and a C FFI.

Back in the day, Smalltalk was criticized as bloated because one ended up with stripped binary+image of about 2MB. Someone around that time got SqueakVM+image down to just under 400k. There were specialized Smalltalks in company R&D that got their image down to 45k.

The Oberon Operating System and compiler was 131 KB. [1]

[1] http://users.cms.caltech.edu/~cs140/140a/Oberon/system_faq.h...

Don't forget, even 2 MB was still a somewhat inconvenient size when floppy disks were only at 1.44 MB capacity.

Floppy disks were well on their way out, and people were still complaining about a 2 MB footprint.

I thought Smalltalk's hayday was early-mid 80s. That would imply 5.25" floppies would still be in use and 3.5" floppies overtaking them circa 1988.

I thought Smalltalk's hayday was early-mid 80s.

My involvement with Smalltalk started in the 90's. Then it still had high penetration in the Fortune 500, and it was a niche language for complex financial programs. It was also used in Energy.

Interesting historical note!

Well, for comparison "Hello World" in Common Lisp, using SBCL on x64 Linux creates a ~70 Mb binary, but nobody would ever claim it's small :-)

Unfortunately, for an arbitrary CL program it's impossible to tell for sure how much of the CL compiler and standard library it will need at run time, so SBCL takes the easy route and just includes everything. Some of the commercial Lisp compilers are a lot smarter at stripping things out and and can create significantly smaller executables.

I usually avoid the issue altogether by not building binaries and running most things from the REPL or using a "#!/usr/bin/lisp --script" shebang line.

SBCL does not create binaries per se, but dumps the program image. While it practically is a binary, it's not equal to an executable that a conventional compiler/linker produces.

Except without a better way to package a binary it will be compared as one.

That distinction isn't important here, though. It is the reason the resulting file is so big, but for all practical purposes it's a binary that statically links the full CL runtime.

I wonder why tree shaking is so ineffective. Is that because any symbol could be accessed dynamically at runtime?

SBCL does treeshaking? I thought it just dumps the image.

Well that would explain it. Any idea why they don't tree shake?

Because no one wrote one. There were some attempts but nothing which would work now. Delivering small applications does not seem to be a focus of SBCL users.

https://www.snellman.net/blog/archive/2005-07-06.html

There is also the possibility to compress the image. See :compression option when saving an image:

http://www.sbcl.org/manual/#index-g_t_0040sbext_007bsave_002...

Commercial Lisp implementations like LispWorks and Allegro CL have a treeshaker for delivery, but that's not too surprising, since these products are for developers of applications and the users pay for such features.

eval is the root of all evil. String to symbol lets you introduce anything.

In lisp eval does not take a string as its input, but a form, i.e. a linked list, containing the program to execute. Maybe you confuse it with JavaScript or Python eval (or the like) where the function both reads in a string and runs the program it parsed from that string. There are diverse functions to convert a string to a symbol in different lisps.

Perhaps i'm rusty.

in this case

    (funcall #'+ 1 2 3)

or the more scheme-ish

    (apply '+ '(1 2 3))

i think the symbol plus gets evaluated to #<function+> and then is called with 1 2 3 - CL will do apply, of course, not sure about scheme and funcall. I'm totally willing to accept my mental model is wrong, i don't have an interpreter handy.

So if that's true, you run into this problem:

    (funcall (string->symbol "+") 1 2 3)
    (apply (string->symbol "+") '(1 2 3))

maybe string->symbol does not introduce in to the function namespace, but i bet there's a way or a flag to do it.

so anyway, at the end, you run into this problem:

    (funcall (string->symbol "any-random-function") 1 2 3)
    (apply (string->symbol "any-random-function") '(1 2 3))

this makes tree shaking hard.

you could, of course, look for the handful of ways to bind symbols to functions, maybe do constant folding and include a minimal set if the compiler can prove what's actually needed.

but in general i think tree shaking is hard when you can load things like that dynamically.

(funcall #'+ ...) does not do a symbol lookup and does not need to retrieve the function from a symbol in Common Lisp. (funcall '+ ...) would retrieve the function from the symbol.

Treeshakers are generally used for application delivery in Lisp. When you do that, then one usually limits the amount of use of runtime dynamics. Typically you can tell the treeshaker what to remove or what to keep: the compiler, the symbol table, debugging info, etc etc.

Allegro CL and LispWorks have extensive facilities for this.

http://franz.com/support/documentation/10.0/doc/delivery.htm

http://www.lispworks.com/documentation/lw70/DV/html/delivery...

More briefly, in a half decent implementation, the + symbol is involved in the processing of (funcall #'+ ...) in the same ways and at the same times as it is involved in the plain call (+ ...).

Some CL compilers even don't save an image. An extreme example is mocl. mocl applications for a 64bit iPad start at 1 MB size.

Other examples of CL compilers not saving an image are ECL and WCL.

The ECL manual had the terminology a bit wrong, but it actually does not dump an image.

https://common-lisp.net/project/ecl/static/manual/ch34.html#...

WCL:

https://github.com/wadehennessey/wcl

> For example, the executable for a Lisp version of the canonical ``Hello World!'' program requires only 20k bytes on 32 bit x86 Linux.

10.3MB definitely is not small. I was expecting couple hundreds of K, or 1M.

I wonder if it's a stripped binary.

Yes, at least on Unixes, stripping could reduce the sizes of binaries by quite a bit.