Hacker News Reader

Comment by amelius | Hacker News Reader

Is there a tool that takes an executable, collects all the required .so files and produces either a static executable, or a package that runs everywhere?

View on HN

There are things like this.

The things I know of and can think of off the top of my head are:

1. appimage https://appimage.org/

2. nix-bundle https://github.com/nix-community/nix-bundle

3. guix via guix pack

4. A small collection of random small projects hardly anyone uses for docker to do this (i.e. https://github.com/NilsIrl/dockerc )

5. A docker image (a package that runs everywhere, assuming a docker runtime is available)

6. https://flatpak.org/

7. https://en.wikipedia.org/wiki/Snap_(software)

AppImage is the closest to what you want I think.

AppImage looks like what I need, thanks.

I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

Depends on the license and the specific piece of software. Redistribution of commercial software is may be restricted or require explicit approval.

You generally still also have to abide by license obligations for OSS too, e. G., GPL.

To be specific for the exampls, Nvidia has historically been quite restrictive (only on approval) here. Firmware has only recently been opened up a bit and drivers continue to be an issue iirc.

Don't forget - AppImage won't work if you package something with glibc, but run on musl/uclibc.

No, that's a copyright violation, and it won't run on AMD or Intel GPUs, or kernels with a different Nvidia driver version.

But this ruins the entire idea of packaging software in a self-contained way, at least for a large class of programs.

It makes me wonder, does the OS still take its job of hardware abstraction seriously these days?

The OS does. Nvidia doesn't.

Does Nvidia not support OpenGL?

Not really. Nvidia-OpenGL is incompatible to all existing OS OpenGL interfaces, so you need to ship a separate libGL.so if you want to run on Nvidia. In some cases you even need separate binaries, because if you dynamically link against Nvidia's libGL.so, it won't run with any other libGL.so. Sometimes also vice versa.

Does AMD use a statically linked OpenGL?

AMD uses the dynamically linked system libGL.so, usually Mesa.

So you still need dynamic linking to load the right driver for your graphics card.

Most stuff like that uses some kind of "icd" mechanism that does 'dlopen' on the vendor-specific parts of the library. Afaik neither OpenGL nor Vulkan nor OpenCL are usable without at least dlopen, if not full dynamic linking.

It does, and one way it does that is by dynamically loading the right driver code for your hardware.

That’s a licensing problem not a packaging problem. A DLL is a DLL - only thing that changes is whether you’re allowed redistribute it

AppImage is not what you need. It's just an executable wrapper for the archive. To make the software cross-distro, you need to compile it manually on an old distro with old glibc, make sure all the dependencies are there, and so on.

https://docs.appimage.org/reference/best-practices.html#bina...

There are several automation tools to make AppImages, but they won't magically allow you to compile on the latest Fedora and expect your executable to work on Debian Stable. It's still require quite a lot of manual labor.

Yeah a lot of Appimage developers make assumptions about what their systems have as well (i.e. "if I depend on something that is installed by default on Ubuntu desktop then it's fine to leave out"). For example, a while ago I installed an Appimage GUI program on a headless server that I wanted to use via X11 forwarding. I ended up having to manually install a bunch of random packages (GTK stuff, fonts, etc) to get it to run. I see Appimage as basically the same as distributing Linux binaries via .tar.gz archives, except everything's in a single file.

>I wonder though, if I package say a .so file from nVidia, is that allowed by the license?

It won't work: drivers usually require exact (or more-or-less the same) kernel module version. That's why you need to explicitly exclude graphics libraries from being packaged into AppImage. This make it non-runnable on musl if you're trying to run it on glibc.

https://github.com/Zaraka/pkg2appimage/blob/master/excludeli...

Typically appimage packaging excludes the .so files that are expected to be provided by the base distro.

Any .so from nvidia is supposed to be one of those things. Because it also depends on the drivers etc.. provided by nvidia.

Also on a side note, a lot of .so files also depends on other files in /usr/share , /etc etc...

I recommend using an AppImage only for the happy path application frameworks they support (eg. Qt, Electron etc...). Otherwise you'd have to manually verify all the libraries you're bundling will work on your user's distros.

I wish AppImage was slightly more user friendly and did not require the user to specifically make it executable.

We fix this issue by distributing ours in a tar file with the executable bit set. Linux novices can just double click on the tar to exact it and double click again on the actual appimage.

Been doing it this way for years now, so it's well battle tested.

That kind of defeats the point of an AppImage though - you could just as well have a tar archive with a c classic collection of binaries + optional launcher script.

A single file is much better to manage on the eyes than a whole bunch of them, plus AppImages can be installed into the desktop using integration.

It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods and also very big for typical systems which include most libraries. They're good as a "compile once, run everywhere" approach but you're really accommodating edge cases here.

A "works in most cases" build should also be available for that that it would benefit. And if you can, why not provide specialized packages for the edge cases?

Of course, don't take my advice as-is, you should always thoroughly benchmark your software on real systems and choose the tradeoffs you're willing to make.

IMO one of the best features of AppImage is that it makes it easy to extract without needing external tools. It's usually pretty easy for me to look at an AppImage and write a PKGBUILD to make a native Arch package; the format already encodes what things need to be installed where, so it's only a question of whether the libraries it contains are the same versions of what I can pull in as dependencies (either from the main repos or the AUR). If they are, my job is basically already done, and if they aren't, I can either choose to include them in the package itself assuming I don't have anything conflicting (which is fine for local use even if it's not something that's usually tolerated when publishing a package) or stick with using the AppImage.

I agree. I've seen quite a few AUR packages built that way and I'm using a few myself too. The end user shouldn't be expected to do this though! :D

I agree with you as well! I don't think AppImage is perfect by any means. I do prefer it overall to the other commonly mentioned "universal" package tools like snap and flatpak, but I think my ideal system would essentially be a middleware protocol between build tool "frontends" and package format "backends" and then an ecosystem of tooling around that. LLVM and LSP haven't magically solved every issue with compilers and editor support for languages by any means, but they have significantly moved the needle on the average experience for quite a large number of end users even if they never directly touch any of the underlying protocols.

> It should be noted that AppImages tend to be noticeably slower at runtime than other packaging methods

'Noticeably slower' at what? I've run, e.g. xemu (original xbox emulator) as both manually built from source and via AppImage-based released and i never noticed any difference in performance. Same with other AppImage-based apps i've been using.

Do you refer to launching the app or something like that? TBH i cannot think of any other way an AppImage would be "slower".

Also from my experience, applications released using AppImages has been the most consistent by far at "just working" on my distro.

I don't think you can link shared objects into a static binary because you'd have to patch all instances where the code reads the PLT/GOT, but this can be arbitrarily mangled by the optimizer, and turn them back into relocations for the linker to then resolve them.

You can change the rpath though, which is sort of like an LD_LIBRARY_PATH baked into the object, which makes it relatively easy to bundle everything but libc with your binary.

edit: Mild correction, there is this: https://sourceforge.net/projects/statifier/ But the way this works is that it has the dynamic linker load everything (without ASLR / in a compact layout, presumably) and then dumps an image of the process. Everything else is just increasingly fancy ways of copying shared objects around and making ld.so prefer the bundled libraries.

15-30 years ago I managed a lot of commercial chip design EDA software that ran on Solaris and Linux. We had wrapper shell scripts for so many programs that used LD_LIBRARY_PATH and LD_PRELOAD to point to the specific versions of various libraries that each program needed. I used "ldd" which prints out the shared libraries a program uses.

Sounds painful. Better to distrib a separate bundle per platform and use RPATH

It you know what you're doing and scrupulous enough, you can package the software in a way that it works 25 years later.

https://xcancel.com/ValdikSS/status/1843044963443253678

There is this project "actually portable executable"/cosmopolitan libc https://github.com/jart/cosmopolitan that allows a compile once execute anywhere style type of C++ binary

Ermine: https://www.magicermine.com/

It works surprisingly well but their pricing is hidden and last time I contacted them as a student it was upwards of $350/year

Someone already mentioned AppImage, but I'd like to draw attention to this alternate implementation that executes as a POSIX shell script, making it possible to dynamic dispatch different programs on different architectures. e.g. a fat binary for ARM and x64.

https://github.com/mgord9518/shappimage

So autotools but for execution instead of compilation?

I don't think it's as simple as "run this one thing to package it", so if the process rather than the format is what you're looking for, this won't work, but that sounds a lot like how AppImages work from the user perspective. My understanding is that an AppImage is basically a static binary paired with a small filesystem image containing the "root" for the application (including the expected libraries under /usr/lib or wherever they belong). I don't line everything about the format, but overall it feels a lot less prescriptive than other "universal" packages like flatpak or snap, and the fact that you can easily extract it and pick out the pieces you want to repackage without needing any external tools (there are built-in flags on the binary like --appimage-extract) in helps a lot.

AppImage comes close to fulfilling this need:

https://appimage.github.io/appimagetool/

Myself, I've committed to using Lua for all my cross-platform development needs, and in that regard I find luastatic very, very useful ..

You can "package" all .so files you need into one file, there are many tools which do this (like a zip file).

But you can't take .so files and make one "static" binary out of them.

Well not a static binary in the sense that's commonly meant when speaking about static linking. But you can pack .so files into the executable as binary data and then dlopen the relevant memory ranges.

Yes, that's true.

But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work.

These are just strange and confusing from the end users' perspective.

> But I'm always a bit sceptical about such approaches. They are not universal. You still need glibc/musl to be the same on the target system. Also, if you compile againt new glibc version, but try to run on old glibc version, it might not work.

Why would you include most of your dynamic libraries but not your libc?

You could still run into problems if you (or your libraries) want to use syscalls that weren't available on older kernels or whatever.

You can include it, but

- either you use chroot, proot or similar to make /lib path contain your executable’s loader

- or you hardcode different loader path into your executable

Both are difficult for an end user.

This isn't that hard (that's not to say this is easy, it is tricky). Your executable should be a statically linked stub loader with an awful lot of data, the stub loader dynamically links your real executable (and libraries, including libc) from the data and runs it.

To add to this, in case of any remaining confusion. You can implement your own execve in userspace. [0] But the kernel's execve is a piece of machinery that invokes the loader so obviously it follows that you're free to make any changes you'd like to the overall process.

Bonus points if you add compression or encryption and manage to trip a virus scanner or three. [1]

[0] https://grugq.github.io/docs/ul_exec.txt

[1] https://blackhat.com/presentations/bh-usa-07/Yason/Whitepape...

No you can't. dlopen signature takes a file path, not a memory range. And if you start to save the libraries to the filesystem before opening them, there's no difference to shipping an archive directly and skip the trouble of your own archive code.

> But you can't take .so files and make one "static" binary out of them.

Yes you can!

This is more-or-less what unexec does

- https://news.ycombinator.com/item?id=21394916

For some reason nobody seems to like this sorcery, probably because it combines the worst of all worlds.

But there's almost[1] nothing special about what the dynamic linker is doing to get those .so files into memory that it can't arrange them in one big file ahead of time!

[1]: ASLR would be one of those things...

What if the library you use calls dlopen later? That’ll fail.

There is no universal, working way to do it. Only some hacks which work in some special cases.

> What if the library you use calls dlopen later? That’ll fail.

Nonsense. xemacs could absolutely call dlopen.

> There is no universal, working way to do it. Only some hacks which work in some special cases.

So you say, but I remember not too long ago you weren't even aware it was possible, and you clearly didn't check one of the most prominent users of this technique, so maybe you should also explain why I or anyone else should give a fuck about what you think is a "hack"?

  mkdir chroot
  cd chroot
  for lib in $(ldd ${executable} | grep -oE '/\S+'); do
    tgt="$(dirname ${lib})"
    mkdir -p .${tgt}
    cp ${lib} .${tgt}
  done
  mkdir -p .$(dirname ${executable})
  cp ${executable} .${executable}
  tar cf ../chroot-run-anywhere.tgz .

You're supposed to do this recursively for all the libs no?

Eg. Your App might just depend on libqt5gui.so but that libqt5gui.so might depend on some libxml etc...

Not to mention all the files from /usr/share etc... That your application might indirectly depend on.

> You're supposed to do this recursively

ldd works recursively.

> Not to mention all the files from /usr/share

Well yeah, there obviously cannot be a generic way to enumerate all the files a program might open...

Exodus (https://github.com/intoli/exodus) used to be good for this but is giving me a Python error these days.

A couple similar projects:

https://github.com/sigurd-dev/mkblob https://github.com/tweag/clodl

https://github.com/gokrazy/freeze is a minimal take on this

(not an endorsement, I do not use it, but I know of it)

https://www.magicermine.com/

Reply on news.ycombinator.com