Hacker News Reader

Comment by latchkey | Hacker News Reader

> Functionally, MicroShift repackages OpenShift core components* into a single binary that weighs in at a relatively tiny 160MB executable (without any compression/optimization).

Sorry, 160meg isn't 'relatively tiny'.

I have many thousands of machines running in multiple datacenters and even getting a ~4mb binary distributed onto them without saturating the network (100mbit) and slowing everything else down, is a bit of a challenge.

Edit: It was just over 4mb using .gz and I recently changed to use xz, which decreased the size by about 1mb and I was excited about that given the scale that I operate at.

View on HN

160mb is indeed relatively tiny compared to a full k8s distribution. The "i have issues distributing a tiny 4mb binary to my severs" sized hole that you've dug yourself doesn't change that.

Relative to Kubernetes (well, specifically OpenShift, which is Kubernetes packaged with a lot of commonly-needed extra functionality / tools)

A "hello world" written in Go is 2mb to begin with so ~4mb is a bit unrealistic for any substantial piece of software written in Go. Although if that colors your opinion of Go itself, you're certainly allowed to have that opinion :)

Your logic doesn't work because adding functionality doesn't necessary translate to a huge increase in binary size.

The complex app (15k lines of code) that is distributed to these machines, written in Go, is just over 3mb compressed with xz.

Plus this is still many orders of magnitude off with trying to get 160mb off to mobile devices per the image in the article. It is a non-starter.

I'm going to take a guess that even a "mini" distribution of Kubernetes is more than 15k lines of code though. k3s is quite a bit smaller than this but it's still only described as "<50 megabytes"

I'm not sure the point of your argument. Somehow we should just accept 160meg downloads to a phone, because why?

I don't understand the point of your argument, because nobody (including the announcement above) is talking about phones.

From the post:

> Imagine one of these small devices located in a vehicle to run AI algorithms for autonomous driving, or being able to monitor oil and gas plants in remote locations to perform predictive maintenance, or running workloads in a satellite as you would do in your own laptop. Now contrast this to centralized, highly controlled data centers where power and network conditions are usually very stable thanks to high available infrastructure — this is one of the key differences that define edge environments.

> Field-deployed devices are often Single Board Computers (SBCs) chosen based on performance-to-energy/cost ratio, usually with lower-end memory and CPU options. These devices are centrally imaged by the manufacturer or the end user’s central IT before getting shipped to remote sites such as roof-top cabinets housing 5G antennas, manufacturing plants, etc.

> At the remote site, a technician will screw the device to the wall, plug the power and the network cables and the work is done. Provisioning of these devices is “plug & go” with no console, keyboard or qualified personnel. In addition, these systems lack out-of-band management controllers so the provisioning model totally differs from those that we use with regular full-size servers.

I don't read this and think "phones". This sounds like it's targeted at embedded industrial / telecom devices. (At least based on the examples they chose, I'm sure you could use it for other things).

The word "mobile" doesn't actually appear anywhere on the page so I'm not sure where you got "mobile devices" from.

I recently wrote a single-process k8s distribution (for tests). Compiled into a linux/amd64 binary with no special settings it's 183M with go1.18beta1.

Kubernetes is a lot of code, plain and simple.

I found go compiler isn't very efficient in not linking in the dead code. I ran into it by switching from aws-sdk-go to aws-sdk-go-v2 and the binary jumped from 27Mb to 66Mb [1]

Granted fully featured app will likely use all of the module's code so it's not a factor.

[1] https://github.com/aws/aws-sdk-go-v2/discussions/1532

Yeah, I have a feeling that these giant autogenerated clients are tough to optimize. Too many layers of indirection; even the compiler is confused and just crosses its fingers and hopes it doesn't have to investigate too much.

If you use the reflection package in Go, the dead code elimination optimization step does not run if you use reflect.Method to look up a method because you can use reflect to reach those methods and the linker preserves the possibility.

Unfortunately, almost every large client library has public methods AND references someone who calls the reflection method somewhere, so in many cases you get the full tree of dependencies.

It’s unfortunate because any one dependency can bring in reflect and trigger the behavior. It hurts large generated method libraries the most since you usually only use a small subset.

See https://go.dev/src/cmd/link/internal/ld/deadcode.go line 318

That makes sense. Reflection is the enemy of anyone trying to produce a compact binary (or webpack bundle). A quick grep of the k8s codebase and there are plenty of situations that kill the dead code elimination:

staging/src/k8s.io/apimachinery/pkg/runtime/scheme.go: if m := reflect.ValueOf(obj).MethodByName("DeepCopyInto"); m.IsValid() && m.Type().NumIn() == 1 && m.Type().NumOut() == 0 && m.Type().In(0) == reflect.TypeOf(obj) {

That's quite the line of code too.

> The complex app (15k lines of code)

I would argue that 15K lines of code is far from a complex app as is normally understood. It's a rather small app, especially in Go which isn't exactly the most expressive language out there.

They reported the 160mb as uncompressed (and probably unstripped?) size. If you compress that it will be aroudn 50mb, still more than your 3mb xz example but at least you're now comparing apples to apples

> I have many thousands of machines running in multiple datacenters and even getting a ~4mb binary distributed onto them without saturating the network (100mbit) and slowing everything else down, is a bit of a challenge.

Maybe I suggest CAS (Content-addressable storage) or something similar for distributing it instead? I've had good success using torrents for distributing large binaries to a large fleet of servers (that were also in close proximity to each other physically, in clusters with some more distance between them) relatively easily.

Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this.

The machines don't all need to be running the same version of the binary at the same time, so I took a simpler approach, which is that each machine checks for updates on a random schedule over a configureable amount of time. This distributes the load evenly and everything becomes eventually consistent. After about an hour everything is updated without issues.

I use CloudFlare workers to cache at the edge. On a push to master, the binary is built in Github CI and a release is made after all the tests pass. There is a simple json file where I can define release channels for a specific version for a specific CIDR (also on a release ci/cd as well, so I can validate the json with tests). I can upgrade/downgrade and test on subsets of machines.

The machines on their random schedule hit the CF worker which checks the cached json file and either returns a 304 or the binary to install depending on the parameters passed in on the query string (current version, ip address, etc.). Binary is downloaded, installed and then it quits and systemd restarts the new version.

Works great.

I’d seriously look into BitTorrent for this use case, as it sounds ideal. You can even configure your client to run a script after torrents are completed, so you could script the update migration with minimal code. You can also setup upload/download limits easily.

I think Resilio Sync might also be a good option; I think it may even use BitTorrent internally. (It was formerly known as BitTorrent Sync, not sure why they changed the name.)

Over engineering. You're trying to solve a problem that I don't have. What I have now works great with a page of well tested code and doesn't have the complexity of BT.

I'll duplicate the comment above that the machines are all on separate vlans and don't really talk to many other machines.

You know your stack best. I appreciate the insights you have into your process and workflow. Do you use anything special to bridge vlans like Tailscale or WireGuard?

Thanks, indeed. P2P file distribution is an interesting idea, but doesn't really solve any major problems here. ~3mb file to a lot of machines in an hour a few times a week. Simplest solution is that each machine can just download the file itself.

The data centers don't need to talk to each other at all. We use wg at each dc for vpn access. I do need to spend a bit more time with my own wg client setup so that I can switch between dc's more easily... right now it is kind of a manual process and that is definitely a pain. One of these days...

I’ve heard that there is third party tooling that makes managing wg easier but I haven’t really looked into that space. I need to do it though, as that’s one of the major pain points of wg, that each client needs to be whitelisted serverside essentially.

Edit: Maybe something simple like rsync or rclone would be best for the update sync/distribution mechanism rather than BT.

A binary diff (or even plugin) mechanism could potentially optimize the download size even further. Such that only the parts of the app that change need to be downloaded.

That said, for now, downloading the binary really is a solved problem for me, with a relatively simple solution. I'd rather work on new features that drive revenue. =)

It’s always good to have some Friday afternoon projects around too! :P

> Thanks for the suggestion, but as weird as this sounds, we also don't have central servers to use at the data centers for this.

Same here, hence the suggestion of using P2P software (BitTorrent) for letting clients fetching the data from each other (together with initial entrypoint for the deployment, obviously), and you'll avoid the congestion issue as clients will fetch the data from whatever node is nearest, and configured properly will only fetch data outside the internal network once, after that it's all local data transfer (within the same data center).

The machines are all on separate vlans. Only a few of them can talk to each other at a time. There isn't a huge amount of benefit there.

Notes re: distributed temporal Data Locality, package mirroring, and CAS such as IPFS: "Draft PEP: PyPI cost solutions: CI, mirrors, containers, and caching to scale" (2020) https://groups.google.com/g/pypa-dev/c/Pdnoi8UeFZ8 https://discuss.python.org/t/draft-pep-pypi-cost-solutions-c...

apt-transport-ipfs: https://github.com/JaquerEspeis/apt-transport-ipfs

Gx: https://github.com/whyrusleeping/gx

IPFS: https://wiki.archlinux.org/title/InterPlanetary_File_System

> without saturating the network (100mbit)

Datacenter to datacenter and only 100Mb? Clearly more to the story here.... :)

Within the data center.

That's even worse though.

The machines don't require a lot of bandwidth.

my fridge has gigabit networking

Even consumer-grade router network has been gigabit for the last decade though.

That's great! These mobos have 100Mb.

"relatively" is the operative word there. Compared to regular/full openshift, it is tiny. I would imagine they chose the word "relative" because in absolute terms nobody would call 160mb tiny.

You may find murder[1] / Herd[2] / Horde[3] -type tool of some use.

[1] https://github.com/lg/murder

[2] https://github.com/russss/Herd

[3] https://github.com/naterh/Horde

If you're really that sensitive to size, may want to try 7z. I can usually get a few % smaller archive sizes than xz with faster decompression to boot. Of course then you might need to install a 7z lib which could be an issue.

Thanks, I just tested that idea out:

  Input binary size: 12996608
  Folder contains a few more bytes for the service file and installer script.

  4476005 (gz)
  3421456 (xz)
  3447940 (7Z)

  tar c -C ./build $(BINARY) | gzip -9 - > $(PKG_NAME_GZ)
  tar c -C ./build $(BINARY) | xz -z -9e - > $(PKG_NAME_XZ)
  tar c -C ./build $(BINARY) | 7z a -si $(PKG_NAME_7Z)

I played around with the compression options on xz. If you have some suggestions on improving 7z, I'm all ears.

Decompression time isn't an issue here.

Installing 7z on the hosts isn't great, but could be done.

You could try the compression options for 7z, not sure if they would help in your case since I don't know the defaults off hand. Here's an example from the man page.

> ... | 7z a -si -m0=lzma -mx=9 -mfb=64 -md=32m

https://linux.die.net/man/1/7za

Thanks again. Using those exact options.

  3421498 (7z)
  3420832 (xz)

Wild that xz is not the same every time.

Sounds like a use case for BitTorrent.

Separate vlans

Reply on news.ycombinator.com