It's strange to me that Alpine Linux won over Unikernels. I guess it makes sense, once you add logging, observability, printf-debugging (just kidding) to your unikernel, you basically have a full OS anyway.
I mean -- isn't that all the good stuff?
Like -- I boot this thing and I don't really have a filesystem in the way we know it. First of all, that's confusing (I say this as a ZFS devotee). Second, I have no idea why this is acting crazy in production, or in testing, compared to the Linux version? Etc., etc.
This is a rather common misconception. Nanos (and many other unikernel projects) have filesystems. Most of the applications we target are webapp servers, databases, etc. They all want to, at the very least, write to a tmp file and more commonly want to load lots of files. What they don't have is an interactive userland.
Logging and observability are valuable. But running a full multiuser OS with kernel and userland for your one process, adding extra context switches and what have you to everything you do, just for the rare occasions where you log in and run a few commands, seems crazy to me. As long as it can output logs/traces/etc. to a collector (which is what I'd want to do anyway, no-one wants to have to SSH to each separate instance to look at log files on the local filesystem) and there's a way to attach a debugger (e.g. the Java style where your debugger connects to it on a given port), I don't see any advantage from having e.g. a filesystem per se. Likewise being able to run it locally the same way as production is important, but that doesn't have to mean running it as a Linux process - people are happy running Docker images for local dev, running a unikernel in a VM isn't a lot different from that.
If I want to have a bunch of instances scaled up and wired together network wise and defined in a nice yaml file, kubernetes makes this easy and replicable and I just have to push a container image somewhere like GHCR and have it all rolled out with CI.
You can deploy almost the same system on top of the managed kubernetes offerings from any of the big cloud providers, so other than working out a few differences generally your config will work on anything.
If I want to do this with unikernels I'm lost, would I do it with AWS services like EC2 and auto scaling groups or something? How do I get it all wired together and configured? Perhaps terraform but I'm not sure.
It's all about the maturity of the glue and the architectures, not to mention training and familiarity. Does (mystery unikernels solution) have the equivalent of custom controllers somewhere in that stack?
K8s provides an abstraction layer over resources and lets you define your system holistically.
If this exists in a mature and flexible state for unikernels I'm all ears.
What real advantage is there to using unikernels over alpine linux + k8s (or whatever) orchestration?
This is a key point that many many people struggle with until they actually push their unikernels out to the cloud. It is one thing to run ops locally but try doing a 'ops image create -t aws' followed by 'ops instance create -t aws' and then you'll understand how we manage to push a lot of the scheduling work back onto the cloud provider itself.
It would be cool to write some sort of book like, ops for k8s people, where it takes a lot of the common concepts in k8s and explains how you'd do them with this system.
ops seems like decent a CLI to get things done but as far as I can see not really an orchestration system in the same way as k8s? Perhaps I misunderstood.
There is a terraform provider that we ended up writing for one company but only because they were a (large) hashicorp shop and nothing went to prod unless it was in tf: https://docs.ops.city/ops/terraform This doesn't really do much though. It's only there to check a box and fit into an existing paradigm.
A lot of the orchestration bits that k8s provides is because when it was initially adopted there wasn't a great way to attach networks/storage/etc. to various containers like people had for on the clouds with vms. OPS just re-uses the same primitives you'l find on every single cloud out there. If you need more advanced functionality like auto-scaling that also can utilize the clouds ASGs.
I guess this means your environment is very "native" to a specific cloud however and not as easily transferrable.
One thing I did like about k8s is that it's easy to self-host with microk8s but tbh, I think in production you'd be using a cloud provider's managed k8s (e.g. AKS) anyway.
Looking at how to attach/configure a ElastiCache, RDS, or some EFS to a project also.
What's the best way to runtime configure an image when you create it as an instance? I couldn't see any options on ops instance create for environment variables or anything like that but this also seems like, weird - IE you wouldn't give a VM environment variables, but with a normal EC2 instance there's an agent that handles this sort of thing. Would you have to bake the configuration into the image? Also how does one output the ELF for the image? Is it just the file of [image name] in ~/.ops/images ?
Or is it the case that terraform should build a correctly pre-configured image as part of the apply process?w
Always good to have other options, of course! I do appreciate the minimalism. This is exciting :)
The terraform docs example uses GCP, perhaps if I figure out how to do it on AWS I could PR the docs with an example.
At that stage where my questions lead to more questions.