Hacker News Reader

Deploying a hypervisor on top of a hypervisor (eg: a public cloud) usually makes very little sense so I don't see that as a trend. However, outside of the valley bubble many large sectors of the economy do not use public clouds (such as banking, health, defense) so hypervisors typically in the form of ESX is still very much a thing and now that we are seeing tighter integration with things like unikernels that is where the 'comeback' is coming from. I would almost go as far to say that it's becoming hip not to be using public cloud today.

Deploying a unikernel on top of commodity cloud is very achievable and easy today - I know because I'm involved with https://ops.city . I'd argue it's ridiculously easier than trying to deploy a container, clearly way safer and even faster and easier than using something like terraform.

I do very much agree with your impedance mismatch re: unix model thought though.

> However, outside of the valley bubble many large sectors of the economy do not use public clouds (such as banking, health, defense)

I work in traditional finance, outside of Silicon Valley and there is a huge drive to move to the public cloud. For the same reason banks do not own their office buildings anymore - managing computing infrastructure is not their core business and - to be honest - they are not very good at it. I know people who have to move servers and applications to a different data centre because the lease on an old DC is ending, it is a huge headache, and they would much rather concentrate on their core business.

I don't know who you specifically work for but I've been to 2 different banking tech conferences in the past month. JPMorgan mentioned they have 13 (or was it 16?) wholly owned datacenters. BoA's tech summit was just this past week in Palo Alto and they were highly negative of anything public cloud. For them it is an extremely rare thing for them to put anything in public cloud. A point they stressed throughout the summit is that the main thing they sell is 'trust' (eg: by holding people's money) and that is quite simply not something they are willing to delegate to others and that's the main driver for them but of course all the other things like cost, security, regulatory, and data gravity are drivers as well.

Since you cite JPMC, read this Sept 2019 interview with CEO Jamie Dimon:

https://www.geekwire.com/2019/interview-jpmorgan-chase-ceo-j...

“Jamie Dimon: The cloud and AI are real, huge, powerful and they’re going to change a lot for us. Of course, good companies are going to adapt to it in due course. They can’t just do it overnight, but it changes a lot more than just using the cloud — how you write your code, how you can do agile, it even changes how you organize your company. Managers start to think about using data in a far different way than they ever did before. It changes everything. It’s a huge opportunity.”

...

“When cloud first got started, there was no reason to use someone else’s data center or their networks. You know, I wasn’t completely wrong, if I can run my data center as efficiently as a cloud guy, which fundamentally is true.”

”But here’s what I missed: the bursting part of the cloud. It reduces the peak capacity that we’ve got to keep in our own data centers, and you can burst excess workloads into the cloud. And it directly relates to AI. There’s just so much compute power in one split second that I simply can’t do in one of my data centers. So if you want to apply AI and these tools, and also the services that are built into some of these platforms [the cloud offers an advantage.]”

// Disclaimer: I work at a global bank.

Hypervisor on hypervisor makes sense if you care about security. Your average FaaS-on-Kubernetes hosting services not from one of the tech giants is a far cry from AWS Firecracker (Lambda) level of isolation. E.g. you run a gaming server and wants to allow arbitrary user plugins, or perhaps a challenger fintech bank that wants some simple user scripting. You want to allow Python/Ruby/any lang except Lua/JS/WASM. You won't be able to find anything hosted that doesn't cost an arm and a leg. (On an unrelated note, if anyone from Zeit is reading this, can you comment on why now.sh container support was deprecated in V2? Was it because of cost?)

I totally agree that if you care about security you should be running in the opposite direction of k8s.

Google Cloud has this option (nested hypervisors) but not for security reasons - it's used to run other software that comes packaged as a vm yet that comes with a serious perf hit and which is why I don't see that ever catching on as a trend. Selling bare metal servers w/pre-installed hosted firecracker would make much more sense (if you want/need hosting).

If you are interested in isolating a particular application to a given interpreter or a given binary the aforementioned unikernels are precisely what you are looking for and those can be provisioned on t2.micros.

Your analysis is on point and brilliant. I would upvote this more if I could. Not enough people on HN understand the weaknesses of Kubernetes and existing self-hosted cloud software. (And more importantly, what hypervisors brings to the table)

> I totally agree that if you care about security you should be running in the opposite direction of k8s.

The parts of k8s that are sticky and hard to change are the API abstractions like Services, Deployments, and Nodes. That's also where a lot of the value add is. Like AWS cloud formation. The only part that is docker-specific are the docker images.

I don't think the sandboxing techniques (i.e. docker) will stay the same for long. There are already multiple initiatives to implement hypervisor-based pods. I think firecracker has a lot of inertia right now, and I'm pretty sure RKT containers with KVM isolation were in beta a couple year ago.

The point is, I don't see how kubernetes has all that much to do with security (at least as far as isolation goes). Docker alternatives will gain traction and k8s will offer them as pod runtimes. Maybe the direction you mean we should be running is the opposite one from Docker?

Wanted to share my first experience with the ops.city site. This is unsolicited, but I do hope it helps your team succeed:

I opened the 'Why' tab and found blurbs I wanted to see more about. For example, security through smaller code+attack surface sounds interesting, 'no ops' (didn't grok the bullets here, but I was interested to know more), and higher performance are all interesting

The bullets were interesting enough for me to want more, so I opened the video.

Unfortunately, the video was really vanilla and didn't expand on these concepts at all. It shows how easy ops.city is, but gives me nothing more on why it is better than ten other technologies that have similar 'easy intro' videos.

What I was hoping for was a 5-10 minute video focused on benefits of your approach vs alternatives. For example, some talk or demo on improved performance, data on how much code/attack surface is reduced, comments on OS/kernel memory footprint (this is a problem for us - clients like using 500MB "tiny" VMs on public clouds that hang running cadvisor), etc.

I could leave your site to answer these questions on my own. I did this and found some promising looking links over on the nanovm website. But that is an awkward experience - leaving your site to try and understand your offering.

Perhaps I'm not your target audience. For context, I do a minor amount of devops as part of delivering initial R&D products to multiple clients. We are not ops experts, but we try to deliver future proof work, which involves us staying apprised of new approaches and making them available to clients when they are a fit.

> From far away, Windows and Fuchsia are very similar, but Windows have to contend with decades of code compatibility needs even if they want to start shoving everything into a locked down app)

DOS and such were designed around everything on the device being shared, with conflicts resolved at the application layer. Windows was built to maintain some (even ideological) compatibility with that world, while adding some of the benefits of separation and sharing (for example, moving away from the world where every piece of a job could crash/co-opt/exfiltrate the entire device).

Unix (Multics, CTSS, ITSS, etc) was built to enable a world where multiple jobs (not processes; even then lots of jobs were multi-process) and multiple people could share the device. Threads, jails, cgroups, etc were added to move to a world where sharing wasn’t entirely cooperative and trusting. VMs, containers, and hypervisors are on the recent end of that same movement, along with wasm and JS sandboxing.

In the end, it’s a balancing act between performance (speed, power, cost) versus safety, often starting from different points. Exo-, uni-, and library-kernels are similar efforts that haven’t (yet?) caught on, but there’s a pretty clear direction of movement towards the strongest isolation that our (currently quite flawed) hardware can afford.

This comment made me remember my first Mac where when you launched an app it was given complete control over the hardware, with ASM calls to the Mac Toolbox.

Remembering these times it makes me wonder why should there be only one direction, and that maybe there is an alternative to putting all our most sensitive data in every internet connected devices? In old times when you had a computer to do some hobby like games, photo editing, music, etc... the data was only related to that activity, and in the worst case of nuking the hard drive you lost only that work.

Other than the newer web tech initiatives like WebAssembly/JS sandboxing

Yes, this is the lightest weight isolation that's somewhat secure for random code.. And considering that we regularly find exploits that escape that sandbox, I would never use it to run some random persons code on a server. Yes, we do that all the time in the browser, but that's a more limited environment where a single user is choosing what to run... whereas on a server, anyone who wants to attack the service can just upload whatever code they want.

I know Cloudflare is doing it in Cloudflare Workers... and I think that's crazy. They're inspecting the code, and they added OS sandboxing... so that's a little better.. but trusting the JS sandbox in that environment is just nuts. Wouldn't surprise me if this ended up being another cloudbleed.

If you want to run random code, the only safe option right now is to use a heavier form of isolation like a VM (and even that is not perfect). The performance loss is the necessary trade off to allow random people to run code (some of whom are deliberately attacking the service). This is the downside to the "cloud"... performance losses to allow random people to share servers.

An idea I've been toying around with but don't have time to fully explore or implement:

The unit of isolation should be the library, not the process. Each time you call into another developer's library, the library should have only the privileges you explicitly pass to it as capabilities, and you should be able to specify whether these capabilities may be delegated outside the receiving library and whether they can outlive the method call. So every time you call outside your own code assembly, you know that that call can't access the network, can't write to disk, can't display anything to screen. If you need it to, you explicitly pass the connection to a particular host or a handle to a particular directory, and then it can't access anything outside of that.

Not sure if this is feasible performance-wise: if you do it at the binary level, it seems like it'd require remapping page tables and flushing the TLB with every external function call, which'd absolutely kill frameworks that have to execute callbacks many times a second. But there's a certain intuitive sense to making the unit of trust & communication the code written by a certain organization, and it also fixes attacks like npm malicious modules that haven't gotten all that much attention nor have effective mitigations beyond "audit all your dependencies".

Given that some of the most commonly used external libraries do networking, IO, graphics and encryption, I don't think this would really be feasible.

Presumably they'd have to be rewritten, but if you're talking a new OS anyway everything's gotten be rewritten, because you won't have the standard UNIX syscalls. This is not something to embark on lightly, which is why I haven't embarked on it yet.

But imagine a future where using a traditional OS means there's close to a 100% chance that you will have your identity and/or financial assets stolen, or where these computers are widely used in the military and a security failure literally means you're dead. We're headed there anyway, judging from all the headlines on data breaches and identity theft. There'll be system failure at that point, as well as a strong incentive to rewrite everything as a secure system.

It's feasible in the same way bounds checking is feasible. Everything could have a capability token attached to it and checked. Compiler could generate all the language level checks, kernel could do OS level checks. Main program could have some built in capabilities and capabilities it gets from the other OS processes, it could then pass them or devise new ones to library functions, they could pass them to other library functions and so on.

But that means you have to upload raw source code. It's closer to Apple's app store deployment than to what current cloud hosting offers (arbitrary compute power for anything but crypto mining).

Not necessarily raw source code, could be a code compiled into an intermediate representation, but it has to have capability-based security deeply integrated into it, like bounds checking is into programming languages.

> But that means you have to upload raw source code.

And what is wrong with that? JavaScript source-code-only is by orders of magnitude the most popular and successful software distribution method.

In a business environment the code you run is typically the code you wrote.

You generally don't run unfamiliar code if you do, you are not being careful. All it takes is to have a container contain a backdoor that phones home, and essentially give someone access to inside of your network.

This gets silly, though. If you are going for full isolation where applications can't just let the user have data, then you have to coordinate every app talking to every other app.

This is why the share button on a phone is ridiculous. Instead of just copy to clip board, you get a ton of options.

Is it safer? Hard to say. I can do less and have less capabilities without more active work from the developers.

There is something to be said for this, of course. But it seems a losing game. I contend there is no technical panacea to security versus convenience.

There is no technical panacea, but containers are a useful abstraction. Containers bring back the original intent of the unix process/user security model, allowing you to scope each process to have it's own 'private' filesystem to get around the broken way executables are built, packaged, and linked.

This doesn't save you if you do dumb things, like have open ports that allow administrative access that are exposed to the rest of your (potentially malicious) system. They do force you to think about some of the questions, though, like "Where is my data stored and how do other tools interact with it?"

Most of the problems that people complain containers don't solve are ones that are open ended, and have no definite solution.

containers are a great tool for corporate servers. For end user machines, it is a tougher sell.

What do you think of the permissions associated with Android apps? Each developer gets a separate and cordoned user account on your phone. They only have access to private storage unless they request it through special APIs.

Android apps are run in containers, for some broad sense of the word. That's how it should be. Every app I run should be run with a simply configured set of permissions, and "per app" or "as user" makes sense, even in the desktop space. That's what snap originally tried to do (but failed at, and have now broken)

Isn't that where I said it can get silly? I don't want developers having an account on my phone. I just want to process some data on my phone.

Consider that on my phone, I am limited to accessing my data for most apps through the apps. On my computer, I typically know where the data is stored.

> This is why the share button on a phone is ridiculous. Instead of just copy to clip board, you get a ton of options.

Off topic, but I don't understand your objection...

On Android Chrome the share button shows you a context sensitive choice of apps that will accept the object (a link, some text, etc), followed by a second choice of what the app can do with the object.

The flow for copy-then-paste is more convoluted and is waaay less discoverable. Copy then paste is still available for the situations where it has an advantage.

copy-then-paste is only more convoluted if you haven't learned how computers have worked for a long time.

And things have gotten some better for the phone option. But I don't have or want all apps installed. So, my sharing is dominated by copy/paste, still.