Hacker News Reader

Comment by bluepizza | Hacker News Reader

I think the issue is that bringing the application down might mean cutting short concurrent ongoing requests, especially requests that will result in data mutation of some sort.

Otherwise, some situations simply don't warrant a full shutdown, and it might be okay to run the application in degraded mode.

View on HN

"I think the issue is that bringing the application down might mean cutting short concurrent ongoing requests, especially requests that will result in data mutation of some sort."

Yes but what is worse is silently corrupting the data or the state because of running in buggy state.

This is a false choice.

If you don't know why a thing that's supposed to never be null ended up being null, you don't know what the state of your app is.

If you don't know what the state of your app is, how do you prevent data corruption or logical errors in further execution?

You can "drop" that request which fails instead of crashing the whole app (and dropping all other requests too).

Sure. You wouldn't want a webserver to crash if someone sends a malformed request.

I'd have to think long and hard about each individual case of running in degraded mode though. Sometimes that's appropriate: an OS kernel should keep going if someone unplugs a keyboard. Other times it's not: it may be better for a database to fail than to return the wrong set of rows because of a storage error.

That's exactly what the attacker wants you to do after their exploit runs: ignore the warning signs.

You don't ignore it. You track errors. What you don't do is crash the server for all users, giving an attacker an easy way to DoS you.

A DoS might be the better option vs. say, data exfiltration.

Most bugs aren't going to create any risk for data exfiltration. In most real application servers (which are very rarely written in C or C++ these days), requests are almost completely isolated from each other except to the extent that they interact with a database. If you detect a bug in one request, you just abort the one request, and there's likely no way it could affect others.

This is part of why something like Rust is usable at all; in the real world a lot of logic has straightforward, linear lifecycles. To the extent that it doesn't, you can push the long-lived state into something like an external database, and now your application has straightforward lifecycles again where the goal of a task is to produce commands to manipulate the database and then exit.

Sure, but i was talking about an individual process. If you don't know what state it's in, you simply can't trust it to run anymore. That's all.

Except you usually can because the state isn't completely unknown. You might not expect some field in a structure to be null, but you still know for example that there's no way for one request to have a reference to another, so you just abort the one request and continue.

No, if you have been compromised, you cannot make these assumptions.

And what does DOS attacker want you to do? Not crashing the whole service to deny others of the service?

That is a valid tradeoff in many situations, yes.

> If you don't know what the state of your app is, how do you prevent data corruption or logical errors in further execution?

Even worse, you might be in an unknown state because someone is trying to exploit a vulnerability.

If you crash then you've handed them a denial of service vulnerability.

That's an issue handled higher up the stack with process isolation etc. It's still not ok to continue running a process that is in an unknown state.

> If you don't know what the state of your app is, how do you prevent data corruption or logical errors in further execution?

There are a lot of patterns for this. Its perfectly fine and often desirable to scope the blast radius of an error short of crashing everything.

OSes shouldn't crash because a process had an error. Servers shouldn't crash because a request had an error. Missing textures shouldn't crash your game. Cars shouldn't crash because the infotainment system had an error.

If you can actually isolate state well enough, and code every isolated component in a way that assumes that all state external to it is untrusted, sure.

How often do you see code written this way?

This is basically all code I've worked on. You have a parsing/validation layer that passes data to your logic layer. I could imagine it working less well for something like a game where your state lives longer than 2 ms and an external database is too slow, but for application servers that manipulate database entries or whatever it's completely normal.

In most real-world application programming languages (i.e. not C and C++), you don't really have the ability to access arbitrary memory, so if you know you never gave task B a reference to task A or its resources, then you know task B couldn't possibly interfere with task A. It's not dissimilar to two processes being unable to interfere with each other when they have different logical address spaces. If B does something odd, you just abort it and continue with A. In something like an application server, it is completely normal for requests to have minimal shared state internal to the application (e.g. a connection pool might be the only shared object, and has a relatively small boundary that doesn't allow its clients to directly manipulate its own internals).

Reply on news.ycombinator.com