Comment by jacques_chester | Hacker News Reader

Comment by jacques_chester | Hacker News Reader

Parent Full thread

jacques_chester

Computer systems, which are deterministic, have long since passed out of the realm of being understandable.

They're chaotic systems[1]. Parameters which are below the threshold of detection can drastically affect their trajectory in phase space. Accidental complexity has zoomed off to live a life all its own, and that takeoff has been continuous for decades [2].

Everything we build around software that's not directly bearing on the problem domain is chipping away at this or that aspect of the ungoverned bits. As we corral each level of new unpredictability, we merely set the scene of the next level of complexity.

There is no universal solution. There is no final boss which, once defeated, leads to the scrolling names and 8-bit themesong. It never stops, ever, because the value of systems that frequently fail unpredictably is lower than the cost of flawless systems and higher than the value of not building such systems.

Edit: sounds very unhelpful, even defeatist. I know. I'm deeply ambivalent about what can be done versus what we should assume is possible. I'm a fan of serious software engineering and tool support and all that jazz ... yet thoroughly pessimistic about the limits of the possible. Deep sighs for every occasion!

[1] http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign... (warning, self linkage)

[2] http://www.cs.nott.ac.uk/~cah/G51ISS/Documents/NoSilverBulle...

They're not even deterministic. Even if your input is totally ordered, entropy from the memory subsystem will bubble up to communicating threads if you have more than one. Sometimes one thread will read the newly written value, sometimes the old. That said, there is some clever research out of UW into how to make such systems deterministic while still recovering almost all parallelism.

I think the more important point jacques_chester was making is that even in the case where computer systems are 100% deterministic it doesn't matter because they are so complex they become chaotic systems. And then hypothetical determinism won't save you because there are too many variables in play simultaneously, and even the slightest variation in any one of them could change the behavior completely.

Sorry guys, but I don't buy your defeatist arguments for one second. Sure using traditional tools, you may be right. However, all we have to do is look at the brain with its 100 trillion Connections (axioms) or a CPU with its billions of transistors to see that your logic is flawed. If it was not possible to design fault tolerant systems on large scale neither of these two examples could exist.

I don't claim to have the answers, but a few years ago, I had to help out some EE's write some testing software in LabVIEW. The first thing that shocked me was the system was immune to bad data. Secondly, how elegantly it took advantage of our multiprocessing and multithreading hardware.

What I wish systems researchers would work on is concurrent, signal-based, synchronous languages. This is a winnable game, but we must swallow the red pill and change the rules of the game with new tools and fresh ideas.

Just reading your second reference, the bit about expert systems is pretty interesting . . . what's your stance on this? It seems that, given sufficient instrumentation information, one could probably construct a fairly decent expert system that might be able to shed some light onto these sorts of problems. (Keep in mind the fact that I'm still very much a junior software developer!)

The example of FDs from the original article seems like a good one; a properly developed expert system might have been able to notice a lack of changes in the current software environment and suggest looking elsewhere. (Problem is, if I got that response from my automated problem-solver I'd just ignore it and keep looking . . .)

So, any insightful points to crush my youthful idealism? :)

Expert systems for monitoring and management of large server farms and networks? Been there. Done that. Got the T-shirt. Wrote papers. Gave one at an AAAI IAAI conference at Stanford. Shipped two commercial products.

First problem: Need some 'experts'. So, only know about problems the experts do. Their knowledge is supposed to be mostly just empirical, from their 'expert experience'. Even if they have 'deep knowledge', that is, how the systems work internally (e.g., the engine connects to the torque converter connects to the transmission connects to the rear differential connects to the rear wheels), the specific problems they know about are usually just the ones they have encountered. So, essentially can only address problems seen before and are well understood.

But for the problems of the OP, they were seen for the first time. Bummer. Actually, with irony, if the problems have been seen before and are well understood, then why the heck have they not already been solved?

Second, with expert systems, we're working just intuitively from just experience. So, we have little idea if what we are doing is the best possible in any reasonable sense or even at all good. In particular, in monitoring, the main, first operational problem on the 'bridge' or in the 'NOC' (network operations center) is the false alarm rate too high with no good way to lower it (except just to ignore some alarms, but we should be able to do much better).

Net, for a serious attack on system monitoring and management, I can't recommend taking expert systems very seriously.

Would love to get an email (address in profile) from you to chat about some tangental stuff (re: intelligence in monitoring systems)

This is a great topic, the idea that contemporary computer systems are not atomically understandable or model-able. And it's a topic that's almost entirely absent from computer science discourse.

We passed the point quite a while back where "applied CS" -- what it's like to build and maintain scalable systems in the real world -- looks a lot more like anthropology than like mathematics.

We have tools like sampling profilers, loggers that do stateful packet inspection on a small percentage of traffic, and sophisticated A/B testing frameworks that reason about user populations rather than individuals. But I think we could probably build a whole 'nother generation of these tools by thinking about this problem of understanding complex systems from some new perspectives. I nominate anthropology as a discipline that has many years under its belt wrestling with the impossibility of complete description. (And wrestling with other tough issues, sometimes not well, but that's another conversation.)

I'm not talking about the kind of "anthropology" work that companies like Intel have done for a long time, which are firmly in the social sciences tradition and are really user studies. (Though that work is valuable.) I'm talking about porting ideas such as "thick description" to our technical toolkit.

  http://en.wikipedia.org/wiki/Thick_description

No one's asking for a solution to complexity; I agree there's no such thing. What we need are better ways to manage complexity -- and that is absolutely within our grasp.

Reply on news.ycombinator.com