Comment by gwern | Hacker News Reader

Comment by gwern | Hacker News Reader

Parent Full thread

> Because let's face it, even if the kernel build process was some super-efficient thing, real life isn't that anyway. I guarantee you that the "tons of small scripts etc" that the kernel build does is a real load somewhere totally unrelated. Optimizing page faults will help other loads.

This is true as far as it goes, but a silly argument. Let's apply a reversal test (http://www.nickbostrom.com/ethics/statusquo.pdf‎).

Suppose the kernel build were as efficient as Chrome's is claimed to be on this page (<5s) and wasn't stressing his system. Would Linus then approve of anyone submitting patches to deliberately slow down the Linux kernel build just to show up slownesses in page fault and encourage kernel devs to spend time on optimizing that?

No, of course not! That would be idiocy and the person submitting the patches would probably be banned by Linus in his titanic rage. However, since the slow build & page faults is the status quo, Linus is making lemonade of it...

I think you missed the point. He's not denying that make is slow and could be faster or that most of it doesn't have to do with page-faults. All he's saying is that page-faults in general are slower and speeding them up would improve lots of various different kinds of loads besides just the kernel build system (Which would probably not improve very much from just that alone).

I'm sure he'd be happy to take patches that make 'make' faster, but that's simply not what he's trying to address here.

So, that is actually an interesting thought experiment, thanks for that.

However I'm not sure it is directly applicable here. There are two courses of action that could solve this problem:

A, an action that improves kernel builds

B, an action that improves several workloads

For A and B of similar cost, it makes sense to do action B in preference to action A.

Your argument speaks to A being of positive utility, but given a finite knapsack of effort smaller than the set of possible actions that fit in the knapsack, a greedy algorithm that places any positive item into the knapsack is not optimal.

I don't follow your reasoning here, but let me expand my observation further: Linus can A. improve his build system (low-hanging fruit which the Chrome numbers suggest could yield an order of magnitude better performance), or B. he can search among a variety of difficult unlikely-to-yield-major improvements (like yelling at Intel engineers 'make it go faster!') which will improve his build system and also other hypothetical loads (which are unlikely to be large gains if any at all; what, is Intel too ignorant to try to make the TLB fast?). He is claiming B is better in part because of the hypothetical loads makes it better in total.

Most people would consider A a more reasonable reaction, especially after hearing that Linus's best idea for doing B is apparently going all the way down to the hardware level in search of some improvement. We can see this by intuitively asking what people's reactions would be to a proposal to induce B if the equilibrium were already at A.

On the other hand, Linus is one of a handful of people in the world who may be in a position to get results by yelling at Intel engineers to 'make it go faster!'. This isn't because Intel is too ignorant to do things on their own, but because practically everything can be optimized further, and Linus may have enough sway to focus the engineers' attention on the problem that he wants solved.

Personally, I don't care much about the speed of the Linux kernel build system, but I do care about the speed with which page faults are handled by the CPU. Even if the chances of success are lower, if he is able to succeed in speeding up every page fault on future Intel processors, I would consider that a much greater good.

The real problem (as I see it) is that I think he's trying to optimize the wrong thing. His worst-case test is based on trying to repeatedly fault in an uncacheable page: every lookup TLB lookup fails at every level of the cache. Likely, Intel has chosen to optimize the real situation where page translations are cached when they are repeatedly accessed.

Most people would consider A a more reasonable reaction

Improve your dev cycle inefficiency that you don't really mind, or improve your product? What's unreasonable about the latter?

especially after hearing that Linus's best idea for doing B is apparently going all the way down to the hardware level

He's building a kernel, not a webapp. "All the way down" is a single layer.

Intuition is probably not a good guide here. An improvement to the kernel build process is relevant to the thousands of machines that are used for kernel development; an improvement to the page fault speed is relevant to the over 1 billion devices that run the kernel. That's a pretty big multiplier.

(I believe the GPs reasoning is that "improve the build system" and "retard the build system" are not symmetrical, because both directions require positive effort to be expended).

On the other hand, if build system improvements truly are low-hanging fruit, then there are a lot of people out there who are capable of helping. A much smaller number of people are capable of productively working on page fault optimizations, so having someone like Linus address low hanging build system fruit instead would be a waste of talent.

We have a different prior expectation of the utility and feasibility of action B. In my view, Linus is uniquely suited to do something useful here, and his chance of success is high. In your view, Intel is already doing the best it can, and his likelihood of success is low. Our positions follow.

A isn't really very low-hanging though, rewriting the build-system for something as complex as the kernel would be quite the undertaking even just from a technical standpoint, let alone that you now have to change thousands of peoples workflow by having them install and use a new build-system (some of those people are probably dedicated to just maintaining the build-system too, so their job would radically change) as well as figure out a suitable build-system in the first place (which one would work well for the kernel? which one works on all the architectures people want to compile linux on? etc)

B from linus' perspective just means "wait a year for things to automatically get better (after throwing some money at it)" which seems like the low-effort solution.

Not sure why, but your URL has a bunch of junk on the end of it.

I don't get what you're arguing against. Even if the kernel were superfast, there's probably another "real load" out there somewhere that legitimately runs into lots of page faults.

It's a question of what should be addressed first.

Linus is actively arguing against a faster kernel build. It sounds like it makes sense, because it gives him leverage with CPU vendors.

But it doesn't, really. If kernel build were already fast, he would never in a million years slow it down just to get that leverage.

Now maybe he's perfectly aware of this status quo bias, and he's taking advantage of it to meliorate something he would otherwise have no power over. Sneaky.

Still, what's the priority? He's made his point now, hasn't he? He could work on making a faster build process, now.

Slowing it down wouldn't give him the same leverage that it has as a real use case.

Anyway he's not saying slowness is better in and of itself, just that there are better places to work on than replacing make.

Actually, his main point is "here is a workload that is kernel-bound that ordinary users deal with, and here is where the time goes." He has explained in the past that he works on whatever interests him, period. So profiling page faults must be more interesting to him than speeding up the build per se. That's all.

What?

"When a proposal to change a certain parameter is thought to have bad overall consequences, consider a change to the same parameter in the opposite direction. If this is also thought to have bad overall consequences..."

How is speeding up kernel build times a "bad overall consequence"? Thats exactly what Linus is trying to do.

Except Linus is not addressing that problem directly. If he was, he would probably be writing a make he'd be satisfied with. Instead, he's working on page faults.

Speeding up page faults does _directly_ improve build times. I don't understand why you think writing yet another make is the obvious direct fix, but fixing page faults is not. Or why you think Linus shouldn't fix things he is clearly interested in.

Reply on news.ycombinator.com