Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially.

The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast.

A lot of frameworks and abstractions will be ripped out and thrown away as 'taking too much time for not enough benefit.'

View on HN

In my experience, poor performance is only very occasionally the fault of abstractions and frameworks. The problem is near universally between the chair and the keyboard.

I don't think we'll be throwing away the old abstractions, but adding new ones in beside them.

As an example, I made a demo a couple years back which processed and rendered about 64 MB of CNC G Code (a programming language for telling robots how to move and spin) in a web browser to match CNC toolpaths (the path of the robots motion) to their realtime run statistics (show me a realtime 3d map of where we can speed up the robot). To do this, you need to simulate the movements of the machine to build a 3d model, and then pair the model to collected data. You then need to cull/average data which is too fine to be visible, and ship this processed data to the browser as arrays of floating point numbers, where it gets rendered as -effectively- thousands of colored 3d arcs.

The naive implementation took minutes to run. The final version took about 1 second. I didn't have to throw away any abstractions or frameworks to make things faster. I had to use my language's new abstractions for massively parallel processing and stack allocated data to speed up server side processing, and then use libraries for decoding base64 binary into floating point arrays on the browser side to speed up data transfer and parsing.

I guess my point is that the end of Moore's law doesn't mean throwing old things away, it means using existing solutions when appropriate, and approaching new problems in different ways.

>Why? Because it is going to demand engineers who can write fast code over engineers who write code fast.

Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code).

Most code by engineers "who write code fast" is not in performance critical domains...

I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

If my experience with the IT world has taught me anything, you'd be surprised for how long :-)

>But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

Why would they need to? They can sell improved cameras and other additions instead of improved CPUs...

I agree there's no need. What phone camera improvements do you envision that don't involve more transistors on the CMOS sensor, GPU, CPU etc? Microfilm?

A lot of the ways programmers write slow code ends up with multiplicative factors you can't practically recover with simply throwing more hardware at it, because it's really easy for us to write something 100x or 1000x slower (even in the same big-O class) by using a slow language with poor cache locality, that makes it easy to do more work than was necessary, by poorly using threading resources, etc. There's a lot of multiplicative factors that can stack together fairly quickly.

On the plus side, it turns out recovering most of that isn't that hard with a bit of careful language selection and just a bit of care in writing code. I think this is one of the somewhat subtle reasons for Go's success is that it trims away most of the slow things that a lot of "scripting" languages do without throwing away too much of the power, and I anticipate the continued entry and success of other languages like Nim into this space on the language chart in the next few years. To get to screaming fast may take a lot of effort, but "pretty fast, let's parallelize what we have instead of hyperoptimizing it" isn't too hard.

Because of all of these multiplicative slowdowns, even at very small scales it's still generally a better idea to start by optimizing and removing some of your multiplicative slowdowns before you just throw more hardware at it. Compute is merely cheap, not free. It's the rare bit of code that nobody's spent any time optimizing, but is also free of these multiplicative slowdown problems.

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%."

The end of Moore's law is the end of passing up opportunities in that critical 3%. There's a lot of performance debt accumulated.

It's only 3%. Optimize when it matters!

It's not just training AI models that's demanding exponentially increasing compute power, but also streaming, storing, and processing the enormous amounts of scientific data coming from (for example) high-end physics experiments at world-class laboratories like CERN or the LHC, and there's also the tremendous amounts of surveillance data and data processing that's sprung up in recent decades, with an insatiable appetite of various agencies and companies to track, record, and analyze everyone and everything.

True, but from what people were telling me before I left university, scientific processing always produced more data than could be analysed, even after accounting for the effort put into efficient algorithms.

Is there basis for thinking that the relative demand of fast-code programmers will increase vs code-fast programmers?

There doesn't seem to be anything in the horizon that would make performance a more pressing concern in the work of the 99% of developers who do make business software and consumer web/mobile apps. They just sometimes call into libraries that have had performance work put into them and get by with rudimentary understanding of what seems to cause slowdowns.

In the other dimension, there does seem to be a continued focus and resulting improvements in velocity.

The recent ML popularity is all about leveraging packaged optimized code (Tensorflow etc) with vast majority of programmers freed from the engine level details and detailed understanding of GPU perf work or parallel/distributed programming.

>Because it is going to demand engineers who can write fast code over engineers who write code fast.

I surely hope so, the waste mentality has to end.

Premature optimization will never become the right thing to do. Most code has absolutely no reason to be particularly fast because it doesn't get executed very frequently.

Nobody said that. Just that you should think about which algorithms you use, profile, and not waste so much. The wasted power at this point is significant.

That is sensible, but I've also heard people basically call any code that is not micro-optimized wasteful and harp on about how back in the day it was second nature for programmers to care about bytes and cycles in every line of code they wrote, and modern day programmers are just lazy and incompetent.

Yeah, I get where you're coming from. There's as always a sweet spot between the two extremes. But needing a new computer for running Word or Excel smoothly just seems like there's a lot of low hanging fruit left.

We will probably start to see this a lot in the video game consoles field. On the hardware front, looking at the early specs of the PS5 for instance show them emphasis faster storage access speed more so than added CPU/GPU/RAM.

Consoles were always more likely to see speed optimizations, but this had slid aside a bit in the last two decades.

I'm not sure that's a given, who's to say we won't be happy with the current performance we have?

There are still problems, like protein folding, or training larger and larger deep learning models, which have nowhere near the performance you'd actually want in a perfect world. There's always going to be a bigger target in some of these domains, and it would be a shame to imagine we'll reach a ceiling in some areas of scientific inquiry because of hardware limitations.

This is true, but imo most the gains outside of traditional CPU improvements will be either in specialized hardware or algorithms, I don't see much room for improvement in like optimizing code there.

Rust will fare well. Ruby won't.

Ruby isn't in the critical path, so that's a bad example.

In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.

> The constant re-invention of databases/operating systems/virtual machines at each layer.

Those are because of the human struggle, the political and economic fights. They're not because of purely technical problems.

Each layer is usually backed by a party with a very clear interest, usually control, and each layer on top tries to escape that control. It's been that way since the times of p-code.

> The constant re-invention of databases/operating systems/virtual machines

I wish that were the case but operating systems and systems software in general seems to be the most stagnant field. Everyone just buys the same FLOSS stack for $0 and compatibility is king, so there's very little research going in this field and much less of it ends up in any product you're ever going to see in use.

I would love to see a vibrant scene of competing operating systems and databases with fresh new ideas and research.

I meant that it is re-implemented at every layer.. Like a browser is practically an OS on its own.

I don't have a hat in the game but why won't Ruby do well or not well based on this? It's not like basic webapps need to be much more efficient, and no one was writing high performance compute in ruby anyways.

Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.

Sure. But an awful lot of folks don't have a hundred web servers. If you're running something simple, you can pretty much use whatever you want.

The vast majority of web apps are CRUD apps, which can achieve sub 100ms responses on any language with caching, background workers and minor optimizations.

There will always be problems where you need faster responses, and the solutions to those problems are already not using Ruby. You don't use a hammer when a screwdriver is appropriate, but nails work pretty well in most places and screws are unnecessary.

100ms with how many responses per second? Throughput matters too. And techempower does some pretty realistic-looking benchmarks. For example:

> In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly.

https://www.techempower.com/benchmarks/

> 100ms with how many responses per second

As always, it depends. Benchmarks are cool, and yes we know that Rust and Go will always be faster than Ruby out of the box. Once the number of responses goes up, you throw caching at the problem. But the benchmark says

> The requirements were clarified to specify caching or memoization of the output from JSON serialization is not permitted.

> the table has 12 rows

Which is low and should be quick regardless, but regardless of your tech stack if you are creating a "modern" UI, you handle this update asynchronously with JS anyway, and if you're not, do you really care about sub 100 ms responses?

If you want performance out of the box, yes you will use Go, but like I said earlier, you use a hammer for nails and a screwdriver for screws. If you want performance and type safety, you choose one, if you want faster prototyping and a more mature web dev ecosystem, you choose another.

FPGAs will fare well, general processors won't.

You know we've reach the end of Moore's law when balanced ternary computers become the norm.

That has a lot to do with why I took on learning Rust. My last year Macbook Pro has a 6-core CPU. At this rate, we should have 32-64-core CPUs for consumers in a few years. Not using all the processors means your program only makes use of 3-1.5% of the available power.

My prediction is that, in a few years, lots of programming languages and software developers will simply become obsolete because what the other guys can produce is heaps of performance, security and manageability atop.

But I might be wrong, so make your career decisions carefully.

I hope you're right.

However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores.

I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so.

Most commercial software just doesn't use it efficiently and consumers don't need it for existing software.

We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores.

For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there.

> However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores.

https://xkcd.com/605/

If you are going to extrapolate, don't pick the date. Take all the numbers you have and trace the average value. If you are picking the numbers, you can extrapolate to any point you want.

> I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so.

We already have 32-core processors for servers. Imo, the biggest challenge will be the laptop and heat management rather than putting a bunch of processors together.

> We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores.

We need that now. Most software I'm using is slow. Part of why I picked the iPhone (and I bet many people did) was speed. It is fast. People care about speed and if you introduce them to something faster then they'll get hooked. Also, you are already using all of your cores because of multi-tasking. It makes your computer experience overall better. At 6 cores, multi-threading for an application doesn't give much noticeable experience, since you probably have 5-6 things running on your computer doing other stuff like network or an audio-player.

> For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there.

> I hope you're right.

Don't worry, just hang in there. The start of multi-cores was slow but then that's all there is right now. Seems intel/amd have mastered how to add more cores but the CPU frequency is pretty much capped going forward!

Correct. As long as your bottleneck is the CPU.

Reply on news.ycombinator.com