fluffything

Heavily depends on the application.

Many applications running on a Top 10 system of the Top 500 are required to show good single threaded performance and good weak and strong scalings up to a big part of the system.

Checking all these 3 check boxes is hard to do if your communication is not fully overlapped by computation. The smaller your problem, the better your single threaded performance (e.g. at some point your problem fits in L3), and the larger fraction of execution time that communication dominates.

Even the poorest applications on these systems use non-blocking MPI to overlap communication with computation nowadays.

Yeah.

FWIW I think it would be rare for somebody to build that kind of node. Applications that fully utilize two CPU sockets while simultaneously maxing 10x V100s are super rare. Usually for GPGPU applications the CPU is mostly idle, and only "orchestrates" the GPU usage.

If one needs to support both GPGPU-only and CPU-only applications in a cluster, one would usually just build two separate partitions. One with e.g. 4-8 V100s per node, but way weaker and cheaper CPUs, and the other one with 2 beefy CPU sockets but no GPUs.

Also, if you are going for that kind of interconnects, you are probably building a system with >100 nodes. For your price, that's 160 million dollars, which is on the ballpark of what these systems costs. You usually get a cheaper price per node at that scale (although not much cheaper), but there are many costs (installation, maintenance, training, etc.) that are not covered there. In particular, those nodes will be running quite hot (just add the watts that those V100s and the CPUs use!), so the cooling infrastructure required to held those in a relatively small space can also end up costing quite a bit. You don't want a water pipe breaking and destroying a rack containing 10 of these nodes costing 1.6 million dollars, right?

Apple created Metal waaay before Vulkan existed and moved their whole ecosystem to it before Vulkan was production ready.

While moving from OpenGL to Metal was a significant technical upgrade for them, moving all their ecosystem from Metal to Vulkan is just a lot of work for very little benefit.

But yeah, sure, maybe they are just sticking with Metal to mess around with people.

Finally! A comedian!

Perfectly setting up the classical joke that banks are too poor to find COBOL programmers because there is a "shortage" ;) ;)

You, sir, are a true connoisseur and made my day! Thank you!

Good question. Of course you can make the same argument about bandwidth, the question is whether you want to make such an argument.

If you are building a system whose purpose is to transfer data (e.g. a router), every second that system is "computing" and not "transfering" you are paying for Watts that aren't doing anything useful.

A supercomputer purpose is to compute stuff, not to move data around, so making that argument about a supercomputer doesn't make much sense because nobody cares whether the supercomputer is using all its bandwidth or not as long as its using all of its compute capabilities.

So that's like making the argument that everything that isn't a plane, well, isn't a plane. Sure, my guitar isn't a plane, but it is not the purpose of my guitar to be one :D

It's not a dumb question, that's just a problem that most people don't have.

If you have hundreds of 80.000$ nodes with 8 V100 each consuming thousands of Watts per node, you start thinking about how much money is the 90% idle time of those nodes, waiting for data from the network, is costing you (probably millions of dollars per year).

So yeah, a network that's 10x faster is worth every penny if it can turn the system from a 90% idle to <10% idle.

That's why people always pay extra for NVLink, HDR Infiniband or HPC Ethernet interconnects, and also why Nvidia bought Mellanox last month...

Essentially, when your machine has 1 ExaFLOP/s of compute performance, every second that it remains idle waiting for the network you... well lose 1 ExaFLOP. That's just a lot.

Also, imagine millions of cores writing to the same harddrive simultaneously - hell try doing `ls` on a normal linux machine on a folder with 10 million empty files. The "storage" infrastructure to support instantaneous I/O on these systems is quite complex as well. Every second that the system is writing to disk you are wasting another ExaFLOP of compute.

> However musl has the additional constraint of being compatible with small/very-low-memory environments.

How many threads do these have ?

If they only have one thread, they'll use 72x less memory than if they would have 72 threads.

The thing is that if you are using 72 threads you probably would like your application to be 72x faster than if you are using only one. So synchronizing all allocations and killing scalability doesn't solve these users problems.

Most allocators, including jemalloc, tcmalloc, mimalloc, etc. have a "hardened" mode, that people can opt into if they want.

If I'm using Rust like the user in the blog post, double frees are caught at compile-time, so I'd rather not pay for them at run-time.

> Warps of threads within a thread block are not guaranteed to run in any order, and there are no guarantees that, e.g., if warp A spins on a lock hold by warp B, that warp B will ever run and make process, and therefore the thread block might never run to completion.

FYI this is wrong, pre-Volta guarantees that all warps will run to completion, what it doesn't guarantee is the same for threads within a warp. post-Volta guarantees that.

Yeah, you should be upvoted.

The author put zero effort into figuring things out.

>Data races don't imply memory unsafety (in the sense of having OOB accesses etc.).

Data races are undefined behavior on all major programming languages (C, C++, Rust, Java, Go, Swift, ...) and all major compilers compile code under the assumption that they do not happen.

If your program has a data-race, neither the language specification nor the compiler make any guarantees about the behavior of such programs.

Needles to say, this is neither safe, nor desirable, nor doing so produces perfectly safe programs.

The idea that data-races are safe because the load and store memory instructions that the CPU has are atomic is completely flawed, because it ignores the fact that the CPU does not execute the program you wrote, but the machine code that the compiler generates, and the compiler can and will reorganize non-atomic loads and stores under the assumption that they don't need to be atomic.

If your program has a data-race, the compiler provides no guarantees. It does not guarantee that the binary will reach the location where the data race happens. It does not even guarantee that the binary produced will be an executable binary, or that it will produce a binary at all (e.g. it does not guarantee that compilation terminates successfully).

> Memory-mapped I/O

If you are using volatile to interact with memory-mapped I/O, that means that there is a _thread of execution_ somewhere also interacting with that memory - otherwise why are you reading/writing to it? - such that if those reads and writes are not synchronized the program has a data-race and the behavior is undefined.

The claim that this is not a data-race because those threads of execution are "outside my process" is incorrect: the C standard doesn't know about processes. Whether one thread runs in a CPU and writes to a memory mapped DMA region, and the other thread is in your network card reading from that same memory region doesn't matter at all.

The right tool here is a `volatile atomic` load / store.

> - Values changed on other threads on x86 if you just don’t give a damn

If by "don't give a damn" you mean "don't care about UB" then this is correct, because doing that is UB, and UB can do anything.

> Values that are changed in interrupts

From the C standard point-of-view, an interrupt handler suspends a thread of execution, and the handler runs in a different thread of execution. Whether these threads are serialized by the OS or not doesn't matter, reading/writing across threads without synchronization is UB. In particular for interrupt handlers, using volatile to modify memory from an interrupt handler doesn't imply that the thread reading that memory will see the modification (e.g. the compiler might have cached that in a register). Using volatile to read the memory isn't sufficient either, because volatile is not a compiler fence, and it doesn't synchronize-with anything, not even volatile. The right tool here is a `volatile atomic` load / store.

> - longjmp/setjmp

The C standard requires the usage of volatile here, otherwise the behavior is undefined. There is only one thread of execution at play here, so no synchronization is necessary.

Similar control-flow constructs like throwing and catching exceptions do not require the use of volatile here, so the reason one must use volatile here is because longjmp/setjmp is a poorly designed language feature.

Yep, same thoughts.

Most of the open-source contributors in my projects that get hit by a bus, get hit by an Apple's bus (they start working at Apple, and cannot say anything publicly anymore or contribute to any project).

> However, I know even that property is not true of all GPUs and thus cannot be assumed by Vulkan spec compliant code. (I actually doubt it's true of any GPU's pre-Volta - it's just that the progress needed is often a natural side-effect of switching between threads for latency hiding).

To be more precise. Pre-Volta GPU (like Pascal, Kepler, etc.) guarantee that once a thread block starts running on an SM it will run there to completion, i.e., its resources won't be freed until the kernel completes.

Warps of threads within a thread block are not guaranteed to run in any order, and there are no guarantees that, e.g., if warp A spins on a lock hold by warp B, that warp B will ever run and make process, and therefore the thread block might never run to completion.

Volta and later architectures guarantees forward progress in this case. That is, if a warp A spin locks on a lock held by warp B, Volta and later guarantee that warp B will make progress at some point, allowing it to release the lock so that warp A can make progress as well.

> There are threads on the LLVM mailing list that discuss this in some detail.

Do you have a link ?

How do people WFH with their new $$$ macbooks ? Is there any external cam you would recommend ?

> So pretty much as ever, if you're round-tripping to the US, picking up an MBP there is nice.

Just keep in mind that if you stay in the US for less than 6 months, you should be paying VAT on it on your way back (see other comments). Otherwise this probably counts as tax evasion, which comes with pretty hefty fines in some EU countries.

The problem is that it is too easy to accidentally touch. Even if you set it to the Fn keys, a slightless faint of electrical conductivity between a side of your finger and the touchbar while you are pressing a number suffices for a key press. With a real key, you actually have to properly press the key. That might happen by accident, but is definetely harder.

Thanks for the correction, 2399$ was the price of the comparable Dell modell. The VAT and AppleCare makes sense.

The iPad Pro is heavier than the 2020 Macbook air, so.. yeah...

On a 2013 Macbook air, MagSafe has worked perfectly for me for the last 7 years. Saved my laptop a couple of times too. Way more worth it than AppleCare.

What about the quality of the webcam?

I need to do a lot of video conferencing nowadays, and the macbook air 2020 webcam is worse than the rear facing camera of a Nokia 3210.

As others have mentioned, get a silicone keyboard cover, cut the touch bar part of it, and use a hot glue gun to glue it over the touchbar.

If you happen to buy a 2019 macbook pro, then don't worry with that, just type on the silicon keyboard cover. It will be bad, but not as bad as using the actual keyboard directly.

I have one sitting on a closet. It sucks too.

I am honestly astonished at how bad hardware companies are at incrementally improving their products. Every new "improvement" (new CPU) comes with a huge completely-unrelated drawback for no reason (horrible keyboard, touchbar, .......)

My GF has a 2019 macbook pro and is using a silicon keyboard protector to avoid the touchbar as well.

I want to upgrade my macbook, but I've laughed so often at its 30$ fix for its 2000$ laptop that I just can't update to a macbook with touchbar with a straight face anymore.

How long does one need to stay in the US for that not to hold ?

I lived there for one year, and bought a lot of stuff in the US for personal use (like clothes). When I moved back to Europe, I just moved everything with me without paying any VAT.

I was worried that they would make me pay VAT from socks and underwear to laptop, phone, and everything. Asked about it, and was told that I didn't had to pay anything.

Unified memory is useful during development when porting the application, but I don't know of any well-tuned applications that use it.

American layout actually adds value to the laptop in Europe.

You can change the layout in the apple store when buying it here, but that adds 4 weeks to the delivery time..

Ehm, customs? VAT import taxes?

As an individual, I can just by it in the US for personal use, and carry it over. But if you intent to transporting them for selling them, the story is very different.

~~2399$~~ 2999$ (EDIT: see replies) in the US, and.... 3924$ for the same model if you buy it in Europe (3629 EUR). That's a bit outrageous.

I still have a Macbook Air 2013, and have been looking for an upgrade for like 5 years, but I was not willing to pay 4k for a laptop with a worse keyboard, a worse power plug, an OS that in 2020 still doesn't have any kind of windows tiling capabilities and quite poor virtual desktops support, a useless touchbar, no wifi6, and the worst webcam ever made.

Selling it at twice the price that in the US doesn't help. I can pay for a plane ticket, and still get it cheaper there.

Also, for a company that markets their laptops as having great displays, its quite ironic that their OS still doesn't support any kind of window tiling to use them efficiently in 2020. The display isn't micro-LED either, so it is instantaneously outdated, and well... the retina display doesn't help you see your macbook-using coworkers any better because their webcams have a resolution of 1x1 pixels.

> and I'm sorry but a project being 'open source' doesn't mean that everyone can or should contribute, not all contributions are good or helpful and if you have a certain standard in mind for your software you have to filter contributions.

So if someone sends you a contribution that isn't good enough, you reject them with a "no", and that's it?

Sure, you can do that. A different project might expend some time teaching that contributor and being nice to them, until their PR can be merged. And then the contributor submits another one that's better, and then another one, and then before you realize it you have another full time maintainer that you can trust.

So when you say that if a contribution isn't good enough you reject it I say "Thank you", because that person will move on to a different open source project, which might be one of mine, and I'll be getting another contributor instead.

You might not think that the work required to attract a new full-time contributor is worth it. To me it is.