hansendc

As mentioned elsewhere, Intel uses "Work Weeks" for everything. It might be great for manufacturing where a wafer takes a specific number of days to make it through the fab, but it inflicts horrible pain on the folks who have to deal with it and the outside world. My doctor, for instance, stubbornly pretends to not understand when I ask for an appointment in WW04!

December 30th, 2019 is WW01.1 2020 in this calendar. I bet no human in the world knows their WW 2020 birthday.

The good news: There's a new MSR which lets you do this:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

The less good news: as far as I can tell, Intel did not commit to how architectural this will be going forward. Considering the role TSX has played in speculation-based attacks, it appears to me to be a generic mitigation that would be great to accompany TSX wherever it is available in the future. Now that MSR_IA32_TSX_CTRL is defined, it should be easier to implement going forward.

Disclaimer: I work on Linux at Intel.

On Linux, most (maybe even all) of the data structures that must grow with memory size can be carved out of the memory being added. The main culprit is 'struct page' which typically needs 64b of metadata for each 4k physical page (on x86 at least). Much of the code to do this stuff is in Linux's "sparsemem" infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

This solution didn't introduce it. The concepts being used have existed in Linux for quite some time.

Here's a patch of mine that added (some of) it from 2005: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

mem= doesn't control the amount of RAM. It actually just specifies the largest physical address the kernel should try to use. The first 4GB of physical memory typically has some non-RAM things in it like the part of the physical address space that's reserved for talking to PCI devices.

If you have 2GB of RAM, 2GB of "other stuff" in the first 4GB, and specify mem=4096m, you'll end up with a kernel that sees 2GB of RAM.

I think you may be confusing some of the aspects of hot-add with hot-remove. Memory which you want to later remove should be onlined as "movable" and it has some content limitations. But, not all hot-added memory must be onlined that way. You can very easily just online it as "normal", which is (believe it or not) normal and can have any allocation type placed there.

If you think employing open-source maintainers gives a company "control" over a project, you haven't met many maintainers.

In any case, IBM is very respectful of the health of the open source projects on which it depends. While the vestiges of Red Hat may become less and less attractive as a place for a maintainer to work over time, I do not expect IBM to exert any control over any of these projects in any meaningful way any more than it has over the areas that its current employees maintain. Look at SCSI in the kernel, or even RCU, both maintained by IBMers (two examples among many).

(I worked at IBM in its Linux Technology Center for over a decade)

Ex-IBMer here...

I'd really suggest finding one of your colleagues that's seen IBM's track record with acquisitions and pick their brain a bit. They'll have much more relevant information for you than your old CEO.

It would have made more sense to call this on "RAMbo First Bleed". Next year it can be "RAMbo First Bleed Part II".

Disclaimer: I work on Linux at Intel.

FWIW, I don't think PR had anything to do with this. Without arguing the merits of embargoes, the goal of them is to give the good guys a head start before the bad guys. But, if you tell too many good guys, the assumption is that the bad guys find out. So, you try to tell as few good guys as possible.

This was (from my blatantly biased perspective) an honest, but imperfect attempt at keeping the bad guys from finding out. A lot of lessons have been learned since this, and the group of good guys involved in recent issues has been much more comprehensive than with the original Spectre/Meltdown bits. I think Greg was alluding to this when he said "Intel has gotten better at this."

Disclaimer: I work on Linux at Intel.

I'll give you an engineer's perspective, and specifically about Meltdown since that's what I worked on.

I posted the first Meltdown mitigations in late October, publicly claiming to mitigate a less dangerous class of side-channel (https://lkml.org/lkml/2017/10/31/884). I figured that Linus would merge it (Linus knew the real reason by that point) and would "wink" at Greg and the other stable maintainers enough to get them to merge it too. At this point, we had months before the embargo ended.

In retrospect, that was a kinda silly assumption, but I was heads down trying to get the patch to work and didn't dwell on it much. I wish I would have made more noise about it, especially as we got closer to the embargo ending.

Disclaimer: I work on Linux at Intel.

I'm biased, of course. But, I think things are much better now. We have ways of having relatively "normal" conversations about these things where a broad set of community folks are involved. It's not perfect, but it's way better than it was a year ago.

The embargo was really painful to work with, even for folks like myself who were inside Intel during the whole thing. I'm used to being able to ask folks for help and to lean on the incredible knowledge of the kernel community. For Meltdown, I was much more on my own than I was comfortable with. It wasn't fun.

Disclaimer: I work on Linux at Intel.

Future hardware will be mitigated against side-channel issues. There's a nice table showing how things are mitigated on future processors here:

https://www.tomshardware.com/news/intel-cascade-lake-details...

But, not everything is mitigated in the silicon or microcode. The mitigation for Spectre / Variant 1 / Bounds Check Bypass, for instance, will continue to be in software.

I worked on the Meltdown mitigations, at Intel, during the embargo. I still work on Linux at Intel.

Intel told quite a few members of the Linux community about this well before late December. Some got told because they work for traditional distributions and others were contacted directly because they are community maintainers or subject matter experts.

Unfortunately, Greg was not one of those folks that was told early. It's pretty clear at this point that it would have been a lot better had he and folks like him been involved earlier. This is especially true for Greg since he plays such a crucial role in stable kernel maintenance, which is how a lot of the world consumes their kernels (including distros like Debian).

I'm glad Greg thinks "Intel has gotten better at this." I'd like to think so too.

Response from Intel that includes a new version of the license is here:

https://twitter.com/imadsousou/status/1032680311753072640

Disclaimer: I work at Intel

That commit disables the lazy mode by default on processors supporting the XSAVEOPT instruction. But, it's possible to override that default with the eagerfpu= kernel command-line option, or that a hypervisor has masked out support for that instruction even on hardware with XSAVEOPT support.

The point is: although relatively unlikely, it is still _possible_ that you need some mitigation even if you have newer hardware (Sandybridge or newer is where XSAVEOPT first showed up, I believe).

Disclaimer: I work on Linux at Intel.

The processor has XSAVE (the mechanism that we use to save/restore FPU state and more these days) optimizations internal to the processor that keep it from having to fully reload the FPU state. OSes like Linux have not been doing lazy FPU switching on processors with these optimizations for a long time.

See information about XSAVEOPT and the "Init and Modified Optimizations" in the SDM: intel.com/sdm .

As @luto said above, recent versions of Linux ripped out the lazy handling entirely.

Disclaimer: I work on software (Linux) at Intel.

There are certainly non-JIT-related mitigations out there.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Disclaimer: I work on software (Linux) at Intel.

Spectre variant 1 is arguably a software problem. It's not about consuming "bad" data that was placed by an adversary like variant 2, or allowing a rogue data cache loads like with Meltdown. It happens as the result of normal execution.

The processor manuals are also very clear that speculation can be wrong and can have observable side-effects[1][2]. The processors also have ways to constrain speculation architecturally (like lfence) in the case that speculation can do something that results in exposure to a vulnerability. Those two things put together make it a software problem _now_, and perhaps forever.

This means that those of us who write security-sensitive software have a long road ahead of us to learn how this affects us and to continue to enhance our defenses. Adding array_index_nospec()'s in Linux, for instance.

1. Intel SDM Vol 3a, 4.10.2.3 Details of TLB Use: "The processor may cache translations required ... for accesses that are a result of speculative execution that would never actually occur in the executed code path." 2. CLFLUSH instruction definition says: "processors are free to speculatively fetch and cache data from system memory regions assigned a memory-type allowing for speculative reads"

I think you are confusing the TLB and processor data caches.

PCIDs exist for virtual addresses and are implemented in the TLB, not the processor caches. The processor caches are physically indexed, not virtually indexed.

So, does the processor cache have a mechanism to tell which "process" to which a given line belongs? Nope. Two processes (or a process and the kernel) that share memory can be in two PCIDs, but share the processor caches.

Sharing an address space is nice. Programs pass lots of pointers into the kernel (think read()/write()). For the kernel to get at that data, if the kernel and userspace share an address space you can just use the pointer. If userspace has a truly separate set of page tables, you have to do some relatively crazy (and slow) stuff to go and access the data from inside the kernel. It's much harder than just accessing a pointer.

We can still do this with KPTI because despite having two page tables, the kernel still continues to map all of userspace.

BTW, the bit flipping is pretty much

// Allocate 8k which is also 8k aligned: pgd = alloc_pages(PAGE_SIZE*2);

That allocates something which might go from (physical address) 0x12340000->12360000. The first top-level pagetable is at 0x12340000 (for the kernel) and the second is at 0x12341000 (for running userspace).

So, when you leave the kernel you just set a bit in your CR3 register (well, you do it in assembly, but here it is logically in C):

write_cr3(read_cr3() | 0x1000); // 0x12340000 -> 0x12341000

When you come back in to the kernel you clear the bit:

write_cr3(read_cr3() & ~0x1000); // 0x12341000 -> 0x12340000

I work on the Linux KPTI/KAISER code (and work at Intel).

The high-level description, along with the set of hardware things that get mapped into both kernel/app page tables are basically identical. Creating that list of things was painful, at least for me while I was getting KAISER working. If I were in their position I definitely would have leveraged the list of things that we found in the Linux work. I hope they were able to do that.

Some minor differences in the implementations: In Linux, we allocate the two top-level page tables next to each other so we can just flip a bit to switch, and in the OpenBSD commit they seem to be allocating them separately and then storing the two pointers in a per-cpu data structure. Both are fine ways to do it.

I don't see anything like the Linux "espfix" mechanism. Guess there isn't one.

The BTS/PEBS buffers seem missing. This hardware may just not be supported.

Yes, I was referring to the Linux kernel: it does barely uses recursion and has small stack sizes compared to normal applications.

Disclaimer: I put a big chunk of that paper together.

That definitely was not the intent! Getting retpoline to the place it is today took a ton of work from a ton of people, including the awesome folks at Google like Paul Turner, and countless people in the Linux community.

Disclaimer: I put a big chunk of that paper together.

You ask how it can be guaranteed with "general code". The first thing to remember is that retpoline is not for "general code". Linux, for instance, does not support arbitrary call depth and barely uses recursion.

Also, take a close look at the "Exploit Composition" section of the paper. Those five conditions are much harder to satisfy at 'RET' than they are during the demonstrated Spectre variant 2 exploit points. For instance, a long speculation window (#5) for 'RET' is interesting to generate since it means a stall while waiting on something to come off the stack.

Meltdown mitigations do not flush the TLB when changing from kernel to/from user when the PCID feature is in place. That's basically everywhere from Haswell on (for Linux and Windows), and in some places as early as Nehalem (>= 4.9 kernels on Linux, not Windows).

This arch/x86/events/intel/ds.c fix is unlikely to have rendered too many things unbootable. I missed ds.c entirely when doing the original implementation. I can't fault the nice folks at Canonical for mis-merging a tiny hunk like this. It really only affects pretty specific hardware anyway.