The new microcode from Intel and AMD adds three new features
lkml.org
lkml.org
https://lkml.org/lkml/2018/1/23/25
It looks like Ingo Molnar, Andi Kleen, and others are coming up with a pure-software alternative to IBRS, which could be much faster (rather similar in spirit to the retpoline trick)
I've always found it really frustrating that lkml.org seems to be the canonical web link that comes up for kernel mailing list discussions because of this, as I really find the threading there frustrating; it's nearly impossible to determine whether you've read the entire (sub-)thread.
Back when Gmane was alive I usually gravitated there as an alternative.
Suggestions (or working links to what you like better) are welcome!
In particular I'd like to be able to, starting with a particular post, repeatedly hit "next" and be assured that once I hit the next sibling post to the original I've read that entire subthread. Gmane let you do this just by whacking the space bar; if you weren't at the bottom of a post it would page down, if you were it would go to the next one.
In other words I'd like it to behave as much like a good NNTP newsreader as possible.
That's lost technology from the golden age of our ancestors, who were so much greater than any of us now ;-)
Today this innovation has moved to the myriad of proprietary alternatives (like Slack or our very own HN) but they are closed worlds so not only can nobody create a new client, we also loose all the contents if the company dies (which isn't rare).
--grumpy greybeard
there is plenty of active usenet groups, comp.arch being one of the best places where the greybeards meet.
I just tried again, still didn't show any children. Then I did a force-refresh. Now it shows them ...
So due to past terrible performance of lkml.org, it had aggressive long-term caching. So since I happened to see this message a couple days ago, it wasn't showing a newer version of the reply tree for me, I only found them in the top-messages list on the lkml.org main page.
Leverage CONFIG_DYNAMIC_FTRACE. This is an OS only fix resulting in effectively zero overhead for chips other than Skylake.
https://lkml.org/lkml/2018/1/23/25
There's another possible method to avoid deep stacks on Skylake, without
compiler support:
- Use the existing mcount based function tracing live patching machinery
(CONFIG_FUNCTION_TRACER=y) to install a _very_ fast and simple stack depth
tracking tracer which would issue a retpoline when stack depth crosses
boundaries of ~16 entries.
The overhead of that would _still_ very likely be much cheaper than a hundreds
(thousands) of cycle expensive MSR write at every kernel entry
(syscall entry, IRQ entry, etc.).
In this case, old processors simply does not predict the jump and stalls. But Skylake instead uses the generic indirect branch predictor, since after all RET is an indirect branch! Except the idea behind retpolines was to avoid the generic branch predictor, and now every single RET can use it! So, if you don't want to use IBRS you need to find a different way to protect the kernel from someone forcing undesired speculative execution after the RET.
Is AMD similar to Intel pre-Skylake, for the purpose of Spectre mitigation?
Could Intel and OEMs restart sales of Broadwell hardware, until they can reverse the Skylake/KabyLake/KabyLakeRefresh changes (in Cannon Lake? Ice Lake)?
This isn't a general fix or "pure-software alternative to IBRS" - it is specific to handling the 16 stack depth ret being vulnerable on skylake+ processors, as it falls back to an indirect branch prediction. The zero overhead for non-skylake+ chips isn't that it is a zero-overhead general IBRS solution, it's that it does not incur additional overhead on non-skylake+ chips as it is not doing any work to mitigate the issue on non-skylake chips.
This is because all of the major distros already compile their kernels with the CONFIG_FUNCTION_TRACER=y option set, so no additional work is being done unless the conditions appear and it is on a skylake+ chip.
That got me interested, who has reptoline support, and from when?
GCC added support in 7.2 [0], which means a lot of people don't have access yet.
LLVM added it in D41723 [1], but I don't think that has landed in a release yet.
I can't find any talk around TCC, but that isn't exactly surprising.
Haven't heard anything about MSVC, and can't find anything, but that's not surprising as I don't think Windows is using reptoline in their mitigations, but I really have no idea.
[0] https://www.phoronix.com/scan.php?page=news_item&px=Clear-Li...
Our tests show the performance impact of /Qspectre to be negligible. We have built all of Windows with /Qspectre enabled and did not notice any performance regressions of concern. Performance gains from speculative execution are lost where the mitigation is applied but the mitigation was needed in a relatively small number of instances across the large codebases that we recompiled.
From Terry's post:
> For context, on newer CPUs such as on Skylake and beyond, Intel has refined the instructions used to disable branch speculation to be more specific to indirect branches, reducing the overall performance penalty of the Spectre mitigation.
And your link:
> /d2guardspecload, that is currently equivalent to /Qspectre.
It seems equivalent to reptoline, at least from a high level perspective.
So MSVC is probably protected, but only optionally. They're leaving it up to developers to use the /Qspectre flag, it isn't something automatically enabled when you use an /O(x) optimise flag.
Edit: I should add that /Qspectre is currently being added to a lot of Windows, but they're assessing where before doing so, because not everywhere should need it.
No, GCC adds support in 7.3, which is scheduled to be released tomorrow.
A few distributions like Clear Linux and FreeBSD patched 7.2, but everyone else needs to wait til 7.3.
More details at the end here: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...
[Edit] I see, link implies the problem is that the 6 month disclosure process lacked urgency, and many affected parties were notified late in the cycle. Perhaps this is an argument against agreements to keep vulnerabilities secret for a long time, as an attitude of, "there's enough time to address this," can be counterproductive.
Also because of 5 days? Embargo was for the 9th of Jan. and it leaked on the 5th after 6 months! What are you going to do in 5 days that you didn't in the last 6 months.
Intel thought they could roll Spector and Meltdown into a single massive bug, that all CPUs were vulnerable to. Instead, AMD leaked the fact that AMD CPUs didn't need the KPTI patch, which meant the very first thing people knew was "it's an Intel bug". This very much put the Intel PR effort on a back foot.
Then Intel thought they could just release their microcode update (hell, they even coordinated with AMD to release the same microcode patches), and tell linux "here are the patches to use it, you must merge it". I assume the plan involved disclosing on the 9th and essentially bully Linus into merging the patches by the 10th.
They wanted people to thank them for spending millions to develop the new microcode and once-and-for-all solving this problem. They wanted the headlines "Intel releases microcode which fixes Spectre/Meltdown flaw" so that people would stop demanding intel replaces their broken silicon.
Then google releases details about their retpoline idea (I'm assuming it was developed not long before the embargo ended). It achieves the same thing, but with less performance penalty and no microcode update needed.
So now the Linux mailing list has something to debate. The Intel engineers are very motivated to get their patches merged, because they spent millions developing that microcode and they want the PR wins for "fixing the problem". Watch how the immediately segway into "oh, but retpolines don't work on skylake"
The Linux engineers are not happy with this, they are angry at Intel for being the source of the problem and probably see through Intel's little PR plan. They are triple checking intel's work and seeing if they can come up with an even better solution that works on skylake too.
The Intel engineer in that email seems to be bragging about it:
I think we also want to expose IBRS to VM guests, even if we don't use
it ourselves. Because Windows guests (and RHEL guests; yay!) do use it.> With Windows 10 on older silicon (2015-era PCs with Haswell or older CPU), some benchmarks show more significant slowdowns, and we expect that some users will notice a decrease in system performance.
> With Windows 8 and Windows 7 on older silicon (2015-era PCs with Haswell or older CPU), we expect most users to notice a decrease in system performance.
> Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance. This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment.
https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/...
EDIT: I just saw replies to walterbell asking the same question as me, so please ignore this one.
Intel engineers are a nice and cool bunch. It's not their fault.
I think a case can be made that as operating systems go, FreeBSD is at a much lower threat level than Linux, MacOS/iOS, Windows (i.e. desktops that run unvetted code) and the VPS platforms that run host operating systems that need to worry about guests breaking isolation (Google, Amazon, others).
Given all that, while it wasn't done perfectly (it was leaked a few days early and some cloud providers had little warning, especially Joyent which also runs a different OS), I'm not sure I would call it botched entirely since it is also likely the largest exploit in history. Others disagree with some or all of this.
EDIT: For reference, consider some of the comments in this thread: https://news.ycombinator.com/item?id=16110750&ref=hvper.com
https://www.itwire.com/security/81338-handling-of-cpu-bug-di...
In this case, however, it was handled horribly. During the embargo period, we kept telling Intel and AMD what they were doing wrong, and they wouldn't listen (or alternatively, they made clear enough that they wouldn't listen that we didn't even try). But really there's little more than I can do than hope that the next time they listen to us.
... for Linux distributions the actual embargo time was a little less than two months. That is actually a very small time to do the amount of work that was needed to mitigate Meltdown and Spectre. No Linux distribution was able to ship retpolines on the date the embargo was lifted (heck, only RHEL and SuSE shipped anything for Spectre at all), and the extra week would have bought us nothing. We would have needed to be notified a month or two earlier.
So you still need some solution there.
This works great for the common pattern of matched pairs, but the stack used to track outstanding calls has a limited size (32 in Skylake), let's call it N. If you have N + M calls followed by N + M rets (or any other pattern where the call chain gets that deep), you will predict correctly the first N rets, but then the stack is exhausted and the last M rets won't be predicted by the RSB.
Prior to Skylake, those last M rets just wouldn't get predicted at all (probably instruction fetch would just fall through to the instructions following the ret), but in Skylake the indirect branch-predictor, which is usually used to predict jmp or call instructions to variable locations, is used as a fallback instead.
So the concern is that people could train the indirect predictor prior to a kernel call, and if the 32-deep RSB was ever exhausted, the indirect predictor could kick in causing a Spectre-like vulnerability.
From whitepaper CSDB is 1101_0101_0000_0011_0010_0010_100_11111
Here some snippets from ARM manuals:
HINT instruction:
1101 0101 0000 0011 0010 0010 100 11111 CRm = 0010 op2 = 100
Some encodings described here are not allocated in this revision of the architecture, and behave as NOPs. (This is important)
Hints 18 to 23 variant Applies when CRm == 0010 && op2 != 00x. HINT #<imm>
Hint is encoded in CRm:op2 pair, existing similar:
0010:000 ESB // Error Synchronization Barrier 0010:001 PSB CSYNC // Profiling Synchronization Barrier
Thus in assembler this is written:
hint #0x14
Which is a NOP if SOC does not understand this hint. It's being used here: http://lkml.iu.edu/hypermail/linux/kernel/1801.0/04191.html
and also here: https://github.com/ARM-software/speculation-barrier/blob/mas... (which is being upstreamed to compilers in cross-platform generic form IIRC)
ARM whitepaper states that conditional selection/conditional move is enough on most ARM implementations. If it's not the case then the new CSDB solves the problem. On older CPUs it's still a NOP.
X-Gene disables branch prediction: http://lkml.iu.edu/hypermail/linux/kernel/1801.2/06482.html
ThunderX2 branch prediction hardening: https://patchwork.kernel.org/patch/10151975/
Whenever I read the mailing list, or threads related to it, I don't understand 99% of the stuff. Do I need to know C _very well_? Do I need to be familiar with kernel? If so, how do I go about doing that? At the moment I'm reading "Code: The Hidden Language of Computer Hardware and Software".
The book you’re reading is very good, your time probably isn’t wasted reading it.
[1] https://www.goodreads.com/book/show/166195.Modern_Operating_...
There probably isn't any good single resource for learning all this, but perhaps getting some good textbook on computer hardware architecture might be good next step after reading the "Code". Unfortunately I don't have any immediate suggestions on the current crop - I remember learning from William Stallings' books some 15-20 years back - not sure if they are the best choice nowadays.
[1] https://www.pearson.com/us/higher-education/program/Tanenbau...
My personal recommendation is to read the Linux Weekly News (https://lwn.net/) every week (if you don't want to subscribe, the previous week editions are available for free). It always has great explanatory articles about the Linux kernel internals, written by Linux kernel developers. For instance, this explanatory article about the "read-copy-update" mechanism, from the author of the Linux kernel implementation of that mechanism: "What is RCU, Fundamentally?" (https://lwn.net/Articles/262464/).
2. Language and Vocabulary: You need to be able to read and write (technical) English at a competent level. Virtually all technical documentation is in English, and most of the resources you could find on the web (like this Wiki and the forum) are, also. Using incorrect terminology will make you look foolish and confuse the people willing to help you.
3. Language and Vocabulary, pt. 2: Most operating systems featured on this site, as well as most of the code snippets and examples, are written in C (or C++). Even if you choose to use another language (like FreeBASIC or Pascal), C is the lingua franca of programming, and you should be competent in making heads and tails of it.
4. Assembly: You should have knowledge about the low-level language Assembly. Read a book. Take a course at school. Write some user-space code to familiarize yourself with it. You will need it, even if you plan to write most of your operating system in a higher-level language.
5. Programming experience: Learning about programming with an OS project is considered a bad idea. Not only should you know the language in which you will be developing inside out, you should also be familiar with version control, debugging, etc. - in short, you should have written quite a few user-space programs in that language successfully before trying OS development.
6. Programming Practices: You should know how to write both code and user documentation, and be prepared to carefully document all aspects of your code and design, even if the project is purely for your personal use. Also, you should learn and use a suitable Code Management practices, including setting up and using an off-site repository (e.g., Savannah, GitHub, GitLab, Heroku). These two practices alone will - not may, but will - save you a great deal of trouble in the future, if you follow them.
7. UNIX experience: You will soon notice that many of the tools used in OS development are developed for Unix, and later ported over to Windows. The Linux kernel is often used as reference or example for how things can be done, and many of the hobby operating systems have some resemblance to Unix. Having experience with the Unix command line (preferably Bash or ksh) is a very important requirement. (Cygwin provides an easy-to-install Unix command line for Windows.). If you haven't, go ahead and use Linux or a BSD for a while - for Windows users, this can be accomplished quite easily by using a virtualizer (see below) to host a virtual system, without the need to re-partition your home system. For macOS users, you can simply use the Terminal - macOS is built on a Unix variant kernel (a combination of the Mach and BSD kernels), so as long as you have Xcode and an emulator or virtualizer installed, no further tools are necessary (though you will need to set up a cross-compiler); the default Terminal shell is the Bourne-Again Shell, but C shell and KornShell are available as well.
8. Toolchain: You must know the behavioral details of your compiler, assembler, linker, and make utility. You should know what do the emitted warnings and errors mean. You should have the documentation of the tools you use at hand, and refer to them before asking the community. Rest assured that any possible beginner's question about GCC, GNU as, NASM, GNU ld, Visual Studio and GRUB has been answered twice over. Search before you ask.
9. Emulators and Virtualizers: Familiarity with tools such as Bochs, VirtualBox, QEMU, or Virtual PC is key to having a reasonable turn-around in development, and provide a buffer between your real hardware and your test system. While these can be learned for the specific purpose of OS dev, you will certainly want to be aware of what they are and where to get them before beginning an OS project.
10. Executable Formats: Kernel space programming has many additional requirements unknown to application development. Being able to parse executable binaries is one of them (you do want your OS to load and execute applications, do you?) Make yourself familiar with Executable File Types, their internal structure, and how a linker generates them.
11. The Platform: You should have studied the manuals for the processor you will be programming for. They contain the information you need to design your kernel in the first place. Asking for information that can easily be obtained from these documents will only give cause to "Read The (...) Manual" responses, or simply RTFM.
12. The Concept: You should have an understanding of how existent operating systems function at the technical level (e.g. by having read some Books), what is good or bad about them, and how you want to go about your own OS. Having finished some tutorial and then asking "what now?" in one of the forums will just make you look silly.
https://ocw.mit.edu/courses/electrical-engineering-and-compu...
After that, I don't know, read some deep dives into a modern architecture and look up any terms you don't understand.
He could be arguing for anywhere from, we need to replace all our hardware now, to we need to start over with how we design chips.
While I wouldn't want to compel people to run only verified binaries, I think it's important that we make it an option. Right now, it really isn't, especially with proprietary software.
Do you trust random JavaScript from the network? No? I didn't think so. So you trust RandomThirdParty verification?
Even banking code (ran on Secure Elements in credit cards) has levels of verification (by certified bodies) and trust (by banks and card processors). Where did you draw the line?
E.g. do all/some “certified bodies” accept financial liability if their verification is later found to be incomplete? Since Spectre/Meltdown were not widely known, what happens if previously-certified code was exploiting this vulnerability?
As far as verification goes, if authors use a GPG private key to sign their code, and distributors (package maintainers, etc.) do the same, they can be "verified" by the web of trust. That verification could also be hardened by a third party certificate body, although I'm personally wary of letting certificate authorities decide which code I can run on my computer.
The second (STIBP) protects a hyperthread sibling from following branch predictions which were learned on another sibling.
Is this about threads in the context of SMT?
I guess what I'm asking is "hyperthread" a thing outside of Intel marketing speak that warrants mentioning AMD?
When are you not? Why have security and process isolation if all processes are considered related?
You might conceivably have this in a corporate environment (I do).
For the vast majority of systems that do run untrusted code, including web browsers, this mitigation is probably appropriate.
Perhaps the word “might” here could be upgraded to “probably”?
*for some arbitrary value of “guarantee”
For this particular use-case, you might well consider that you don't need to patch for meltdown and spectre given the perceived risk/performance tradeoff for the current patches.
Unless your computer is air-gapped you can’t trust it to run only the code you expect.
There's usually a cost associated with greater security[0], and generally a balance that is struck. You can argue whether a particular system should lean further towards security, or further towards lower-cost, but ultimately there'll be a judgement call.
I'd suggest there isn't a single correct answer for this.
[0]whether in actual dollars, ease-of-use for users, time-to-market or possible reduction in feature set
In our case if the malicious code will run on those shared boxes, it implies either a bug in ssh or an Ubuntu software update bringing malicious code or an attack through a hacked computer of one of the users. In either case protecting other users from the effect of malicious code on those boxes will be the least of our worries. Yet the separated accounts do have substantial usability overhead.
>So now we mostly don't need IBRS. We build with retpoline, use IBPB on context switches/vmexit (which is in the first part of this patch series before IBRS is added), and we're safe. We even refactored the patch series to put retpoline first.
Unless I'm reading this wrong, it looks like retpoline is not enough, and you still need the microcode updates from Intel or AMD that offer IBPB.
If this is the case, why has Google been saying that retpoline is all we need?
vmexits and context switches are a very different thing than the deep stack ret calls.
> about: Visually Handicapped but can handle computer online work
Thanks!
https://www.computerworld.com/article/3250297/microsoft-wind...
Is this not the ultimate fate for any monolithic kernel? Because by design you need to add everything to the kernel.
While these patches might apply to micro kernels as well. I wonder if there is kernels that abstract the cpu into a device driver to even keep microcode bloat out of your kernel?
As a side thought; If you abstract your cpu like this; will it allow you to hot-swap your os between devices and architectures?
That could (and, I guess, would) mean a much, much, larger performance hit than a combined microcode/kernel fix would give. I doubt people would be happy about that.
Given that we don’t want to give up all speculated instruction execution, it will be difficult to find the spot where we get safety at minimal performance impact.
If devs do anything it should be learning to write software that doesn’t depend on speculative execution for performance
Now, maybe it is possible to reduce use of indirect branches . C++ for example, is pretty gratuitous in that respect. But in the end, most software is going to perform a lot slower without speculative execution. I don't think that's a price worth paying.
What will happen is the CPU vendors will fix meltdown (which should never have happened) and Spectre 2. Sharing the branch history buffer between contexts is something that fundamentally can be avoided, in the same way we can tag TLB entries with the PCID on recent hardware.
In the long run, the real issue is Spectre 1. Using software to contain untrusted software in the same address space is fundamentally hard with speculative execution. After all the fuss has died down, that's where we'll see innovative software techniques. It's already starting to happen, if you look at the webkit changes (scroll down to Branchless Security Checks):
https://webkit.org/blog/8048/what-spectre-and-meltdown-mean-...
No, what it matters is understanding the issue. And the issue has no "shenanigans". All vendors suffer from Spectre (and some non-Intel models from Meltdown too), but more importantly, none of these was done "on purpose" which is the definition of a shenanigan.
>If devs do anything it should be learning to write software that doesn’t depend on speculative execution for performance
Whatever.
The “shenanigans” are that it has become acceptable practice to cut corners on security to improve benchmarks.
Maybe speculative execution is just a thing that shouldn’t be done? People computed perfectly well before it. Itanium isn’t impacted by Meltdown so Intel is capable of it.
Nobody "cut corners". Those were unexpected bugs that took more than a decade for someone to even discover, not some deliberate decision to sacrifice security for performance.
>Maybe speculative execution is just a thing that shouldn’t be done?
No, it should 100% be done.
>People computed perfectly well before it.
Perfectly slowly too.
>Itanium isn’t impacted by Meltdown so Intel is capable of it.
Itanium tanked.
What do you call accessing memory before checking permissions then? AMD do the check!
Perfectly slowly too.
A simple predictor that assumes backwards branches will be taken (loops) and forward branches (most likely exceptions) will not is really all you need, if you insist.
Also, you severely underestimate the speedups gained by a better branch predictor. There's a nice writeup at https://danluu.com/branch-prediction/ if you're interested.
That is not just a convoluted scheme with no chance of effect, it is also morally reprehensible. You're willfully harming people for reasons almost entirely outside their control and expertise.
Imagine the coast guard letting people drown "because that's what happens if you buy a cheap boat". Except these people didn't buy cheap boats: Intel is quite obviously the default option, and usually more expensive in case where there actually are alternative.
And we haven't even gotten into the difficulty in making moral judgements about Intel's past behavior that led to these problems. Considering these bugs have existed for decades without being exposed, I would tend to think the pitchforks might just be an overreaction. There's a scale that goes intent to harm > accepting harm > accepting the possibility of harm (gross negligence) > not anticipating the risk of harm (negligence). One should calibrate one's outrage to it.
Oh please. If company X sells cars with defective brakes, we don’t change the entire country’s speed limit just in case - it is 100% on the manufacturer to make it right.
In what universe this is what the try to do?
That might be a second order effect of what they do (although even the idea that Intel's share price will crumble and people will go to another vendor for their next hardware refresh is bogus -- Intel is going nowhere, not to mention that AMD also has their issues with Spectre, and IIRC, part of Meltdown) but it's nowhere near the target, which obviously is to ensure that a billion PCs operating today continue to operate safe tomorrow -- and not just when people refresh their hardware (!!!).
The "not my problem - let someone else fix it" thinking is no different from the thinking that caused issues at Equifax.
Of course they should. Linux needs to work on hardware that exists not just on hardware that may, one day, exist.
> Don't further pollute the kernel.
One of the primary jobs of kernels, drivers and compilers is to abstract some of the underlying hardware complexity away. Kernels, drivers and compilers have always dealt with bugs and limitations in hardware.
Things like retpoline and CPU barriers aren't especially unusual. Emergency fixing such a pervasive issue is unusual though, hence the attention.
> ubiquity is not an excuse
No, but it's a really, really good reason
If I were inclined to facetiousness (which I'm not), I would point out that there is a large number of other kernels out there that aren't especially performant and don't work well on most hardware. If you really object to this, you can always use one of those.
what a ridiculous sentiment to defend a patch from Intel thats effectively disabled by default for one of the most egregious bugs in microcode since F00F.
"The new microcode from Intel and AMD adds three new features." No ones talking about AMD, David, not Linus and certainly not hackernews or other outlets. Spectre patching will happen regardless of this fiasco but This NOT SPECTRE. this is a taylor made shit sandwich thats been warned about by researchers and lurking on Intels plate for more than a decade. intel chose speed over security, and intel lost. Now they're trying to find a way --any way-- to avoid having to eat an 8 generation recall or worse.
its been said on HN before, but this is now more of an 'if not when' of intels loss of dominance in cloud. customers no longer care about the fastest chip in the west if its going to get neutered to half speed due to wreckless design that allows for a complete security disaster. Theyll buy twice as many AMD, at half the price of intel, and make up the difference in sheer volume.
"GPZ Variant 2 (Branch Target Injection or Spectre) is applicable to AMD processors."
As we sit here and watch how this dystopia future plays of around us and how corporations want to dodge responsibility.
It is derogatory. It generally means people who are inconsequential and ill informed, but they are going to loudly give you their opinion anyway as if you should care when you probably don't ("you" being someone whose opinion actually matters).
I can't figure out how to link to it, but this definition was my top search result:
a group of people who criticize someone, often by focusing on insignificant details