Hacker News Reader

Comment by alibert | Hacker News Reader

FYI, Red Hat (and CentOS) already have updated kernels with mitigation.

And that made me asking how Red Hat (and the free derivative CentOS) could have already patched their current kernel (on Jan 4th) while the patch was still being discussed on kernel ML the same day?

I don't see how they could come up with new kernel with mitigation for the three variants, excellent KB articles on the CVEs, tunables flag, preliminary performance test, etc. without having already patches implemented and tested for some time.

Do they have internal kernel devs? And so, do Redhat have another implementation?

(This is my first comment on HN, I hope I'm following adequately the guidelines)

View on HN

There's such a thing called: 'embargo period', wherein CVEs are (in most cases) responsibly disclosed to various parties---including several Linux distributions---much ahead than the general public, to coordinate security errata.

So you can be assured that Red Hat coordinates with upstream. It also has a serious "Upstream First" policy (with sensible exceptions, of course).

As to whether Red Hat has kernel developers, of course it does. :-) Much of Red Hat's popularly began as a Linux distribution. If you're curious, the LWN publishes statistics about who contributes to the Linux kernel, and the most recent one is for the 4.11 development cycle -- https://lwn.net/Articles/720336/

(Disclosure: /me is a Red Hatter.)

I really dislike the term "responsible disclosure" in this context. I might consider it responsible to notify a software vendor about a vulnerability in their software so they have a chance to issue a fix to their customers before the rest of the world finds out. But this situation is different. These vulnerabilities affect everyone, and only a special few were allowed to prepare in advance. That's just preferential disclosure.

The Register had some quite pointed remarks on the use of the word "responsible" by Intel in its press releases.

* http://www.theregister.co.uk/2018/01/04/intel_meltdown_spect...

* https://news.ycombinator.com/item?id=16043952

> These vulnerabilities affect everyone, and only a special few were allowed to prepare in advance. That's just preferential disclosure.

In this particular case, most of the fix had to be done in the operating system (the new microcode only enabled extra functionality needed by part of the operating system fixes), so it makes sense that operating system developers were allowed (and required) to prepare in advance. The three most relevant operating systems are Windows, OSX, and Linux; for Linux, one of the most important distributors is Red Hat. That gives two of the groups which were notified in advance: hardware (Intel, AMD, ARM) and operating systems (Microsoft, Apple, Red Hat, a few others).

Ubuntu and BSD still have no fixes. It indeed seems like a preferrential disclosure. Plus no 2nd-tier cloud providers like DO were notified.

It will be pretty unfortunate if it turns out that the projects that maintain a kernel (FreeBSD, various others) only received notification at Christmas, while various Linux distros (who have to deal with packaging, release, QA but not developing their own kernel patch since that comes from upstream) got a long warning period. It seems that way... Looking forward to reading about how this played out when the dust settles.

There was a post to OpenBSD- tech list telling no BSDs were told anything. And a blog post from Canonical says that the patch will be available 9th January (IIRC).

OpenBSD didn't respect the embargo a few months ago with the wifi issue, no surprise that they weren't told about this up front.

Yep, FreeBSD got notified a week before Christmas: https://www.freebsd.org/news/newsflash.html#event20180104:01

The Spectre mitigation is expected soon but there is no ETA on Meltdown yet: https://www.reddit.com/r/freebsd/comments/7och5a/freebsd_was...

As others say, OpenBSD indeed violated embargoes a few times.

There is basically no independent upstream (other than a handful of people). Essentially all the kernel developers work for the companies that have distros plus organizations like Intel, Qualcomm, etc. that do a lot of device enablement.

I agree in general. In this particular case, though, a good fraction of the work was done by me and tglx. I'm independent. Tglx is sort of independent.

This is not to diminish the work done upstream by less independent people. Dave Hansen, in particular, is the one who actually got the code to function.

Canonical said they were told in November.

The lay public, aka everyone, isn't considered to be experts in this domain to address the issues. Hence, they disclosed to top tier OS distributions and partners.

It's similar to the Recalls of Takanas airbags in this regard. The manufacturers are informed and they tell us the consumer to bring in our cars. I'm not qualified to replace an explosive airbag, even though I drive a car daily as a lay person.

There are other OSes and affected companies beyond the ones who were chosen to receive advance warning.

Imagine how it would work out in your airbag analogy if only Toyota and GM knew, and they withheld information for months while they worked on their own fixes, meanwhile the public remains in danger and other automakers have no chance to implement their own recall plans.

Can you mention which OS didn't get early enough warning? I'm sure there is some niche ones which may miss out but they are usually not the target of hacking either. And many of the derivative Linux distributions don't do active kernel development and just compile the stock kernel or derive from Red Hat, Ubuntu or others. The more people that are informed early, the more likely it will leak out early.

My particular interest is in OpenBSD, which was not included.

There is some drama over OpenBSD and KRACK that may have been why they didn't get told. But that is just pure speculation.

It's now called coordinated disclosure, for reasons.

And, correct me if I'm wrong, but apparently calling Linus Torvalds and Greg KH the general public (as far as Spectre and its (poor for now) mitigations are concerned) is the new norm. I'm not sure of what the community will think of that. And of the corporate actors involved in that curious choice.

Again, I am pretty sure that they were disclosed the details. This however is very different from being able to commit patches on the day of the disclosure. For one, neither Linus nor Greg are actually writing the code.

The problem here was that discussion was confined within company barriers (with small occasional exceptions) until last Wednesday. Linux thrives because it is based on reaching consensus among all stakeholders, and not being able to discuss the approach to take with respect to Spectre directly resulted in chaos.

I am very happy with the work that we at Red Hat did on this project (though of course not perfect in any way!) and I am proud that our articles are published openly and are being widely shared among the Linux user community at large. However, I certainly would have preferred to work together with all other vendors on a common solution before. But unfortunately, even if (as is the case with Red Hat) upper management is extremely supportive of upstream collaboration, you do what you can.

I should have used the term 'coordinated disclosure', which probably reflects reality more. The links (with remarks from tptacek) pointed out by JdeBP (thanks) up-thread are a useful read.

Kernel ml is public - embargoed/sensitive issues are typically disclosed on closed lists - and discussed internally. RedHat does have many devs that work on the kernel, as does other distros.

Typically, a sensitive issue will be disclosed privately, and devs for various groups will share progress privately. Then ideally, all distros will have patches and documentation ready on the agreed date of public disclosure.

See:

http://oss-security.openwall.org/wiki/mailing-lists/distros

The bug was disclosed to various organizations in early november so work was going for some time. And yes, Red Hat makes significant contributions to the Linux kernel. Red Hat maintains their own version of the kernel and integrating those changes back to the mainstream version can take time to reach consensus.

It's a good first comment. Generally: as long as you're not being deliberately mean or derisive, you're doing fine by the guidelines. Welcome!

I finally found the changelog for the patch that Red Hat applied to their kernel: https://git.centos.org/patch/rpms!kernel.git/6202b1a49906b21... (scroll to mid)

I think we can say that Red Hat does have its own implementation for now.

edit : formatting

Reply on news.ycombinator.com