Hacker News Reader

Comment by cobookman | Hacker News Reader

If helpful a few Cloud Providers Responses:

Google Cloud

- Google Cloud's protections against this new vulnerability: - https://cloud.google.com/blog/products/gcp/protecting-agains...)

- GCE Related information: - https://cloud.google.com/compute/docs/security-bulletins

- GKE Related information: - https://cloud.google.com/kubernetes-engine/docs/security-bul...

Oracle Cloud

- https://blogs.oracle.com/oraclesecurity/intel-l1tf

Azure

- https://blogs.technet.microsoft.com/virtualization/2018/08/1...

View on HN

For AWS: https://aws.amazon.com/security/security-bulletins/AWS-2018-...

(Disclaimer: I work at AWS, but I am not linking this in any sort of official capacity. I don't know any more details beyond what is listed in that bulletin, and can't answer any questions related to this, unfortunately.)

Contrast:

"Meanwhile, we suggest using the stronger security and isolation properties of EC2 instances to separate any untrusted workloads."

with:

"Google Compute Engine employs host isolation features which ensure that an individual core is never concurrently shared between distinct virtual machines. This isolation also ensures that, in the case that different virtual machines are scheduled sequentially, the L1 data cache is completely flushed to ensure that no vulnerable state remains."

The former does not inspire confidence. Given the hypervisor on EC2 is opaque to me, I'm not sure how I'm supposed to avoid co-tenanting in a risky fashion.

How is the latter possible in case of 1 vCPU VM instances if 1 vCPU == 1 hyperthread? Do they give you a full core in this case?

Or map pairs of hyperthreads to each requested vCPU, or disable hyperthreading. Which kills co-tenancy overcommit and makes the infra more expensive for a cloud provider.

1 vCPU instances do not simultaneously share cores with other customer instances via SMT (Intel Hyper-Threading Technology). A core can be sequentially time sliced between customer instances when you use a fractional (m3.medium) or burst (T2 instances) CPU instance type.

https://aws.amazon.com/ec2/instance-types/ includes a note that provides some additional information about when vCPUs utilize Intel HT Technology: Each vCPU is a hyperthread of an Intel Xeon core except for T2 and m3.medium. (emphasis added)

https://aws.amazon.com/ec2/virtualcores/ provides a table of the number of cores allocated per instance. Notice that an instance like m5.large that has 2 vCPUs and 1 core. An instance like t2.small has 1 vCPU and 1 core.

Disclosure: I work for AWS

Um, no. If I schedule a micro instance in a particular zone and nothing else, for the duration of my timeslice the VM will have to monopolize the core. Optimal use of vCPUs would demand that two VM threads get scheduled to the core to take advantage of HT, which Google says it won’t do. Timeslicing doesn’t solve this problem. At least that’s my reading of what they said.

Re-read what I wrote above. It covered both simultaneous and sequential sharing. Here is the part you are concerned about: instances do not simultaneously share cores with other customer instances via SMT.

Re-read what I wrote. Imagine a machine with 8 cores and 16 hyperthreads. Now imagine you need to optimally use the execution ports in each core. To do that you would need to have 16 VM threads all feeding the ports simultaneously, 2 per core. If all your workloads on that machine are from different customers, you can’t do more than 8. If you timeslice more than 8, then the question is, what is it that you’re selling as a vCPU? If it’s a timesliced core, then it’s not what they say they’re selling (a hyperthread).

You:

> If it’s a timesliced core, then it’s not what they say they’re selling (a hyperthread).

vs.

_msw_:

> A core can be sequentially time sliced between customer instances when you use a fractional (m3.medium) or burst (T2 instances) CPU instance type.

AWS instance types page ( https://aws.amazon.com/ec2/instance-types/ ):

> Each vCPU is a hyperthread of an Intel Xeon core except for T2 and m3.medium.

Not in Google’s case. There vCPU == hyperthread. To quote https://cloud.google.com/compute/docs/machine-types: "For the n1 series of machine types, a vCPU is implemented as a single hardware hyper-thread". One more area where Google is more explicit. Only their partial vCPU instances are timesliced: "Shared-core machine types provide one vCPU that is allowed to run for a portion of the time on a single hardware hyper-thread on the host CPU running your instance".

Except this thread has been about AWS, not Google.

Who cares how Google defines things?

It was about both.

timeslicing works fine, you just have to flush the cache when switching

I’m not disputing that. I’m just saying that for the specific case of jobs with 1 vCPU which are all from different users you can only schedule half the billable capacity.

> Optimal use of vCPUs would demand that two VM threads get scheduled to the core to take advantage of HT

It would.

So those cores won't be used optimally. No big deal.

Yeah, and the throughput of HT threads on an already busy core was not that high, last I checked.

It does say, "All EC2 host infrastructure has been updated with these new protections, and no customer action is required at the infrastructure level."

"Meanwhile, we suggest using the stronger security and isolation properties of EC2 instances to separate any untrusted workloads." As I read it, it is talking about running code within your instance - if you have untrusted workloads, rather run them in a separate instance, so as not to encounter issues like this cross-process.

Amazon offers "dedicated hosts" and "dedicated instances", which you can specify in the "tenancy" field when launching an EC2 instance. I bet that's the stronger isolation that they're referring to. Costs more, though.

https://aws.amazon.com/ec2/dedicated-hosts/

This guidance is primarily for customers that are running untrusted code within their EC2 instances.

The "strong isolation of EC2 instances" refers to the properties of isolation provided by EC2's virtualization compared to processes within an operating system. It is challenging to safely and securely run untrusted code within sandboxes and processes using general purpose software. However, the hypervisor and hardware based virtualization of EC2 instances is engineered to provide isolation between mutually untrusted instances.

There are several reasons that customers may want to use dedicated instances or dedicated hosts, so we provide those tenancy options as well. The most common reason customers use dedicated hosts so that they can bring their own software licenses, which are often tied to a physical host.

Disclosure: I work for AWS.

> Given the hypervisor on EC2 is opaque to me, I'm not sure how I'm supposed to avoid co-tenanting in a risky fashion.

In Linux, at least, the Xen hypervisor on EC2 exposes some information about itself at /sys/hypervisor. In particular, I think /sys/hypervisor/uuid would allow you to detect co-tenanting (between two VMs of your own).

Not saying that I think you should do that, or that'd I'd want to — it'd be a PITA to coordinate amongst VMs, and I'm not sure it would matter (what if you're co-tentant w/ a malicious VM? even if you detect it, how do you get out of it?). That is, inside the VM seems like wholly the wrong place to attempt to deal with this. But I don't think many people realize /sys/hypervisor exists.

Google writes that ‘Google Compute Engine employs host isolation features which ensure that an individual core is never concurrently shared between distinct virtual machines’.

However, GCE does offer shared core machine types (f1-micro and g1-small) with 0.2 and 0.5 vCPUs respectively. This seems to contradict their statement (unless the cores are not shared after all, but that doesn’t make sense from an economical standpoint).

Also, they offer machines with one vCPU, but since a vCPU is only a single hyper-thread and not a full core, this still allows for the core to be shared over multiple VMs. If this means that Google will stop using hyperthreading and instead give everyone a full CPU core per vCPU, that will likely give noticeable performance benefits (but cost more for them).

I think the key word is 'concurrently'. My guess is that micro and small instances indeed share CPUs, just never 'at the same time' and L1 is flushed before transition from one VM to another.

I work for Google Cloud, but not related to security or OS development, so am not aware how it is actually being done.

seems slow. Having to flush on every swap, I mean. I wonder if there wont be a move away from offering these kinds of offers: if offering 0.2 of a core means putting in 0.3 effort, that's a 20-30% drop in how many you can run per core.

512 entries, 12 cycles to refill each entry from L2, at 3GHz that's only 2 microseconds worst case to refill the L1 cache. And that's if you keep the same task in it. If you're switching tasks most of it is going to get evicted anyway.

GCE uses KVM, which defaults to the linux scheduler with time slices from 0.75ms to 6ms, so the extra impact should be negligible. It's possible they tuned it weirdly, but I can't think of any reason to do so.

Flushes that occur from hypervisor calls could possibly have an impact, but those will happen whether you share a CPU or not.

> and L1 is flushed before transition from one VM to another.

Is this true?

it's what the Google blog post was indicating

I believe that this is likely true as they seemed to implement this for the TLbleed flaw.

The key word is "Concurrently".

They can give both hyperthreads to one machine for 50ms then both hyperthreads to another machine for 50ms.

For VMs with less than 2 vCPUs I suspect they give them two virtual threads and just schedule them for half the time they would otherwise.

Reply on news.ycombinator.com