Question about MDS mitigation

Question about MDS mitigation

[wencongyang (A)]

Hi all

Fill buffers, load ports are shared between threads on the same physical core.
We need to run more than one vm on the same physical core.
Is there any complete mitigation for environments utilizing SMT?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Question about MDS mitigation

[wencongyang (A)]

Hi all

Fill buffers, load ports are shared between threads on the same physical core.
We need to run more than one vm on the same physical core.
Is there any complete mitigation for environments utilizing SMT?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 03:46, wencongyang (A) wrote:
> Hi all
>
> Fill buffers, load ports are shared between threads on the same physical core.
> We need to run more than one vm on the same physical core.
> Is there any complete mitigation for environments utilizing SMT?

No - not really.

An approach which was worked on was that of synchronised scheduling,
whereby privilege transitions are syncrhonised to ensure that we're
never running code from different privilege levels concurrently on
adjacent threads.  (This is the mitigation described as Group Scheduling
in
https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
)

However...

First of all, it depends on core scheduling as a prerequisite, and as
you can see, that is still a work-in-progress for Xen.  There are many
other good reasons for doing core scheduling, so we will be continuing
with that work.

For partners who already had core scheduling and experimented with
synchronised scheduling, no-one found a way of making it function with
less overhead than disabling hyperthreading in the first place.  This
was was a native OS case - the virtualised case gets a compound
performance hit because every time the guest kernel tries to
synchronise, it forces Xen to synchronise, because there is no
virtualisation of IPIs available in affected hardware.


Overall, it looks like the one mitigation option which would allow
hyperthreading to remain active had a larger performance penalty than
disabling hyperthreading (on native, and remember that it is compounded
when virtualised), and is a very invasive change to the entry/exit code.

Unless someone finds a really clever alternative plan, there doesn't
appear to be a viable alternative to turning off hyperthreading when
cross-thread leakage is a concern.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 03:46, wencongyang (A) wrote:
> Hi all
>
> Fill buffers, load ports are shared between threads on the same physical core.
> We need to run more than one vm on the same physical core.
> Is there any complete mitigation for environments utilizing SMT?

No - not really.

An approach which was worked on was that of synchronised scheduling,
whereby privilege transitions are syncrhonised to ensure that we're
never running code from different privilege levels concurrently on
adjacent threads.  (This is the mitigation described as Group Scheduling
in
https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
)

However...

First of all, it depends on core scheduling as a prerequisite, and as
you can see, that is still a work-in-progress for Xen.  There are many
other good reasons for doing core scheduling, so we will be continuing
with that work.

For partners who already had core scheduling and experimented with
synchronised scheduling, no-one found a way of making it function with
less overhead than disabling hyperthreading in the first place.  This
was was a native OS case - the virtualised case gets a compound
performance hit because every time the guest kernel tries to
synchronise, it forces Xen to synchronise, because there is no
virtualisation of IPIs available in affected hardware.


Overall, it looks like the one mitigation option which would allow
hyperthreading to remain active had a larger performance penalty than
disabling hyperthreading (on native, and remember that it is compounded
when virtualised), and is a very invasive change to the entry/exit code.

Unless someone finds a really clever alternative plan, there doesn't
appear to be a viable alternative to turning off hyperthreading when
cross-thread leakage is a concern.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:38, Andrew Cooper wrote:
> On 16/05/2019 03:46, wencongyang (A) wrote:
>> Hi all
>>
>> Fill buffers, load ports are shared between threads on the same physical core.
>> We need to run more than one vm on the same physical core.
>> Is there any complete mitigation for environments utilizing SMT?
> 
> No - not really.
> 
> An approach which was worked on was that of synchronised scheduling,
> whereby privilege transitions are syncrhonised to ensure that we're
> never running code from different privilege levels concurrently on
> adjacent threads.  (This is the mitigation described as Group Scheduling
> in
> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
> )

synchronised scheduling is not a complete mitigation. Guest A and Guest B
run on the same physical core, and the privilege level is the same. So
Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
because they are in different privilege levels.

Thansk
Wen Congyang

> 
> However...
> 
> First of all, it depends on core scheduling as a prerequisite, and as
> you can see, that is still a work-in-progress for Xen.  There are many
> other good reasons for doing core scheduling, so we will be continuing
> with that work.
> 
> For partners who already had core scheduling and experimented with
> synchronised scheduling, no-one found a way of making it function with
> less overhead than disabling hyperthreading in the first place.  This
> was was a native OS case - the virtualised case gets a compound
> performance hit because every time the guest kernel tries to
> synchronise, it forces Xen to synchronise, because there is no
> virtualisation of IPIs available in affected hardware.
> 
> 
> Overall, it looks like the one mitigation option which would allow
> hyperthreading to remain active had a larger performance penalty than
> disabling hyperthreading (on native, and remember that it is compounded
> when virtualised), and is a very invasive change to the entry/exit code.
> 
> Unless someone finds a really clever alternative plan, there doesn't
> appear to be a viable alternative to turning off hyperthreading when
> cross-thread leakage is a concern.
> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:38, Andrew Cooper wrote:
> On 16/05/2019 03:46, wencongyang (A) wrote:
>> Hi all
>>
>> Fill buffers, load ports are shared between threads on the same physical core.
>> We need to run more than one vm on the same physical core.
>> Is there any complete mitigation for environments utilizing SMT?
> 
> No - not really.
> 
> An approach which was worked on was that of synchronised scheduling,
> whereby privilege transitions are syncrhonised to ensure that we're
> never running code from different privilege levels concurrently on
> adjacent threads.  (This is the mitigation described as Group Scheduling
> in
> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
> )

synchronised scheduling is not a complete mitigation. Guest A and Guest B
run on the same physical core, and the privilege level is the same. So
Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
because they are in different privilege levels.

Thansk
Wen Congyang

> 
> However...
> 
> First of all, it depends on core scheduling as a prerequisite, and as
> you can see, that is still a work-in-progress for Xen.  There are many
> other good reasons for doing core scheduling, so we will be continuing
> with that work.
> 
> For partners who already had core scheduling and experimented with
> synchronised scheduling, no-one found a way of making it function with
> less overhead than disabling hyperthreading in the first place.  This
> was was a native OS case - the virtualised case gets a compound
> performance hit because every time the guest kernel tries to
> synchronise, it forces Xen to synchronise, because there is no
> virtualisation of IPIs available in affected hardware.
> 
> 
> Overall, it looks like the one mitigation option which would allow
> hyperthreading to remain active had a larger performance penalty than
> disabling hyperthreading (on native, and remember that it is compounded
> when virtualised), and is a very invasive change to the entry/exit code.
> 
> Unless someone finds a really clever alternative plan, there doesn't
> appear to be a viable alternative to turning off hyperthreading when
> cross-thread leakage is a concern.
> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 08:56, wencongyang (A) wrote:
>
> On 2019/5/16 15:38, Andrew Cooper wrote:
>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>> Hi all
>>>
>>> Fill buffers, load ports are shared between threads on the same physical core.
>>> We need to run more than one vm on the same physical core.
>>> Is there any complete mitigation for environments utilizing SMT?
>> No - not really.
>>
>> An approach which was worked on was that of synchronised scheduling,
>> whereby privilege transitions are syncrhonised to ensure that we're
>> never running code from different privilege levels concurrently on
>> adjacent threads.  (This is the mitigation described as Group Scheduling
>> in
>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>> )
> synchronised scheduling is not a complete mitigation. Guest A and Guest B
> run on the same physical core, and the privilege level is the same. So
> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
> because they are in different privilege levels.

This is (one of the reasons) why core scheduling is a prerequisite to
synchronised scheduling.

With core scheduling active, you will never have guest A and B
concurrently running on adjacent threads of the same core.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 08:56, wencongyang (A) wrote:
>
> On 2019/5/16 15:38, Andrew Cooper wrote:
>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>> Hi all
>>>
>>> Fill buffers, load ports are shared between threads on the same physical core.
>>> We need to run more than one vm on the same physical core.
>>> Is there any complete mitigation for environments utilizing SMT?
>> No - not really.
>>
>> An approach which was worked on was that of synchronised scheduling,
>> whereby privilege transitions are syncrhonised to ensure that we're
>> never running code from different privilege levels concurrently on
>> adjacent threads.  (This is the mitigation described as Group Scheduling
>> in
>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>> )
> synchronised scheduling is not a complete mitigation. Guest A and Guest B
> run on the same physical core, and the privilege level is the same. So
> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
> because they are in different privilege levels.

This is (one of the reasons) why core scheduling is a prerequisite to
synchronised scheduling.

With core scheduling active, you will never have guest A and B
concurrently running on adjacent threads of the same core.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:58, Andrew Cooper wrote:
> On 16/05/2019 08:56, wencongyang (A) wrote:
>>
>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>> Hi all
>>>>
>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>> We need to run more than one vm on the same physical core.
>>>> Is there any complete mitigation for environments utilizing SMT?
>>> No - not really.
>>>
>>> An approach which was worked on was that of synchronised scheduling,
>>> whereby privilege transitions are syncrhonised to ensure that we're
>>> never running code from different privilege levels concurrently on
>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>> in
>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>> )
>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>> run on the same physical core, and the privilege level is the same. So
>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>> because they are in different privilege levels.
> 
> This is (one of the reasons) why core scheduling is a prerequisite to
> synchronised scheduling.
> 
> With core scheduling active, you will never have guest A and B
> concurrently running on adjacent threads of the same core.

Thanks for you explanation. I got it.

Wen Congyang

> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:58, Andrew Cooper wrote:
> On 16/05/2019 08:56, wencongyang (A) wrote:
>>
>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>> Hi all
>>>>
>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>> We need to run more than one vm on the same physical core.
>>>> Is there any complete mitigation for environments utilizing SMT?
>>> No - not really.
>>>
>>> An approach which was worked on was that of synchronised scheduling,
>>> whereby privilege transitions are syncrhonised to ensure that we're
>>> never running code from different privilege levels concurrently on
>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>> in
>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>> )
>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>> run on the same physical core, and the privilege level is the same. So
>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>> because they are in different privilege levels.
> 
> This is (one of the reasons) why core scheduling is a prerequisite to
> synchronised scheduling.
> 
> With core scheduling active, you will never have guest A and B
> concurrently running on adjacent threads of the same core.

Thanks for you explanation. I got it.

Wen Congyang

> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:58, Andrew Cooper wrote:
> On 16/05/2019 08:56, wencongyang (A) wrote:
>>
>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>> Hi all
>>>>
>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>> We need to run more than one vm on the same physical core.
>>>> Is there any complete mitigation for environments utilizing SMT?
>>> No - not really.
>>>
>>> An approach which was worked on was that of synchronised scheduling,
>>> whereby privilege transitions are syncrhonised to ensure that we're
>>> never running code from different privilege levels concurrently on
>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>> in
>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>> )
>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>> run on the same physical core, and the privilege level is the same. So
>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>> because they are in different privilege levels.
> 
> This is (one of the reasons) why core scheduling is a prerequisite to
> synchronised scheduling.
> 
> With core scheduling active, you will never have guest A and B
> concurrently running on adjacent threads of the same core.

Another question:
There are a CPUID bit(TSX_FORCE_ABORT) and MSR(MSR_TSX_FORCE_ABORT) in xsa297/xsa297-4.7-1.patch.
But we do not find them in the patchs of kvm.

IIUC, this MSR is not for MDS mitigation. Is this right?

Thanks
Wen Congyang

> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[wencongyang (A)]

On 2019/5/16 15:58, Andrew Cooper wrote:
> On 16/05/2019 08:56, wencongyang (A) wrote:
>>
>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>> Hi all
>>>>
>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>> We need to run more than one vm on the same physical core.
>>>> Is there any complete mitigation for environments utilizing SMT?
>>> No - not really.
>>>
>>> An approach which was worked on was that of synchronised scheduling,
>>> whereby privilege transitions are syncrhonised to ensure that we're
>>> never running code from different privilege levels concurrently on
>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>> in
>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>> )
>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>> run on the same physical core, and the privilege level is the same. So
>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>> because they are in different privilege levels.
> 
> This is (one of the reasons) why core scheduling is a prerequisite to
> synchronised scheduling.
> 
> With core scheduling active, you will never have guest A and B
> concurrently running on adjacent threads of the same core.

Another question:
There are a CPUID bit(TSX_FORCE_ABORT) and MSR(MSR_TSX_FORCE_ABORT) in xsa297/xsa297-4.7-1.patch.
But we do not find them in the patchs of kvm.

IIUC, this MSR is not for MDS mitigation. Is this right?

Thanks
Wen Congyang

> 
> ~Andrew
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 12:48, wencongyang (A) wrote:
>
> On 2019/5/16 15:58, Andrew Cooper wrote:
>> On 16/05/2019 08:56, wencongyang (A) wrote:
>>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>>> Hi all
>>>>>
>>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>>> We need to run more than one vm on the same physical core.
>>>>> Is there any complete mitigation for environments utilizing SMT?
>>>> No - not really.
>>>>
>>>> An approach which was worked on was that of synchronised scheduling,
>>>> whereby privilege transitions are syncrhonised to ensure that we're
>>>> never running code from different privilege levels concurrently on
>>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>>> in
>>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>>> )
>>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>>> run on the same physical core, and the privilege level is the same. So
>>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>>> because they are in different privilege levels.
>> This is (one of the reasons) why core scheduling is a prerequisite to
>> synchronised scheduling.
>>
>> With core scheduling active, you will never have guest A and B
>> concurrently running on adjacent threads of the same core.
> Another question:
> There are a CPUID bit(TSX_FORCE_ABORT) and MSR(MSR_TSX_FORCE_ABORT) in xsa297/xsa297-4.7-1.patch.
> But we do not find them in the patchs of kvm.
>
> IIUC, this MSR is not for MDS mitigation. Is this right?

Oops - I intended to make this a bit more clear in the advisory.

Correct - this isn't to do with MDS, but the behavioural change was in
the same piece of microcode as the MDS fixes, which is why it was included.

See
https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#vpmu-x86
and rtm-abort=<bool> for the details.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Question about MDS mitigation

[Andrew Cooper]

On 16/05/2019 12:48, wencongyang (A) wrote:
>
> On 2019/5/16 15:58, Andrew Cooper wrote:
>> On 16/05/2019 08:56, wencongyang (A) wrote:
>>> On 2019/5/16 15:38, Andrew Cooper wrote:
>>>> On 16/05/2019 03:46, wencongyang (A) wrote:
>>>>> Hi all
>>>>>
>>>>> Fill buffers, load ports are shared between threads on the same physical core.
>>>>> We need to run more than one vm on the same physical core.
>>>>> Is there any complete mitigation for environments utilizing SMT?
>>>> No - not really.
>>>>
>>>> An approach which was worked on was that of synchronised scheduling,
>>>> whereby privilege transitions are syncrhonised to ensure that we're
>>>> never running code from different privilege levels concurrently on
>>>> adjacent threads.  (This is the mitigation described as Group Scheduling
>>>> in
>>>> https://software.intel.com/security-software-guidance/insights/deep-dive-intel-analysis-microarchitectural-data-sampling
>>>> )
>>> synchronised scheduling is not a complete mitigation. Guest A and Guest B
>>> run on the same physical core, and the privilege level is the same. So
>>> Guest A can infer data from Guest B. Guest A cannot infer data from hypervisor
>>> because they are in different privilege levels.
>> This is (one of the reasons) why core scheduling is a prerequisite to
>> synchronised scheduling.
>>
>> With core scheduling active, you will never have guest A and B
>> concurrently running on adjacent threads of the same core.
> Another question:
> There are a CPUID bit(TSX_FORCE_ABORT) and MSR(MSR_TSX_FORCE_ABORT) in xsa297/xsa297-4.7-1.patch.
> But we do not find them in the patchs of kvm.
>
> IIUC, this MSR is not for MDS mitigation. Is this right?

Oops - I intended to make this a bit more clear in the advisory.

Correct - this isn't to do with MDS, but the behavioural change was in
the same piece of microcode as the MDS fixes, which is why it was included.

See
https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#vpmu-x86
and rtm-abort=<bool> for the details.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Question about MDS mitigation

[Liran Alon]

Indeed those CPU resources are shared between sibling hyperthreads on same CPU core.
There is currently no mechanism merged upstream to completely mitigate SMT-enabled scenarios.
Note that this is also true for L1TF.

There are several proposal to address this but they are still in early research mode.
For example, see this KVM address space isolation patch series developed by myself and Alexandre:
https://lkml.org/lkml/2019/5/13/515
(Which should be integrated with a mechanism which kick sibling hyperthreads when switching from KVM isolated address space to full kernel address space)
This partially mimics Microsoft work regarding HyperClear which you can read more about it here:
https://techcommunity.microsoft.com/t5/Virtualization/Hyper-V-HyperClear-Mitigation-for-L1-Terminal-Fault/ba-p/382429

-Liran

> On 16 May 2019, at 5:42, wencongyang (A) <wencongyang2@huawei.com> wrote:
> 
> Hi all
> 
> Fill buffers, load ports are shared between threads on the same physical core.
> We need to run more than one vm on the same physical core.
> Is there any complete mitigation for environments utilizing SMT?
> 

Question about MDS mitigation

[wencongyang (A)]

Hi all

Fill buffers, load ports are shared between threads on the same physical core.
We need to run more than one vm on the same physical core.
Is there any complete mitigation for environments utilizing SMT?