Standardizing an MSR or other hypercall to get an RNG seed?

Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

Hi all-

I would like to standardize on a very simple protocol by which a guest
OS can obtain an RNG seed early in boot.

The main design requirements are:

 - The interface should be very easy to use.  Linux, at least, will
want to use it extremely early in boot as part of kernel ASLR.  This
means that PCI and ACPI will not work.

 - It should be synchronous.  We don't want to delay boot while
waiting for a slow host RNG.  (On Linux, at least, we have a separate
interface for that: virtio-rng.  I think that Windows has some support
for virtio-rng as well.)

 - Random numbers obtained through this interface should be
best-effort.  We want the best quality randomness that the host can
provide immediately.

It seems to me that the best interface for the actual request for a
random number is rdmsr.  This is supported on all hypervisors and all
virtualization technologies.  It can return a 64 bit random number,
and it is easy to rdmsr the same register more than once to get a
larger random number.

The main questions are what MSR index to use and how to detect the
presence of the MSR.  I've played with two approaches:

1. Use CPUID to detect the presence of this feature.  This is very
easy for KVM to implement by using a KVM-specific CPUID feature.  The
problem is that this will necessarily be KVM-specific, as the guest
must first probe for KVM and then probe for the KVM feature.  I doubt
that Hyper-V, for example, wants to claim to be KVM.  If we could
standardize a non-hypervisor-specific CPUID feature, then this problem
would go away.

2. Detect the existence of the MSR by trying to read it and handling
the #GP(0) that will occur if the MSR is not present.  Linux, at
least, is okay with doing this, and I have code to enable an IDT and
an rdmsr fixup early enough in boot to use it for ASLR.  I don't know
whether other operating systems can do this, though.

The major questions, then, are what enumeration mechanism should be
used and what MSR index should be used.

For the MSR index, we could use an MSR from the Intel range if Intel
were to give explicit approval, thus guaranteeing that nothing would
conflict.  Or we could try to agree on an MSR index in the
0x40000000-0x4fffffff range that is unlikely to conflict with
anything.

For enumeration, we could just probe the MSR if all relevant guests
are okay with this or we could standardize on a CPUID-based mechanism.
If we do the latter, I don't know what that mechanism would be.

NB: This thread will be cc'd to Microsoft and possibly Hyper-V people
shortly.  I very much appreciate Jun Nakajima's help with this!

Thanks,
Andy

-- 
Andy Lutomirski
AMA Capital Management, LLC

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: virtualization-bounces@lists.linux-foundation.org
> [mailto:virtualization-bounces@lists.linux-foundation.org] On Behalf Of Andy
> Lutomirski
> Sent: Wednesday, September 17, 2014 7:51 PM
> To: Linux Virtualization; kvm list
> Cc: Gleb Natapov; Paolo Bonzini; Theodore Ts'o; H. Peter Anvin
> Subject: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> Hi all-
> 
> I would like to standardize on a very simple protocol by which a guest OS can
> obtain an RNG seed early in boot.
> 
> The main design requirements are:
> 
>  - The interface should be very easy to use.  Linux, at least, will want to use it
> extremely early in boot as part of kernel ASLR.  This means that PCI and ACPI
> will not work.
> 
>  - It should be synchronous.  We don't want to delay boot while waiting for a
> slow host RNG.  (On Linux, at least, we have a separate interface for that:
> virtio-rng.  I think that Windows has some support for virtio-rng as well.)
> 
>  - Random numbers obtained through this interface should be best-effort.
> We want the best quality randomness that the host can provide
> immediately.
> 
> It seems to me that the best interface for the actual request for a random
> number is rdmsr.  This is supported on all hypervisors and all virtualization
> technologies.  It can return a 64 bit random number, and it is easy to rdmsr
> the same register more than once to get a larger random number.
> 
> The main questions are what MSR index to use and how to detect the
> presence of the MSR.  I've played with two approaches:
> 
> 1. Use CPUID to detect the presence of this feature.  This is very easy for
> KVM to implement by using a KVM-specific CPUID feature.  The problem is
> that this will necessarily be KVM-specific, as the guest must first probe for
> KVM and then probe for the KVM feature.  I doubt that Hyper-V, for
> example, wants to claim to be KVM.  If we could standardize a non-
> hypervisor-specific CPUID feature, then this problem would go away.

We would prefer a CPUID feature bit to detect this feature.
 
> 
> 2. Detect the existence of the MSR by trying to read it and handling the
> #GP(0) that will occur if the MSR is not present.  Linux, at least, is okay with
> doing this, and I have code to enable an IDT and an rdmsr fixup early enough
> in boot to use it for ASLR.  I don't know whether other operating systems can
> do this, though.
> 
> The major questions, then, are what enumeration mechanism should be
> used and what MSR index should be used.
> 
> For the MSR index, we could use an MSR from the Intel range if Intel were to
> give explicit approval, thus guaranteeing that nothing would conflict.  Or we
> could try to agree on an MSR index in the 0x40000000-0x4fffffff range that is
> unlikely to conflict with anything.
> 
> For enumeration, we could just probe the MSR if all relevant guests are okay
> with this or we could standardize on a CPUID-based mechanism.
> If we do the latter, I don't know what that mechanism would be.
> 
> NB: This thread will be cc'd to Microsoft and possibly Hyper-V people shortly.
> I very much appreciate Jun Nakajima's help with this!
> 
> Thanks,
> Andy


Regards,

K. Y
> 
> --
> Andy Lutomirski
> AMA Capital Management, LLC
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: virtualization-bounces@lists.linux-foundation.org
> [mailto:virtualization-bounces@lists.linux-foundation.org] On Behalf Of Andy
> Lutomirski
> Sent: Wednesday, September 17, 2014 7:51 PM
> To: Linux Virtualization; kvm list
> Cc: Gleb Natapov; Paolo Bonzini; Theodore Ts'o; H. Peter Anvin
> Subject: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> Hi all-
> 
> I would like to standardize on a very simple protocol by which a guest OS can
> obtain an RNG seed early in boot.
> 
> The main design requirements are:
> 
>  - The interface should be very easy to use.  Linux, at least, will want to use it
> extremely early in boot as part of kernel ASLR.  This means that PCI and ACPI
> will not work.
> 
>  - It should be synchronous.  We don't want to delay boot while waiting for a
> slow host RNG.  (On Linux, at least, we have a separate interface for that:
> virtio-rng.  I think that Windows has some support for virtio-rng as well.)
> 
>  - Random numbers obtained through this interface should be best-effort.
> We want the best quality randomness that the host can provide
> immediately.
> 
> It seems to me that the best interface for the actual request for a random
> number is rdmsr.  This is supported on all hypervisors and all virtualization
> technologies.  It can return a 64 bit random number, and it is easy to rdmsr
> the same register more than once to get a larger random number.
> 
> The main questions are what MSR index to use and how to detect the
> presence of the MSR.  I've played with two approaches:
> 
> 1. Use CPUID to detect the presence of this feature.  This is very easy for
> KVM to implement by using a KVM-specific CPUID feature.  The problem is
> that this will necessarily be KVM-specific, as the guest must first probe for
> KVM and then probe for the KVM feature.  I doubt that Hyper-V, for
> example, wants to claim to be KVM.  If we could standardize a non-
> hypervisor-specific CPUID feature, then this problem would go away.

We would prefer a CPUID feature bit to detect this feature.
 
> 
> 2. Detect the existence of the MSR by trying to read it and handling the
> #GP(0) that will occur if the MSR is not present.  Linux, at least, is okay with
> doing this, and I have code to enable an IDT and an rdmsr fixup early enough
> in boot to use it for ASLR.  I don't know whether other operating systems can
> do this, though.
> 
> The major questions, then, are what enumeration mechanism should be
> used and what MSR index should be used.
> 
> For the MSR index, we could use an MSR from the Intel range if Intel were to
> give explicit approval, thus guaranteeing that nothing would conflict.  Or we
> could try to agree on an MSR index in the 0x40000000-0x4fffffff range that is
> unlikely to conflict with anything.
> 
> For enumeration, we could just probe the MSR if all relevant guests are okay
> with this or we could standardize on a CPUID-based mechanism.
> If we do the latter, I don't know what that mechanism would be.
> 
> NB: This thread will be cc'd to Microsoft and possibly Hyper-V people shortly.
> I very much appreciate Jun Nakajima's help with this!
> 
> Thanks,
> Andy


Regards,

K. Y
> 
> --
> Andy Lutomirski
> AMA Capital Management, LLC
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/18/2014 07:40 AM, KY Srinivasan wrote:
>>
>> The main questions are what MSR index to use and how to detect the
>> presence of the MSR.  I've played with two approaches:
>>
>> 1. Use CPUID to detect the presence of this feature.  This is very easy for
>> KVM to implement by using a KVM-specific CPUID feature.  The problem is
>> that this will necessarily be KVM-specific, as the guest must first probe for
>> KVM and then probe for the KVM feature.  I doubt that Hyper-V, for
>> example, wants to claim to be KVM.  If we could standardize a non-
>> hypervisor-specific CPUID feature, then this problem would go away.
> 
> We would prefer a CPUID feature bit to detect this feature.
>  

I guess if we're introducing the concept of pan-OS MSRs we could also
have pan-OS CPUID.  The real issue is to get a single non-conflicting
standard.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 7:43 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 09/18/2014 07:40 AM, KY Srinivasan wrote:
>>>
>>> The main questions are what MSR index to use and how to detect the
>>> presence of the MSR.  I've played with two approaches:
>>>
>>> 1. Use CPUID to detect the presence of this feature.  This is very easy for
>>> KVM to implement by using a KVM-specific CPUID feature.  The problem is
>>> that this will necessarily be KVM-specific, as the guest must first probe for
>>> KVM and then probe for the KVM feature.  I doubt that Hyper-V, for
>>> example, wants to claim to be KVM.  If we could standardize a non-
>>> hypervisor-specific CPUID feature, then this problem would go away.
>>
>> We would prefer a CPUID feature bit to detect this feature.
>>
>
> I guess if we're introducing the concept of pan-OS MSRs we could also
> have pan-OS CPUID.  The real issue is to get a single non-conflicting
> standard.

Agreed.

KVM currently puts 0 in 0x40000000.EAX, meaning that a feature bit in
Microsoft's leaf 0x40000003 would probably not work well for KVM.  I
don't expect that Microsoft wants to start claiming to be KVM for the
purpose of using a KVM-style feature bit, so, if we went the CPUID
route, we would probably need something new.

--Andy

>
>         -hpa
>
>



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 8:38 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Thu, Sep 18, 2014 at 7:43 AM, H. Peter Anvin <hpa@zytor.com> wrote:
>> On 09/18/2014 07:40 AM, KY Srinivasan wrote:
>>>>
>>>> The main questions are what MSR index to use and how to detect the
>>>> presence of the MSR.  I've played with two approaches:
>>>>
>>>> 1. Use CPUID to detect the presence of this feature.  This is very easy for
>>>> KVM to implement by using a KVM-specific CPUID feature.  The problem is
>>>> that this will necessarily be KVM-specific, as the guest must first probe for
>>>> KVM and then probe for the KVM feature.  I doubt that Hyper-V, for
>>>> example, wants to claim to be KVM.  If we could standardize a non-
>>>> hypervisor-specific CPUID feature, then this problem would go away.
>>>
>>> We would prefer a CPUID feature bit to detect this feature.
>>>
>>
>> I guess if we're introducing the concept of pan-OS MSRs we could also
>> have pan-OS CPUID.  The real issue is to get a single non-conflicting
>> standard.
>
> Agreed.
>
> KVM currently puts 0 in 0x40000000.EAX, meaning that a feature bit in
> Microsoft's leaf 0x40000003 would probably not work well for KVM.  I
> don't expect that Microsoft wants to start claiming to be KVM for the
> purpose of using a KVM-style feature bit, so, if we went the CPUID
> route, we would probably need something new.

Slight correction: QEMU/KVM has optional support for Hyper-V feature
enumeration.  Ideally the RNG seed mechanism would be enabled by
default, but I don't know whether the QEMU maintainers would be okay
with enabling the Hyper-V cpuid mechanism in a default configuration.

--Andy

>
> --Andy
>
>>
>>         -hpa
>>
>>
>
>
>
> --
> Andy Lutomirski
> AMA Capital Management, LLC



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 18/09/2014 17:44, Andy Lutomirski ha scritto:
> Slight correction: QEMU/KVM has optional support for Hyper-V feature
> enumeration.  Ideally the RNG seed mechanism would be enabled by
> default, but I don't know whether the QEMU maintainers would be okay
> with enabling the Hyper-V cpuid mechanism in a default configuration.

Some guests cannot find the KVM leaves at 0x40000100, so it wouldn't be
great.  And I also don't know what VMware folks would think, but I think
they would be even less thrilled than me.

Note that even if there is no well-defined CPUID leaf, and the main
detection mechanism is #GP, each hypervisor is free to define a CPUID
bit of its own.

However, if it's going to be an architectural (Intel-defined) MSR, I
think the right place for a feature bit is in the low leaves (like
EAX=7, ECX=0).

Paolo

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: Andy Lutomirski [mailto:luto@amacapital.net]
> Sent: Thursday, September 18, 2014 8:38 AM
> To: H. Peter Anvin
> Cc: KY Srinivasan; Linux Virtualization; kvm list; Gleb Natapov; Paolo Bonzini;
> Theodore Ts'o
> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> On Thu, Sep 18, 2014 at 7:43 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> > On 09/18/2014 07:40 AM, KY Srinivasan wrote:
> >>>
> >>> The main questions are what MSR index to use and how to detect the
> >>> presence of the MSR.  I've played with two approaches:
> >>>
> >>> 1. Use CPUID to detect the presence of this feature.  This is very
> >>> easy for KVM to implement by using a KVM-specific CPUID feature.
> >>> The problem is that this will necessarily be KVM-specific, as the
> >>> guest must first probe for KVM and then probe for the KVM feature.
> >>> I doubt that Hyper-V, for example, wants to claim to be KVM.  If we
> >>> could standardize a non- hypervisor-specific CPUID feature, then this
> problem would go away.
> >>
> >> We would prefer a CPUID feature bit to detect this feature.
> >>
> >
> > I guess if we're introducing the concept of pan-OS MSRs we could also
> > have pan-OS CPUID.  The real issue is to get a single non-conflicting
> > standard.
> 
> Agreed.
> 
> KVM currently puts 0 in 0x40000000.EAX, meaning that a feature bit in
> Microsoft's leaf 0x40000003 would probably not work well for KVM.  I don't
> expect that Microsoft wants to start claiming to be KVM for the purpose of
> using a KVM-style feature bit, so, if we went the CPUID route, we would
> probably need something new.
> 
> --Andy

I am copying other Hyper-V engineers to this discussion.

Regards,

K. Y
> 
> >
> >         -hpa
> >
> >
> 
> 
> 
> --
> Andy Lutomirski
> AMA Capital Management, LLC

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: Andy Lutomirski [mailto:luto@amacapital.net]
> Sent: Thursday, September 18, 2014 8:38 AM
> To: H. Peter Anvin
> Cc: KY Srinivasan; Linux Virtualization; kvm list; Gleb Natapov; Paolo Bonzini;
> Theodore Ts'o
> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> On Thu, Sep 18, 2014 at 7:43 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> > On 09/18/2014 07:40 AM, KY Srinivasan wrote:
> >>>
> >>> The main questions are what MSR index to use and how to detect the
> >>> presence of the MSR.  I've played with two approaches:
> >>>
> >>> 1. Use CPUID to detect the presence of this feature.  This is very
> >>> easy for KVM to implement by using a KVM-specific CPUID feature.
> >>> The problem is that this will necessarily be KVM-specific, as the
> >>> guest must first probe for KVM and then probe for the KVM feature.
> >>> I doubt that Hyper-V, for example, wants to claim to be KVM.  If we
> >>> could standardize a non- hypervisor-specific CPUID feature, then this
> problem would go away.
> >>
> >> We would prefer a CPUID feature bit to detect this feature.
> >>
> >
> > I guess if we're introducing the concept of pan-OS MSRs we could also
> > have pan-OS CPUID.  The real issue is to get a single non-conflicting
> > standard.
> 
> Agreed.
> 
> KVM currently puts 0 in 0x40000000.EAX, meaning that a feature bit in
> Microsoft's leaf 0x40000003 would probably not work well for KVM.  I don't
> expect that Microsoft wants to start claiming to be KVM for the purpose of
> using a KVM-style feature bit, so, if we went the CPUID route, we would
> probably need something new.
> 
> --Andy

I am copying other Hyper-V engineers to this discussion.

Regards,

K. Y
> 
> >
> >         -hpa
> >
> >
> 
> 
> 
> --
> Andy Lutomirski
> AMA Capital Management, LLC

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 9:36 AM, KY Srinivasan <kys@microsoft.com> wrote:
>
> I am copying other Hyper-V engineers to this discussion.
>

Thanks, K.Y.

In terms of the address for the MSR, I suggest that you choose one
from the range between 40000000H - 400000FFH. The SDM (35.1
ARCHITECTURAL MSRS) says "All existing and
future processors will not implement any features using any MSR in
this range." Hyper-V already defines many synthetic MSRs in this
range, and I think it would be reasonable for you to pick one for this
to avoid a conflict?

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> In terms of the address for the MSR, I suggest that you choose one
> from the range between 40000000H - 400000FFH. The SDM (35.1
> ARCHITECTURAL MSRS) says "All existing and
> future processors will not implement any features using any MSR in
> this range." Hyper-V already defines many synthetic MSRs in this
> range, and I think it would be reasonable for you to pick one for this
> to avoid a conflict?

KVM is not using any MSR in that range.

However, I think it would be better to have the MSR (and perhaps CPUID)
outside the hypervisor-reserved ranges, so that it becomes
architecturally defined.  In some sense it is similar to the HYPERVISOR
CPUID feature.

Paolo

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> In terms of the address for the MSR, I suggest that you choose one
> from the range between 40000000H - 400000FFH. The SDM (35.1
> ARCHITECTURAL MSRS) says "All existing and
> future processors will not implement any features using any MSR in
> this range." Hyper-V already defines many synthetic MSRs in this
> range, and I think it would be reasonable for you to pick one for this
> to avoid a conflict?

KVM is not using any MSR in that range.

However, I think it would be better to have the MSR (and perhaps CPUID)
outside the hypervisor-reserved ranges, so that it becomes
architecturally defined.  In some sense it is similar to the HYPERVISOR
CPUID feature.

Paolo

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[Jake Oshins]

That certainly sound reasonable to me.  How do you see discovery of that working?

Thanks,
Jake Oshins


-----Original Message-----
From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
Sent: Thursday, September 18, 2014 10:18 AM
To: Nakajima, Jun; KY Srinivasan
Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> In terms of the address for the MSR, I suggest that you choose one
> from the range between 40000000H - 400000FFH. The SDM (35.1
> ARCHITECTURAL MSRS) says "All existing and
> future processors will not implement any features using any MSR in
> this range." Hyper-V already defines many synthetic MSRs in this
> range, and I think it would be reasonable for you to pick one for this
> to avoid a conflict?

KVM is not using any MSR in that range.

However, I think it would be better to have the MSR (and perhaps CPUID)
outside the hypervisor-reserved ranges, so that it becomes
architecturally defined.  In some sense it is similar to the HYPERVISOR
CPUID feature.

Paolo

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[Jake Oshins]

That certainly sound reasonable to me.  How do you see discovery of that working?

Thanks,
Jake Oshins


-----Original Message-----
From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
Sent: Thursday, September 18, 2014 10:18 AM
To: Nakajima, Jun; KY Srinivasan
Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> In terms of the address for the MSR, I suggest that you choose one
> from the range between 40000000H - 400000FFH. The SDM (35.1
> ARCHITECTURAL MSRS) says "All existing and
> future processors will not implement any features using any MSR in
> this range." Hyper-V already defines many synthetic MSRs in this
> range, and I think it would be reasonable for you to pick one for this
> to avoid a conflict?

KVM is not using any MSR in that range.

However, I think it would be better to have the MSR (and perhaps CPUID)
outside the hypervisor-reserved ranges, so that it becomes
architecturally defined.  In some sense it is similar to the HYPERVISOR
CPUID feature.

Paolo

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Thursday, September 18, 2014 10:18 AM
> To: Nakajima, Jun; KY Srinivasan
> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels
> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux
> Virtualization
> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> > In terms of the address for the MSR, I suggest that you choose one
> > from the range between 40000000H - 400000FFH. The SDM (35.1
> > ARCHITECTURAL MSRS) says "All existing and future processors will not
> > implement any features using any MSR in this range." Hyper-V already
> > defines many synthetic MSRs in this range, and I think it would be
> > reasonable for you to pick one for this to avoid a conflict?
> 
> KVM is not using any MSR in that range.
> 
> However, I think it would be better to have the MSR (and perhaps CPUID)
> outside the hypervisor-reserved ranges, so that it becomes architecturally
> defined.  In some sense it is similar to the HYPERVISOR CPUID feature.

Yes, given that we want this to be hypervisor agnostic.

K. Y

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[KY Srinivasan]

> -----Original Message-----
> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
> Bonzini
> Sent: Thursday, September 18, 2014 10:18 AM
> To: Nakajima, Jun; KY Srinivasan
> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels
> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux
> Virtualization
> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
> 
> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
> > In terms of the address for the MSR, I suggest that you choose one
> > from the range between 40000000H - 400000FFH. The SDM (35.1
> > ARCHITECTURAL MSRS) says "All existing and future processors will not
> > implement any features using any MSR in this range." Hyper-V already
> > defines many synthetic MSRs in this range, and I think it would be
> > reasonable for you to pick one for this to avoid a conflict?
> 
> KVM is not using any MSR in that range.
> 
> However, I think it would be better to have the MSR (and perhaps CPUID)
> outside the hypervisor-reserved ranges, so that it becomes architecturally
> defined.  In some sense it is similar to the HYPERVISOR CPUID feature.

Yes, given that we want this to be hypervisor agnostic.

K. Y

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 10:20 AM, KY Srinivasan <kys@microsoft.com> wrote:
>
>
>> -----Original Message-----
>> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
>> Bonzini
>> Sent: Thursday, September 18, 2014 10:18 AM
>> To: Nakajima, Jun; KY Srinivasan
>> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels
>> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux
>> Virtualization
>> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
>>
>> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
>> > In terms of the address for the MSR, I suggest that you choose one
>> > from the range between 40000000H - 400000FFH. The SDM (35.1
>> > ARCHITECTURAL MSRS) says "All existing and future processors will not
>> > implement any features using any MSR in this range." Hyper-V already
>> > defines many synthetic MSRs in this range, and I think it would be
>> > reasonable for you to pick one for this to avoid a conflict?
>>
>> KVM is not using any MSR in that range.
>>
>> However, I think it would be better to have the MSR (and perhaps CPUID)
>> outside the hypervisor-reserved ranges, so that it becomes architecturally
>> defined.  In some sense it is similar to the HYPERVISOR CPUID feature.
>
> Yes, given that we want this to be hypervisor agnostic.
>

Actually, that MSR address range has been reserved for that purpose, along with:
- CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
- CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)


-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 10:42 AM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> On Thu, Sep 18, 2014 at 10:20 AM, KY Srinivasan <kys@microsoft.com> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo
>>> Bonzini
>>> Sent: Thursday, September 18, 2014 10:18 AM
>>> To: Nakajima, Jun; KY Srinivasan
>>> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels
>>> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake Oshins; Linux
>>> Virtualization
>>> Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?
>>>
>>> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
>>> > In terms of the address for the MSR, I suggest that you choose one
>>> > from the range between 40000000H - 400000FFH. The SDM (35.1
>>> > ARCHITECTURAL MSRS) says "All existing and future processors will not
>>> > implement any features using any MSR in this range." Hyper-V already
>>> > defines many synthetic MSRs in this range, and I think it would be
>>> > reasonable for you to pick one for this to avoid a conflict?
>>>
>>> KVM is not using any MSR in that range.
>>>
>>> However, I think it would be better to have the MSR (and perhaps CPUID)
>>> outside the hypervisor-reserved ranges, so that it becomes architecturally
>>> defined.  In some sense it is similar to the HYPERVISOR CPUID feature.
>>
>> Yes, given that we want this to be hypervisor agnostic.
>>
>
> Actually, that MSR address range has been reserved for that purpose, along with:
> - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
> - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)

I don't know whether this is documented anywhere, but Linux tries to
detect a hypervisor by searching CPUID leaves 0x400xyz00 for
"KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being
in a somewhat variable location.

Do we consider this mechanism to work across all hypervisors and
guests?  That is, could we put something like "CrossHVPara\0"
somewhere in that range, where each hypervisor would be free to decide
exactly where it ends up?

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

Quite frankly it might make more sense to define a cross-VM *cpuid* range.  The cpuid leaf can just point to the MSR.  The big question is who will be willing to be the registrar.

On September 18, 2014 11:35:39 AM PDT, Andy Lutomirski <luto@amacapital.net> wrote:
>On Thu, Sep 18, 2014 at 10:42 AM, Nakajima, Jun
><jun.nakajima@intel.com> wrote:
>> On Thu, Sep 18, 2014 at 10:20 AM, KY Srinivasan <kys@microsoft.com>
>wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of
>Paolo
>>>> Bonzini
>>>> Sent: Thursday, September 18, 2014 10:18 AM
>>>> To: Nakajima, Jun; KY Srinivasan
>>>> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb
>Natapov; Niels
>>>> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake
>Oshins; Linux
>>>> Virtualization
>>>> Subject: Re: Standardizing an MSR or other hypercall to get an RNG
>seed?
>>>>
>>>> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
>>>> > In terms of the address for the MSR, I suggest that you choose
>one
>>>> > from the range between 40000000H - 400000FFH. The SDM (35.1
>>>> > ARCHITECTURAL MSRS) says "All existing and future processors will
>not
>>>> > implement any features using any MSR in this range." Hyper-V
>already
>>>> > defines many synthetic MSRs in this range, and I think it would
>be
>>>> > reasonable for you to pick one for this to avoid a conflict?
>>>>
>>>> KVM is not using any MSR in that range.
>>>>
>>>> However, I think it would be better to have the MSR (and perhaps
>CPUID)
>>>> outside the hypervisor-reserved ranges, so that it becomes
>architecturally
>>>> defined.  In some sense it is similar to the HYPERVISOR CPUID
>feature.
>>>
>>> Yes, given that we want this to be hypervisor agnostic.
>>>
>>
>> Actually, that MSR address range has been reserved for that purpose,
>along with:
>> - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
>> - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
>
>I don't know whether this is documented anywhere, but Linux tries to
>detect a hypervisor by searching CPUID leaves 0x400xyz00 for
>"KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being
>in a somewhat variable location.
>
>Do we consider this mechanism to work across all hypervisors and
>guests?  That is, could we put something like "CrossHVPara\0"
>somewhere in that range, where each hypervisor would be free to decide
>exactly where it ends up?
>
>--Andy

-- 
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[Niels Ferguson]

Defining a standard way of transferring random numbers between the host and the guest is an excellent idea.

As the person who writes the RNG code in Windows, I have a few comments:

DETECTION:
It should be possible to detect this feature through CPUID or similar mechanism. That allows the code that uses this feature to be written without needing the ability to catch CPU exceptions. I could be wrong, but as far as I know there is no support for exception handling in the Windows OS loader where we gather our initial random state.

EFFICIENCY:
Is there a way we can transfer more bytes per interaction? With a single 64-bit MSR we always need multiple reads to get a seed, and each of them results in a context switch to the host, which is expensive. This is even worse for 32-bit guests. Windows would typically need to fetch 64 bytes of random data at boot and at regular intervals. It is not a show-stopper, but better efficiency would be nice.

GUEST-TO-HOST:
Can we also define a way to have random values flow from the guest to the host? Guests are also gathering entropy from their own sources, and if we allow the guests to send random data to the host, then the host can treat it as an entropy source and all the VMs on a single host can share their entropy. (This is not a security problem; any reasonable host RNG cannot be hurt even by maliciously chosen entropy inputs.)


I don't know much about how hypervisors work on the inside, but maybe we can define a mechanism for standardized hypervisor calls that work on all hypervisors that support this feature. Then we could define a function to do an entropy exchange: the guest provides N bytes of random data to the host, and the host replies with N bytes of random data. The data exchange can now be done through memory.

A standardized hypervisor-call mechanism also seems generally useful for future features, whereas the MSR solution is very limited in what it can do. We might end up with standardized hypervisor-calls in the future for some other reason, and then the MSR solution looks very odd.

Niels


-----Original Message-----
From: H. Peter Anvin [mailto:hpa@zytor.com] 
Sent: Thursday, September 18, 2014 11:39 AM
To: Andy Lutomirski; Nakajima, Jun
Cc: KY Srinivasan; Paolo Bonzini; Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels Ferguson; David Hepkin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

Quite frankly it might make more sense to define a cross-VM *cpuid* range.  The cpuid leaf can just point to the MSR.  The big question is who will be willing to be the registrar.

On September 18, 2014 11:35:39 AM PDT, Andy Lutomirski <luto@amacapital.net> wrote:
>On Thu, Sep 18, 2014 at 10:42 AM, Nakajima, Jun 
><jun.nakajima@intel.com> wrote:
>> On Thu, Sep 18, 2014 at 10:20 AM, KY Srinivasan <kys@microsoft.com>
>wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of
>Paolo
>>>> Bonzini
>>>> Sent: Thursday, September 18, 2014 10:18 AM
>>>> To: Nakajima, Jun; KY Srinivasan
>>>> Cc: Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb
>Natapov; Niels
>>>> Ferguson; Andy Lutomirski; David Hepkin; H. Peter Anvin; Jake
>Oshins; Linux
>>>> Virtualization
>>>> Subject: Re: Standardizing an MSR or other hypercall to get an RNG
>seed?
>>>>
>>>> Il 18/09/2014 19:13, Nakajima, Jun ha scritto:
>>>> > In terms of the address for the MSR, I suggest that you choose
>one
>>>> > from the range between 40000000H - 400000FFH. The SDM (35.1 
>>>> > ARCHITECTURAL MSRS) says "All existing and future processors will
>not
>>>> > implement any features using any MSR in this range." Hyper-V
>already
>>>> > defines many synthetic MSRs in this range, and I think it would
>be
>>>> > reasonable for you to pick one for this to avoid a conflict?
>>>>
>>>> KVM is not using any MSR in that range.
>>>>
>>>> However, I think it would be better to have the MSR (and perhaps
>CPUID)
>>>> outside the hypervisor-reserved ranges, so that it becomes
>architecturally
>>>> defined.  In some sense it is similar to the HYPERVISOR CPUID
>feature.
>>>
>>> Yes, given that we want this to be hypervisor agnostic.
>>>
>>
>> Actually, that MSR address range has been reserved for that purpose,
>along with:
>> - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
>> - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
>
>I don't know whether this is documented anywhere, but Linux tries to 
>detect a hypervisor by searching CPUID leaves 0x400xyz00 for 
>"KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being in 
>a somewhat variable location.
>
>Do we consider this mechanism to work across all hypervisors and 
>guests?  That is, could we put something like "CrossHVPara\0"
>somewhere in that range, where each hypervisor would be free to decide 
>exactly where it ends up?
>
>--Andy

--
Sent from my mobile phone.  Please pardon brevity and lack of formatting.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

> >> However, I think it would be better to have the MSR (and perhaps CPUID)
> >> outside the hypervisor-reserved ranges, so that it becomes architecturally
> >> defined.  In some sense it is similar to the HYPERVISOR CPUID feature.
> >
> > Yes, given that we want this to be hypervisor agnostic.
> 
> Actually, that MSR address range has been reserved for that purpose, along
> with:
> - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
> - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)

No, that has been reserved for hypervisor-specific information (same for the MSR).
Here we want a feature that is standardized across all hypervisors.

Of course we could just agree to have a common 4000_00C0H to 4000_00FFH range
agreed upon by KVM/Xen/Hyper-V/VMware for both MSRs and CPUID.  But it would
be nice for Intel to act as the registrar, also because this particular
feature in principle can be implemented by processors too (not that it makes
much sense since you could use RDRAND, but it _could_).

Paolo

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

> > Actually, that MSR address range has been reserved for that purpose, along
> > with:
> > - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
> > - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
> 
> I don't know whether this is documented anywhere, but Linux tries to
> detect a hypervisor by searching CPUID leaves 0x400xyz00 for
> "KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being
> in a somewhat variable location.
> 
> Do we consider this mechanism to work across all hypervisors and
> guests?  That is, could we put something like "CrossHVPara\0"
> somewhere in that range, where each hypervisor would be free to decide
> exactly where it ends up?

That's also possible, but extending the hypervisor CPUID range
beywond 400000FFH is not officially sanctioned by Intel.

Xen started doing that in order to expose both Hyper-V and Xen
CPUID leaves, and KVM followed the practice.

Paolo

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 11:54 AM, Niels Ferguson <niels@microsoft.com> wrote:
> Defining a standard way of transferring random numbers between the host and the guest is an excellent idea.
>
> As the person who writes the RNG code in Windows, I have a few comments:
>
> DETECTION:
> It should be possible to detect this feature through CPUID or similar mechanism. That allows the code that uses this feature to be written without needing the ability to catch CPU exceptions. I could be wrong, but as far as I know there is no support for exception handling in the Windows OS loader where we gather our initial random state.
>

Linux is like this, too, except that I have experimental code to
create an IDT in that code, so we can handle it.  I agree, though,
that using CPUID in early boot is easier.

> EFFICIENCY:
> Is there a way we can transfer more bytes per interaction? With a single 64-bit MSR we always need multiple reads to get a seed, and each of them results in a context switch to the host, which is expensive. This is even worse for 32-bit guests. Windows would typically need to fetch 64 bytes of random data at boot and at regular intervals. It is not a show-stopper, but better efficiency would be nice.

I thought about this for a while and didn't come up with anything that
wouldn't messy.  We could fudge the MSR rax/rdx high bits to get 128
bits, but that's nonportable and awful to implement.  We could return
a random number directly from CPUID, but that's weird.

In very informal benchmarking, rdmsr wasn't that bad.  On the other
hand, I wasn't immediately planning on using the msr on an ongoing
basis on Linux guests except after suspend/resume.

>
> GUEST-TO-HOST:
> Can we also define a way to have random values flow from the guest to the host? Guests are also gathering entropy from their own sources, and if we allow the guests to send random data to the host, then the host can treat it as an entropy source and all the VMs on a single host can share their entropy. (This is not a security problem; any reasonable host RNG cannot be hurt even by maliciously chosen entropy inputs.)
>

wrmsr on the same MSR?

>
> I don't know much about how hypervisors work on the inside, but maybe we can define a mechanism for standardized hypervisor calls that work on all hypervisors that support this feature. Then we could define a function to do an entropy exchange: the guest provides N bytes of random data to the host, and the host replies with N bytes of random data. The data exchange can now be done through memory.
>
> A standardized hypervisor-call mechanism also seems generally useful for future features, whereas the MSR solution is very limited in what it can do. We might end up with standardized hypervisor-calls in the future for some other reason, and then the MSR solution looks very odd.

I think there'll be resistance to a standardized hypercall mechanism,
just because the implementations tend to be complex.  Hyper-V uses a
special page in guest physical memory that contains a trampoline.

We could use wrmsr to a register where the payload is a pointer to a
buffer to receive random bytes, but that loses some of the simplicity
of just calling rdmsr a few times.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 11:58 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> > Actually, that MSR address range has been reserved for that purpose, along
>> > with:
>> > - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
>> > - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
>>
>> I don't know whether this is documented anywhere, but Linux tries to
>> detect a hypervisor by searching CPUID leaves 0x400xyz00 for
>> "KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being
>> in a somewhat variable location.
>>
>> Do we consider this mechanism to work across all hypervisors and
>> guests?  That is, could we put something like "CrossHVPara\0"
>> somewhere in that range, where each hypervisor would be free to decide
>> exactly where it ends up?
>
> That's also possible, but extending the hypervisor CPUID range
> beywond 400000FFH is not officially sanctioned by Intel.
>
> Xen started doing that in order to expose both Hyper-V and Xen
> CPUID leaves, and KVM followed the practice.
>

Whoops.

Might Intel be willing to extend that range to 0x40000000 -
0x400fffff?  And would Microsoft be okay with using this mechanism for
discovery?

Do we have anyone from VMware in this thread?  I don't have any VMware contacts.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 12:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:

> Might Intel be willing to extend that range to 0x40000000 -
> 0x400fffff?  And would Microsoft be okay with using this mechanism for
> discovery?

So, for CPUID, the SDM (Table 3-17. Information Returned by CPUID) says today:
"No existing or future CPU will return processor identification or
feature information if the initial EAX value is in the range 40000000H
to 4FFFFFFFH."

We can define a cross-VM CPUID range from there. The CPUID can return
the index of the MSR if needed.

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 2:21 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> On Thu, Sep 18, 2014 at 12:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>
>> Might Intel be willing to extend that range to 0x40000000 -
>> 0x400fffff?  And would Microsoft be okay with using this mechanism for
>> discovery?
>
> So, for CPUID, the SDM (Table 3-17. Information Returned by CPUID) says today:
> "No existing or future CPU will return processor identification or
> feature information if the initial EAX value is in the range 40000000H
> to 4FFFFFFFH."
>
> We can define a cross-VM CPUID range from there. The CPUID can return
> the index of the MSR if needed.

Right, sorry.  I was looking at this sentence in SDM Volume 3 Section 35.1:

MSR address range between 40000000H - 400000FFH is marked as a
specially reserved range. All existing and
future processors will not implement any features using any MSR in this range.

That's not really a large enough range for us to reserve an MSR for
this.  However, KVM, is already using MSRs outside that range: it uses
0x4b564d00-0x4b564d04 or so.  I wonder whether KVM got confused by the
differing ranges for cpuid leaves and MSR indices.

Any chance that Intel could reserve a larger range to include the KVM
MSRs?  It would also be easier if the MSR indices for cross-HV
features were constants.

Thanks,
Andy

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[David Hepkin]

I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.

I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.

Thanks...

David

-----Original Message-----
From: Andy Lutomirski [mailto:luto@amacapital.net] 
Sent: Thursday, September 18, 2014 12:07 PM
To: Paolo Bonzini
Cc: Jun Nakajima; KY Srinivasan; Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels Ferguson; David Hepkin; H. Peter Anvin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

On Thu, Sep 18, 2014 at 11:58 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> > Actually, that MSR address range has been reserved for that 
>> > purpose, along
>> > with:
>> > - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
>> > - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
>>
>> I don't know whether this is documented anywhere, but Linux tries to 
>> detect a hypervisor by searching CPUID leaves 0x400xyz00 for 
>> "KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being 
>> in a somewhat variable location.
>>
>> Do we consider this mechanism to work across all hypervisors and 
>> guests?  That is, could we put something like "CrossHVPara\0"
>> somewhere in that range, where each hypervisor would be free to 
>> decide exactly where it ends up?
>
> That's also possible, but extending the hypervisor CPUID range beywond 
> 400000FFH is not officially sanctioned by Intel.
>
> Xen started doing that in order to expose both Hyper-V and Xen CPUID 
> leaves, and KVM followed the practice.
>

Whoops.

Might Intel be willing to extend that range to 0x40000000 - 0x400fffff?  And would Microsoft be okay with using this mechanism for discovery?

Do we have anyone from VMware in this thread?  I don't have any VMware contacts.

--Andy

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[David Hepkin]

I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.

I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.

Thanks...

David

-----Original Message-----
From: Andy Lutomirski [mailto:luto@amacapital.net] 
Sent: Thursday, September 18, 2014 12:07 PM
To: Paolo Bonzini
Cc: Jun Nakajima; KY Srinivasan; Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; Niels Ferguson; David Hepkin; H. Peter Anvin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

On Thu, Sep 18, 2014 at 11:58 AM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
>> > Actually, that MSR address range has been reserved for that 
>> > purpose, along
>> > with:
>> > - CPUID.EAX=1 -> ECX bit 31 (always returns 0 on bare metal)
>> > - CPUID.EAX=4000_00xxH leaves (i.e. HYPERVISOR CPUID)
>>
>> I don't know whether this is documented anywhere, but Linux tries to 
>> detect a hypervisor by searching CPUID leaves 0x400xyz00 for 
>> "KVMKVMKVM\0\0\0", so at least Linux can handle the KVM leaves being 
>> in a somewhat variable location.
>>
>> Do we consider this mechanism to work across all hypervisors and 
>> guests?  That is, could we put something like "CrossHVPara\0"
>> somewhere in that range, where each hypervisor would be free to 
>> decide exactly where it ends up?
>
> That's also possible, but extending the hypervisor CPUID range beywond 
> 400000FFH is not officially sanctioned by Intel.
>
> Xen started doing that in order to expose both Hyper-V and Xen CPUID 
> leaves, and KVM followed the practice.
>

Whoops.

Might Intel be willing to extend that range to 0x40000000 - 0x400fffff?  And would Microsoft be okay with using this mechanism for discovery?

Do we have anyone from VMware in this thread?  I don't have any VMware contacts.

--Andy

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[David Hepkin]

The chief advantage I see to using a hypercall based mechanism is that it would work across more architectures.  MSR's and CPUID's are specific to X86.  If we ever wanted this same mechanism to be available on an architecture that doesn't support MSR's,  a hypercall based approach would allow for a more consistent mechanism across the architectures.

I agree, though, that converging on a common hypercall interface that would be implemented by all of the hypervisors would likely be much harder to achieve.

Thanks...

David

-----Original Message-----
From: Andy Lutomirski [mailto:luto@amacapital.net] 
Sent: Thursday, September 18, 2014 12:04 PM
To: Niels Ferguson
Cc: H. Peter Anvin; Nakajima, Jun; KY Srinivasan; Paolo Bonzini; Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; David Hepkin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

On Thu, Sep 18, 2014 at 11:54 AM, Niels Ferguson <niels@microsoft.com> wrote:
> Defining a standard way of transferring random numbers between the host and the guest is an excellent idea.
>
> As the person who writes the RNG code in Windows, I have a few comments:
>
> DETECTION:
> It should be possible to detect this feature through CPUID or similar mechanism. That allows the code that uses this feature to be written without needing the ability to catch CPU exceptions. I could be wrong, but as far as I know there is no support for exception handling in the Windows OS loader where we gather our initial random state.
>

Linux is like this, too, except that I have experimental code to create an IDT in that code, so we can handle it.  I agree, though, that using CPUID in early boot is easier.

> EFFICIENCY:
> Is there a way we can transfer more bytes per interaction? With a single 64-bit MSR we always need multiple reads to get a seed, and each of them results in a context switch to the host, which is expensive. This is even worse for 32-bit guests. Windows would typically need to fetch 64 bytes of random data at boot and at regular intervals. It is not a show-stopper, but better efficiency would be nice.

I thought about this for a while and didn't come up with anything that wouldn't messy.  We could fudge the MSR rax/rdx high bits to get 128 bits, but that's nonportable and awful to implement.  We could return a random number directly from CPUID, but that's weird.

In very informal benchmarking, rdmsr wasn't that bad.  On the other hand, I wasn't immediately planning on using the msr on an ongoing basis on Linux guests except after suspend/resume.

>
> GUEST-TO-HOST:
> Can we also define a way to have random values flow from the guest to 
> the host? Guests are also gathering entropy from their own sources, 
> and if we allow the guests to send random data to the host, then the 
> host can treat it as an entropy source and all the VMs on a single 
> host can share their entropy. (This is not a security problem; any 
> reasonable host RNG cannot be hurt even by maliciously chosen entropy 
> inputs.)
>

wrmsr on the same MSR?

>
> I don't know much about how hypervisors work on the inside, but maybe we can define a mechanism for standardized hypervisor calls that work on all hypervisors that support this feature. Then we could define a function to do an entropy exchange: the guest provides N bytes of random data to the host, and the host replies with N bytes of random data. The data exchange can now be done through memory.
>
> A standardized hypervisor-call mechanism also seems generally useful for future features, whereas the MSR solution is very limited in what it can do. We might end up with standardized hypervisor-calls in the future for some other reason, and then the MSR solution looks very odd.

I think there'll be resistance to a standardized hypercall mechanism, just because the implementations tend to be complex.  Hyper-V uses a special page in guest physical memory that contains a trampoline.

We could use wrmsr to a register where the payload is a pointer to a buffer to receive random bytes, but that loses some of the simplicity of just calling rdmsr a few times.

--Andy

RE: Standardizing an MSR or other hypercall to get an RNG seed?

[David Hepkin]

The chief advantage I see to using a hypercall based mechanism is that it would work across more architectures.  MSR's and CPUID's are specific to X86.  If we ever wanted this same mechanism to be available on an architecture that doesn't support MSR's,  a hypercall based approach would allow for a more consistent mechanism across the architectures.

I agree, though, that converging on a common hypercall interface that would be implemented by all of the hypervisors would likely be much harder to achieve.

Thanks...

David

-----Original Message-----
From: Andy Lutomirski [mailto:luto@amacapital.net] 
Sent: Thursday, September 18, 2014 12:04 PM
To: Niels Ferguson
Cc: H. Peter Anvin; Nakajima, Jun; KY Srinivasan; Paolo Bonzini; Mathew John; Theodore Ts'o; John Starks; kvm list; Gleb Natapov; David Hepkin; Jake Oshins; Linux Virtualization
Subject: Re: Standardizing an MSR or other hypercall to get an RNG seed?

On Thu, Sep 18, 2014 at 11:54 AM, Niels Ferguson <niels@microsoft.com> wrote:
> Defining a standard way of transferring random numbers between the host and the guest is an excellent idea.
>
> As the person who writes the RNG code in Windows, I have a few comments:
>
> DETECTION:
> It should be possible to detect this feature through CPUID or similar mechanism. That allows the code that uses this feature to be written without needing the ability to catch CPU exceptions. I could be wrong, but as far as I know there is no support for exception handling in the Windows OS loader where we gather our initial random state.
>

Linux is like this, too, except that I have experimental code to create an IDT in that code, so we can handle it.  I agree, though, that using CPUID in early boot is easier.

> EFFICIENCY:
> Is there a way we can transfer more bytes per interaction? With a single 64-bit MSR we always need multiple reads to get a seed, and each of them results in a context switch to the host, which is expensive. This is even worse for 32-bit guests. Windows would typically need to fetch 64 bytes of random data at boot and at regular intervals. It is not a show-stopper, but better efficiency would be nice.

I thought about this for a while and didn't come up with anything that wouldn't messy.  We could fudge the MSR rax/rdx high bits to get 128 bits, but that's nonportable and awful to implement.  We could return a random number directly from CPUID, but that's weird.

In very informal benchmarking, rdmsr wasn't that bad.  On the other hand, I wasn't immediately planning on using the msr on an ongoing basis on Linux guests except after suspend/resume.

>
> GUEST-TO-HOST:
> Can we also define a way to have random values flow from the guest to 
> the host? Guests are also gathering entropy from their own sources, 
> and if we allow the guests to send random data to the host, then the 
> host can treat it as an entropy source and all the VMs on a single 
> host can share their entropy. (This is not a security problem; any 
> reasonable host RNG cannot be hurt even by maliciously chosen entropy 
> inputs.)
>

wrmsr on the same MSR?

>
> I don't know much about how hypervisors work on the inside, but maybe we can define a mechanism for standardized hypervisor calls that work on all hypervisors that support this feature. Then we could define a function to do an entropy exchange: the guest provides N bytes of random data to the host, and the host replies with N bytes of random data. The data exchange can now be done through memory.
>
> A standardized hypervisor-call mechanism also seems generally useful for future features, whereas the MSR solution is very limited in what it can do. We might end up with standardized hypervisor-calls in the future for some other reason, and then the MSR solution looks very odd.

I think there'll be resistance to a standardized hypercall mechanism, just because the implementations tend to be complex.  Hyper-V uses a special page in guest physical memory that contains a trampoline.

We could use wrmsr to a register where the payload is a pointer to a buffer to receive random bytes, but that loses some of the simplicity of just calling rdmsr a few times.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/18/2014 02:46 PM, David Hepkin wrote:
> I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.
> 
> I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.
> 

Yes, I would agree that if anything we should define a new range unique
to this cross-VM interface, e.g. 0x48000000.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 2:46 PM, David Hepkin <davidhep@microsoft.com> wrote:
> I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.

Linux already does this, which is arguably unfortunate.  But it's not
quite that bad; the KVM and Xen code is only scanning at increments of
0x100.

I think that Linux as a guest would have no problem with checking the
Hyper-V range or some new range.  I don't think that Linux would want
to have to set a guest OS identity, and it's not entirely clear to me
whether this would be necessary to use the Hyper-V mechanism.

>
> I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.

I also don't know whether QEMU and KVM would be okay with implementing
the host side of the Hyper-V mechanism by default.  They would have to
implement at least leaves 0x40000001 and 0x4000002, plus correctly
reporting zeros through whatever leaf is used for this new feature.
Gleb?  Paolo?

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/18/2014 03:00 PM, Andy Lutomirski wrote:
> On Thu, Sep 18, 2014 at 2:46 PM, David Hepkin <davidhep@microsoft.com> wrote:
>> I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.
> 
> Linux already does this, which is arguably unfortunate.  But it's not
> quite that bad; the KVM and Xen code is only scanning at increments of
> 0x100.
> 
> I think that Linux as a guest would have no problem with checking the
> Hyper-V range or some new range.  I don't think that Linux would want
> to have to set a guest OS identity, and it's not entirely clear to me
> whether this would be necessary to use the Hyper-V mechanism.
> 

We really don't want to have to do this in early code, though.

>>
>> I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.
> 
> I also don't know whether QEMU and KVM would be okay with implementing
> the host side of the Hyper-V mechanism by default.  They would have to
> implement at least leaves 0x40000001 and 0x4000002, plus correctly
> reporting zeros through whatever leaf is used for this new feature.
> Gleb?  Paolo?
> 

The problem is what happens with a noncooperating hypervisor.  I guess
we could put a magic number in one of the leaf registers, but still...

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 2:57 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 09/18/2014 02:46 PM, David Hepkin wrote:
>> I'm not sure what you mean by "this mechanism?"  Are you suggesting that each hypervisor put "CrossHVPara\0" somewhere in the 0x40000000 - 0x400fffff CPUID range, and an OS has to do a full scan of this CPUID range on boot to find it?  That seems pretty inefficient.  An OS will take 1000's of hypervisor intercepts on every boot just to search this CPUID range.
>>
>> I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.
>>
>
> Yes, I would agree that if anything we should define a new range unique
> to this cross-VM interface, e.g. 0x48000000.

So, as a concrete straw-man:

CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
EBX, ECX, and EDX.

CPUID 0x48000001.EAX would contain an MSR number to read to get a
random number if supported and zero if not supported.

Questions:

1. Can we use a fixed MSR number?  This would be a little bit simpler,
but it would depend on getting a wider MSR range from Intel.

2. Who would host and maintain such a spec?  I could do it on github,
but this seems a bit silly.  Other options would include Intel,
Microsoft, or perhaps the Linux Foundation.  I don't know whether
Intel or LF would want to do this, and MS isn't exactly
vendor-neutral.  (Even L-F isn't entirely neutral, since they sort of
represent two hypervisors.)  Or we could do something temporary and
then try to work with a group like OASIS, but that might end up being
a lot of work.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:

> So, as a concrete straw-man:
>
> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
> EBX, ECX, and EDX.
>
> CPUID 0x48000001.EAX would contain an MSR number to read to get a
> random number if supported and zero if not supported.
>
> Questions:
>
> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
> but it would depend on getting a wider MSR range from Intel.
>

Why do you need a wider MSR range if you always detect the feature by
CPUID.0x48000001?
Or are you still trying to avoid the detection by CPUID?

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:

> So, as a concrete straw-man:
>
> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
> EBX, ECX, and EDX.
>
> CPUID 0x48000001.EAX would contain an MSR number to read to get a
> random number if supported and zero if not supported.
>
> Questions:
>
> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
> but it would depend on getting a wider MSR range from Intel.
>

Why do you need a wider MSR range if you always detect the feature by
CPUID.0x48000001?
Or are you still trying to avoid the detection by CPUID?

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 5:49 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
> On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>
>> So, as a concrete straw-man:
>>
>> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
>> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
>> EBX, ECX, and EDX.
>>
>> CPUID 0x48000001.EAX would contain an MSR number to read to get a
>> random number if supported and zero if not supported.
>>
>> Questions:
>>
>> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
>> but it would depend on getting a wider MSR range from Intel.
>>
>
> Why do you need a wider MSR range if you always detect the feature by
> CPUID.0x48000001?
> Or are you still trying to avoid the detection by CPUID?

Detecting the feature is one thing, but figuring out the MSR index is
another.  We could shove the index into the cpuid leaf, but that seems
unnecessarily indirect.  I'd much rather just say that CPUID leaves
*and* MSR indexes 0x48000000-0x4800ffff or so are reserved for the
cross-HV mechanism, but we can't do that without either knowingly
violating the SDM assignments or asking Intel to consider allocating
more MSR indexes.

Also, KVM is already conflicting with the SDM right now in its MSR
choice :(  I *think* that KVM could be changed to fix that, but 256
MSRs is rather confining given that KVM currently implements its own
MSR index *and* part of the Hyper-V index.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 6:03 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Thu, Sep 18, 2014 at 5:49 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>> On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>
>>> So, as a concrete straw-man:
>>>
>>> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
>>> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
>>> EBX, ECX, and EDX.
>>>
>>> CPUID 0x48000001.EAX would contain an MSR number to read to get a
>>> random number if supported and zero if not supported.
>>>
>>> Questions:
>>>
>>> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
>>> but it would depend on getting a wider MSR range from Intel.
>>>
>>
>> Why do you need a wider MSR range if you always detect the feature by
>> CPUID.0x48000001?
>> Or are you still trying to avoid the detection by CPUID?
>
> Detecting the feature is one thing, but figuring out the MSR index is
> another.  We could shove the index into the cpuid leaf, but that seems
> unnecessarily indirect.  I'd much rather just say that CPUID leaves
> *and* MSR indexes 0x48000000-0x4800ffff or so are reserved for the
> cross-HV mechanism, but we can't do that without either knowingly
> violating the SDM assignments or asking Intel to consider allocating
> more MSR indexes.
>
> Also, KVM is already conflicting with the SDM right now in its MSR
> choice :(  I *think* that KVM could be changed to fix that, but 256
> MSRs is rather confining given that KVM currently implements its own
> MSR index *and* part of the Hyper-V index.

Correction and update:

KVM currently implements its own MSRs and, optionally, some of the
Hyper-V MSRs.  By my count, Linux knows about 68 Hyper-V MSRs (in a
header file), and there are current 7 KVM MSRs, so over 1/4 of the
available MSR indices are taken (and even more would be taken if KVM
were to move its MSRs into the correct range).

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Thu, Sep 18, 2014 at 6:03 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Thu, Sep 18, 2014 at 5:49 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>> On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>
>>> So, as a concrete straw-man:
>>>
>>> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
>>> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
>>> EBX, ECX, and EDX.
>>>
>>> CPUID 0x48000001.EAX would contain an MSR number to read to get a
>>> random number if supported and zero if not supported.
>>>
>>> Questions:
>>>
>>> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
>>> but it would depend on getting a wider MSR range from Intel.
>>>
>>
>> Why do you need a wider MSR range if you always detect the feature by
>> CPUID.0x48000001?
>> Or are you still trying to avoid the detection by CPUID?
>
> Detecting the feature is one thing, but figuring out the MSR index is
> another.  We could shove the index into the cpuid leaf, but that seems
> unnecessarily indirect.  I'd much rather just say that CPUID leaves
> *and* MSR indexes 0x48000000-0x4800ffff or so are reserved for the
> cross-HV mechanism, but we can't do that without either knowingly
> violating the SDM assignments or asking Intel to consider allocating
> more MSR indexes.
>
> Also, KVM is already conflicting with the SDM right now in its MSR
> choice :(  I *think* that KVM could be changed to fix that, but 256
> MSRs is rather confining given that KVM currently implements its own
> MSR index *and* part of the Hyper-V index.

Correction and update:

KVM currently implements its own MSRs and, optionally, some of the
Hyper-V MSRs.  By my count, Linux knows about 68 Hyper-V MSRs (in a
header file), and there are current 7 KVM MSRs, so over 1/4 of the
available MSR indices are taken (and even more would be taken if KVM
were to move its MSRs into the correct range).

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 18/09/2014 23:54, David Hepkin ha scritto:
> The chief advantage I see to using a hypercall based mechanism is
> that it would work across more architectures.  MSR's and CPUID's are
> specific to X86.  If we ever wanted this same mechanism to be
> available on an architecture that doesn't support MSR's,  a hypercall
> based approach would allow for a more consistent mechanism across the
> architectures.
> 
> I agree, though, that converging on a common hypercall interface that
> would be implemented by all of the hypervisors would likely be much
> harder to achieve.

There are differences between architectures at the hypercall level,
starting with the calling convention.  So I don't think it makes much
sense to use a hypercall.

Paolo

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Thu, Sep 18, 2014 at 6:28 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Thu, Sep 18, 2014 at 6:03 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>> On Thu, Sep 18, 2014 at 5:49 PM, Nakajima, Jun <jun.nakajima@intel.com> wrote:
>>> On Thu, Sep 18, 2014 at 3:07 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>>>
>>>> So, as a concrete straw-man:
>>>>
>>>> CPUID leaf 0x48000000 would return a maximum leaf number in EAX (e.g.
>>>> 0x48000001) along with a signature value (e.g. "CrossHVPara\0") in
>>>> EBX, ECX, and EDX.
>>>>
>>>> CPUID 0x48000001.EAX would contain an MSR number to read to get a
>>>> random number if supported and zero if not supported.
>>>>
>>>> Questions:
>>>>
>>>> 1. Can we use a fixed MSR number?  This would be a little bit simpler,
>>>> but it would depend on getting a wider MSR range from Intel.
>>>>
>>>
>>> Why do you need a wider MSR range if you always detect the feature by
>>> CPUID.0x48000001?
>>> Or are you still trying to avoid the detection by CPUID?
>>
>> Detecting the feature is one thing, but figuring out the MSR index is
>> another.  We could shove the index into the cpuid leaf, but that seems
>> unnecessarily indirect.  I'd much rather just say that CPUID leaves
>> *and* MSR indexes 0x48000000-0x4800ffff or so are reserved for the
>> cross-HV mechanism, but we can't do that without either knowingly
>> violating the SDM assignments or asking Intel to consider allocating
>> more MSR indexes.
>>
>> Also, KVM is already conflicting with the SDM right now in its MSR
>> choice :(  I *think* that KVM could be changed to fix that, but 256
>> MSRs is rather confining given that KVM currently implements its own
>> MSR index *and* part of the Hyper-V index.
>
> Correction and update:
>
> KVM currently implements its own MSRs and, optionally, some of the
> Hyper-V MSRs.  By my count, Linux knows about 68 Hyper-V MSRs (in a
> header file), and there are current 7 KVM MSRs, so over 1/4 of the
> available MSR indices are taken (and even more would be taken if KVM
> were to move its MSRs into the correct range).
>

I slept on it, and I think using the CPUID instruction alone would be
simple and efficient:
- We have a huge space for CPUID leaves
- CPUID also works for user-level
- It can take an additional 32-bit parameter (ECX), and returns 4
32-bit values (EAX, EBX, ECX, and EDX).  RDMSR, for example, returns a
64-bit value.

Basically we can use it to implement a hypercall (rather than VMCALL).

For example,
- CPUID 0x48000001.EAX would return the feature presence (e.g. in
EBX), and the result in EDX:EAX (if present) at the same time, or
- CPUID 0x48000001.EAX would return the feature presence only, and
CPUID 0x48000002.EAX (acts like a hypercall) returns up to 4 32-bit
values.

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 19/09/2014 18:14, Nakajima, Jun ha scritto:
> For example,
> - CPUID 0x48000001.EAX would return the feature presence (e.g. in
> EBX), and the result in EDX:EAX (if present) at the same time, or
> - CPUID 0x48000001.EAX would return the feature presence only, and
> CPUID 0x48000002.EAX (acts like a hypercall) returns up to 4 32-bit
> values.

The latter is much better, because an "unknown" CPUID will return the
value of the highest leaf below 0x80000000, and conflicts can happen easily.

Paolo

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Thu, Sep 18, 2014 at 03:00:05PM -0700, Andy Lutomirski wrote:
> On Thu, Sep 18, 2014 at 2:46 PM, David Hepkin <davidhep@microsoft.com> wrote:
> > I suggest we come to consensus on a specific CPUID leaf where an OS needs to look to determine if a hypervisor supports this capability.  We could define a new CPUID leaf range at a well-defined location, or we could just use one of the existing CPUID leaf ranges implemented by an existing hypervisor.  I'm not familiar with the KVM CPUID leaf range, but in the case of Hyper-V, the Hyper-V CPUID leaf range was architected to allow for other hypervisors to implement it and just show through specific capabilities supported by the hypervisor.  So, we could define a bit in the Hyper-V CPUID leaf range (since Xen and KVM also implement this range), but that would require Linux to look in that range on boot to discover this capability.
> 
> I also don't know whether QEMU and KVM would be okay with implementing
> the host side of the Hyper-V mechanism by default.  They would have to
> implement at least leaves 0x40000001 and 0x4000002, plus correctly
> reporting zeros through whatever leaf is used for this new feature.
> Gleb?  Paolo?
> 
KVM and any other hypervisor out there already implement capability
detection mechanism in 0x40000000 range, and of course all of them do
it differently. Linux detects what hypervior it runs on very early and
switch to correspondent code to handle each hypervisor. Quite frankly
I do not see what problem you are trying to fix with standardizing MSR
to get RND and detection mechanism for this MSR. RND MSR is in no way
unique here. There are other mechanisms that are virtually identical
between hypervisors but have different gust/hypervisor interfaces and
are detected differently on different hypervisors. Examples are pvclock,
pveoi may be others.

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>
> Linux detects what hypervior it runs on very early

Not anywhere close to early enough.  We're talking for uses like kASLR.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 09:14 AM, Nakajima, Jun wrote:
> 
> I slept on it, and I think using the CPUID instruction alone would be
> simple and efficient:
> - We have a huge space for CPUID leaves
> - CPUID also works for user-level
> - It can take an additional 32-bit parameter (ECX), and returns 4
> 32-bit values (EAX, EBX, ECX, and EDX).  RDMSR, for example, returns a
> 64-bit value.
> 
> Basically we can use it to implement a hypercall (rather than VMCALL).
> 
> For example,
> - CPUID 0x48000001.EAX would return the feature presence (e.g. in
> EBX), and the result in EDX:EAX (if present) at the same time, or
> - CPUID 0x48000001.EAX would return the feature presence only, and
> CPUID 0x48000002.EAX (acts like a hypercall) returns up to 4 32-bit
> values.
> 

There is a huge disadvantage to the fact that CPUID is a user space
instruction, though.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >
> > Linux detects what hypervior it runs on very early
> 
> Not anywhere close to early enough.  We're talking for uses like kASLR.
> 
Still to early to do:

   h = cpuid(HYPERVIOR_SIGNATURE)
   if (h == KVMKVMKVM) {
      if (cpuid(kvm_features) & kvm_rnd)
         rdmsr(kvm_rnd)
   else (h == HyperV) {
      if (cpuid(hv_features) & hv_rnd)
        rdmsr(hv_rnd) 
   else (h == XenXenXen) {
      if (cpuid(xen_features) & xen_rnd)
        rdmsr(xen_rnd)
  }

?

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>>>
>>> Linux detects what hypervior it runs on very early
>>
>> Not anywhere close to early enough.  We're talking for uses like kASLR.
>>
> Still to early to do:
> 
>    h = cpuid(HYPERVIOR_SIGNATURE)
>    if (h == KVMKVMKVM) {
>       if (cpuid(kvm_features) & kvm_rnd)
>          rdmsr(kvm_rnd)
>    else (h == HyperV) {
>       if (cpuid(hv_features) & hv_rnd)
>         rdmsr(hv_rnd) 
>    else (h == XenXenXen) {
>       if (cpuid(xen_features) & xen_rnd)
>         rdmsr(xen_rnd)
>   }
> 

If we need to do chase loops, especially not so...

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> > On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> >> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >>>
> >>> Linux detects what hypervior it runs on very early
> >>
> >> Not anywhere close to early enough.  We're talking for uses like kASLR.
> >>
> > Still to early to do:
> > 
> >    h = cpuid(HYPERVIOR_SIGNATURE)
> >    if (h == KVMKVMKVM) {
> >       if (cpuid(kvm_features) & kvm_rnd)
> >          rdmsr(kvm_rnd)
> >    else (h == HyperV) {
> >       if (cpuid(hv_features) & hv_rnd)
> >         rdmsr(hv_rnd) 
> >    else (h == XenXenXen) {
> >       if (cpuid(xen_features) & xen_rnd)
> >         rdmsr(xen_rnd)
> >   }
> > 
> 
> If we need to do chase loops, especially not so...
> 
What loops exactly? As a non native English speaker I fail to understand
if your answer is "yes" or "no" ;)

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
>> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
>>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
>>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>>>>>
>>>>> Linux detects what hypervior it runs on very early
>>>>
>>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
>>>>
>>> Still to early to do:
>>>
>>>    h = cpuid(HYPERVIOR_SIGNATURE)
>>>    if (h == KVMKVMKVM) {
>>>       if (cpuid(kvm_features) & kvm_rnd)
>>>          rdmsr(kvm_rnd)
>>>    else (h == HyperV) {
>>>       if (cpuid(hv_features) & hv_rnd)
>>>         rdmsr(hv_rnd) 
>>>    else (h == XenXenXen) {
>>>       if (cpuid(xen_features) & xen_rnd)
>>>         rdmsr(xen_rnd)
>>>   }
>>>
>>
>> If we need to do chase loops, especially not so...
>>
> What loops exactly? As a non native English speaker I fail to understand
> if your answer is "yes" or "no" ;)
> 

The above isn't actually the full algorithm used.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
>> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
>>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
>>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>>>>>
>>>>> Linux detects what hypervior it runs on very early
>>>>
>>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
>>>>
>>> Still to early to do:
>>>
>>>    h = cpuid(HYPERVIOR_SIGNATURE)
>>>    if (h == KVMKVMKVM) {
>>>       if (cpuid(kvm_features) & kvm_rnd)
>>>          rdmsr(kvm_rnd)
>>>    else (h == HyperV) {
>>>       if (cpuid(hv_features) & hv_rnd)
>>>         rdmsr(hv_rnd) 
>>>    else (h == XenXenXen) {
>>>       if (cpuid(xen_features) & xen_rnd)
>>>         rdmsr(xen_rnd)
>>>   }
>>>
>>
>> If we need to do chase loops, especially not so...
>>
> What loops exactly? As a non native English speaker I fail to understand
> if your answer is "yes" or "no" ;)
> 

The above isn't actually the full algorithm used.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Sep 19, 2014 9:53 AM, "Gleb Natapov" <gleb@kernel.org> wrote:
>
> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> > On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> > >
> > > Linux detects what hypervior it runs on very early
> >
> > Not anywhere close to early enough.  We're talking for uses like kASLR.
> >
> Still to early to do:
>
>    h = cpuid(HYPERVIOR_SIGNATURE)
>    if (h == KVMKVMKVM) {
>       if (cpuid(kvm_features) & kvm_rnd)
>          rdmsr(kvm_rnd)
>    else (h == HyperV) {
>       if (cpuid(hv_features) & hv_rnd)
>         rdmsr(hv_rnd)
>    else (h == XenXenXen) {
>       if (cpuid(xen_features) & xen_rnd)
>         rdmsr(xen_rnd)
>   }
>

I think that there's a lot of value in having each guest
implementation be automatically compatible with all hypervisors.  For
example, you forgot VMware, and VMware might be less excited about
implementing this feature if all the guests won't immediately start
using it.

> ?
>
> --
>                         Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Sep 19, 2014 9:40 AM, "H. Peter Anvin" <hpa@zytor.com> wrote:
>
> On 09/19/2014 09:14 AM, Nakajima, Jun wrote:
> >
> > I slept on it, and I think using the CPUID instruction alone would be
> > simple and efficient:
> > - We have a huge space for CPUID leaves
> > - CPUID also works for user-level
> > - It can take an additional 32-bit parameter (ECX), and returns 4
> > 32-bit values (EAX, EBX, ECX, and EDX).  RDMSR, for example, returns a
> > 64-bit value.
> >
> > Basically we can use it to implement a hypercall (rather than VMCALL).
> >
> > For example,
> > - CPUID 0x48000001.EAX would return the feature presence (e.g. in
> > EBX), and the result in EDX:EAX (if present) at the same time, or
> > - CPUID 0x48000001.EAX would return the feature presence only, and
> > CPUID 0x48000002.EAX (acts like a hypercall) returns up to 4 32-bit
> > values.
> >
>
> There is a huge disadvantage to the fact that CPUID is a user space
> instruction, though.

We can always make cpuid on the leaf in question return all zeros if CPL > 0.

>
>         -hpa
>

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 10:21 AM, Andy Lutomirski wrote:
>>
>> There is a huge disadvantage to the fact that CPUID is a user space
>> instruction, though.
> 
> We can always make cpuid on the leaf in question return all zeros if CPL > 0.
> 

Not sure that is better...

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 10:36 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> On 09/19/2014 10:21 AM, Andy Lutomirski wrote:
>>>
>>> There is a huge disadvantage to the fact that CPUID is a user space
>>> instruction, though.
>>
>> We can always make cpuid on the leaf in question return all zeros if CPL > 0.
>>
>
> Not sure that is better...

It's better than #GP...

This is why I prefer rdmsr: the privilege semantics are already
appropriate.  Also, I wouldn't be surprised if shoehorning
non-constant results into cpuid implementations might be awkward for
some hypervisors.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >>>>>
> >>>>> Linux detects what hypervior it runs on very early
> >>>>
> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
> >>>>
> >>> Still to early to do:
> >>>
> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >>>    if (h == KVMKVMKVM) {
> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >>>          rdmsr(kvm_rnd)
> >>>    else (h == HyperV) {
> >>>       if (cpuid(hv_features) & hv_rnd)
> >>>         rdmsr(hv_rnd) 
> >>>    else (h == XenXenXen) {
> >>>       if (cpuid(xen_features) & xen_rnd)
> >>>         rdmsr(xen_rnd)
> >>>   }
> >>>
> >>
> >> If we need to do chase loops, especially not so...
> >>
> > What loops exactly? As a non native English speaker I fail to understand
> > if your answer is "yes" or "no" ;)
> > 
> 
> The above isn't actually the full algorithm used.
> 
What part of actually algorithm cannot be implemented? Loop that searches
for KVM leaf in case KVM pretend to be HyperV (is this what you called
"chase loops"?)? First of all there is no need to implement it, if KVM
pretends to be HyperV use HyperV's way to obtain RNG, but what is the
problem with the loop?

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 10:21:27AM -0700, Andy Lutomirski wrote:
> On Sep 19, 2014 9:53 AM, "Gleb Natapov" <gleb@kernel.org> wrote:
> >
> > On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> > > On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> > > >
> > > > Linux detects what hypervior it runs on very early
> > >
> > > Not anywhere close to early enough.  We're talking for uses like kASLR.
> > >
> > Still to early to do:
> >
> >    h = cpuid(HYPERVIOR_SIGNATURE)
> >    if (h == KVMKVMKVM) {
> >       if (cpuid(kvm_features) & kvm_rnd)
> >          rdmsr(kvm_rnd)
> >    else (h == HyperV) {
> >       if (cpuid(hv_features) & hv_rnd)
> >         rdmsr(hv_rnd)
> >    else (h == XenXenXen) {
> >       if (cpuid(xen_features) & xen_rnd)
> >         rdmsr(xen_rnd)
> >   }
> >
> 
> I think that there's a lot of value in having each guest
> implementation be automatically compatible with all hypervisors.  For
> example, you forgot VMware, and VMware might be less excited about
> implementing this feature if all the guests won't immediately start
> using it.
> 
I "forgot" VMware because I do not see VMware people to be CCed. They may
be even less excited about them being told _how_ this feature need to be
implemented (e.g implement HyperV leafs for the feature detection). I
do not want to and cannot speak for VMware, but my guess is that for
them it would be much easier to add an else clause for VMware in above
"if" then to coordinate with all hypervisor developers about MSR/cpuid
details. And since this is security feature implementing it for Linux
is in their best interest.

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 10:21:27AM -0700, Andy Lutomirski wrote:
> On Sep 19, 2014 9:53 AM, "Gleb Natapov" <gleb@kernel.org> wrote:
> >
> > On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> > > On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> > > >
> > > > Linux detects what hypervior it runs on very early
> > >
> > > Not anywhere close to early enough.  We're talking for uses like kASLR.
> > >
> > Still to early to do:
> >
> >    h = cpuid(HYPERVIOR_SIGNATURE)
> >    if (h == KVMKVMKVM) {
> >       if (cpuid(kvm_features) & kvm_rnd)
> >          rdmsr(kvm_rnd)
> >    else (h == HyperV) {
> >       if (cpuid(hv_features) & hv_rnd)
> >         rdmsr(hv_rnd)
> >    else (h == XenXenXen) {
> >       if (cpuid(xen_features) & xen_rnd)
> >         rdmsr(xen_rnd)
> >   }
> >
> 
> I think that there's a lot of value in having each guest
> implementation be automatically compatible with all hypervisors.  For
> example, you forgot VMware, and VMware might be less excited about
> implementing this feature if all the guests won't immediately start
> using it.
> 
I "forgot" VMware because I do not see VMware people to be CCed. They may
be even less excited about them being told _how_ this feature need to be
implemented (e.g implement HyperV leafs for the feature detection). I
do not want to and cannot speak for VMware, but my guess is that for
them it would be much easier to add an else clause for VMware in above
"if" then to coordinate with all hypervisor developers about MSR/cpuid
details. And since this is security feature implementing it for Linux
is in their best interest.

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb@kernel.org> wrote:
> On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
>> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
>> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
>> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
>> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
>> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>> >>>>>
>> >>>>> Linux detects what hypervior it runs on very early
>> >>>>
>> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
>> >>>>
>> >>> Still to early to do:
>> >>>
>> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
>> >>>    if (h == KVMKVMKVM) {
>> >>>       if (cpuid(kvm_features) & kvm_rnd)
>> >>>          rdmsr(kvm_rnd)
>> >>>    else (h == HyperV) {
>> >>>       if (cpuid(hv_features) & hv_rnd)
>> >>>         rdmsr(hv_rnd)
>> >>>    else (h == XenXenXen) {
>> >>>       if (cpuid(xen_features) & xen_rnd)
>> >>>         rdmsr(xen_rnd)
>> >>>   }
>> >>>
>> >>
>> >> If we need to do chase loops, especially not so...
>> >>
>> > What loops exactly? As a non native English speaker I fail to understand
>> > if your answer is "yes" or "no" ;)
>> >
>>
>> The above isn't actually the full algorithm used.
>>
> What part of actually algorithm cannot be implemented? Loop that searches
> for KVM leaf in case KVM pretend to be HyperV (is this what you called
> "chase loops"?)? First of all there is no need to implement it, if KVM
> pretends to be HyperV use HyperV's way to obtain RNG, but what is the
> problem with the loop?
>

It can be implemented, and I've done it.  But it's a mess.  Almost the
very first thing we do in boot (even before decompressing the kernel)
will be to scan a bunch of cpuid leaves looking for a hypervisor with
an rng source that we can use for kASLR.  And we'll have to update
that code and make it bigger every time another hypervisor adds
exactly the same feature.

And then we have another copy of almost exactly the same code in the
normal post-boot part of the kernel.

We can certainly do this, but I'd much rather solve the problem once
and let all of the hypervisors and guests opt in and immediately be
compatible with each other.

> I "forgot" VMware because I do not see VMware people to be CCed. They may
> be even less excited about them being told _how_ this feature need to be
> implemented (e.g implement HyperV leafs for the feature detection). I
> do not want to and cannot speak for VMware, but my guess is that for
> them it would be much easier to add an else clause for VMware in above
> "if" then to coordinate with all hypervisor developers about MSR/cpuid
> details. And since this is security feature implementing it for Linux
> is in their best interest.

Do you know any of them who should be cc'd?

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb@kernel.org> wrote:
> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >> >>>>>
> >> >>>>> Linux detects what hypervior it runs on very early
> >> >>>>
> >> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
> >> >>>>
> >> >>> Still to early to do:
> >> >>>
> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >>>    if (h == KVMKVMKVM) {
> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >>>          rdmsr(kvm_rnd)
> >> >>>    else (h == HyperV) {
> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >>>         rdmsr(hv_rnd)
> >> >>>    else (h == XenXenXen) {
> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >>>         rdmsr(xen_rnd)
> >> >>>   }
> >> >>>
> >> >>
> >> >> If we need to do chase loops, especially not so...
> >> >>
> >> > What loops exactly? As a non native English speaker I fail to understand
> >> > if your answer is "yes" or "no" ;)
> >> >
> >>
> >> The above isn't actually the full algorithm used.
> >>
> > What part of actually algorithm cannot be implemented? Loop that searches
> > for KVM leaf in case KVM pretend to be HyperV (is this what you called
> > "chase loops"?)? First of all there is no need to implement it, if KVM
> > pretends to be HyperV use HyperV's way to obtain RNG, but what is the
> > problem with the loop?
> >
> 
> It can be implemented, and I've done it.  But it's a mess.  Almost the
> very first thing we do in boot (even before decompressing the kernel)
> will be to scan a bunch of cpuid leaves looking for a hypervisor with
> an rng source that we can use for kASLR.  And we'll have to update
> that code and make it bigger every time another hypervisor adds
> exactly the same feature.
IMO implementing this feature is in hypervisor's best interest, so the task
of updating the code will scale by virtue of hypervisor's developers each
adding it for hypervisor he cares about.

> 
> And then we have another copy of almost exactly the same code in the
> normal post-boot part of the kernel.
> 
> We can certainly do this, but I'd much rather solve the problem once
> and let all of the hypervisors and guests opt in and immediately be
> compatible with each other.
> 
> > I "forgot" VMware because I do not see VMware people to be CCed. They may
> > be even less excited about them being told _how_ this feature need to be
> > implemented (e.g implement HyperV leafs for the feature detection). I
> > do not want to and cannot speak for VMware, but my guess is that for
> > them it would be much easier to add an else clause for VMware in above
> > "if" then to coordinate with all hypervisor developers about MSR/cpuid
> > details. And since this is security feature implementing it for Linux
> > is in their best interest.
> 
> Do you know any of them who should be cc'd?
> 
No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may help.

But VMware is an elephant in the room here. There are other hypervisors out there.
VirtualBox, bhyve...

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

[cc: Alok Kataria at VMware]

On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb@kernel.org> wrote:
> On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
>> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb@kernel.org> wrote:
>> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
>> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
>> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
>> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
>> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
>> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>> >> >>>>>
>> >> >>>>> Linux detects what hypervior it runs on very early
>> >> >>>>
>> >> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
>> >> >>>>
>> >> >>> Still to early to do:
>> >> >>>
>> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
>> >> >>>    if (h == KVMKVMKVM) {
>> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
>> >> >>>          rdmsr(kvm_rnd)
>> >> >>>    else (h == HyperV) {
>> >> >>>       if (cpuid(hv_features) & hv_rnd)
>> >> >>>         rdmsr(hv_rnd)
>> >> >>>    else (h == XenXenXen) {
>> >> >>>       if (cpuid(xen_features) & xen_rnd)
>> >> >>>         rdmsr(xen_rnd)
>> >> >>>   }
>> >> >>>
>> >> >>
>> >> >> If we need to do chase loops, especially not so...
>> >> >>
>> >> > What loops exactly? As a non native English speaker I fail to understand
>> >> > if your answer is "yes" or "no" ;)
>> >> >
>> >>
>> >> The above isn't actually the full algorithm used.
>> >>
>> > What part of actually algorithm cannot be implemented? Loop that searches
>> > for KVM leaf in case KVM pretend to be HyperV (is this what you called
>> > "chase loops"?)? First of all there is no need to implement it, if KVM
>> > pretends to be HyperV use HyperV's way to obtain RNG, but what is the
>> > problem with the loop?
>> >
>>
>> It can be implemented, and I've done it.  But it's a mess.  Almost the
>> very first thing we do in boot (even before decompressing the kernel)
>> will be to scan a bunch of cpuid leaves looking for a hypervisor with
>> an rng source that we can use for kASLR.  And we'll have to update
>> that code and make it bigger every time another hypervisor adds
>> exactly the same feature.
> IMO implementing this feature is in hypervisor's best interest, so the task
> of updating the code will scale by virtue of hypervisor's developers each
> adding it for hypervisor he cares about.

I assume that you mean guest, not hypervisor.

>
>>
>> And then we have another copy of almost exactly the same code in the
>> normal post-boot part of the kernel.
>>
>> We can certainly do this, but I'd much rather solve the problem once
>> and let all of the hypervisors and guests opt in and immediately be
>> compatible with each other.
>>
>> > I "forgot" VMware because I do not see VMware people to be CCed. They may
>> > be even less excited about them being told _how_ this feature need to be
>> > implemented (e.g implement HyperV leafs for the feature detection). I
>> > do not want to and cannot speak for VMware, but my guess is that for
>> > them it would be much easier to add an else clause for VMware in above
>> > "if" then to coordinate with all hypervisor developers about MSR/cpuid
>> > details. And since this is security feature implementing it for Linux
>> > is in their best interest.
>>
>> Do you know any of them who should be cc'd?
>>
> No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may help.
>
> But VMware is an elephant in the room here. There are other hypervisors out there.
> VirtualBox, bhyve...

Exactly.  The amount of effort to get everything to be compatible with
everything scales quadratically in the number of hypervisors, and the
probability that some combination is broken also increases.

If we can get everyone to back something common here then this problem
goes away.

--Andy

>
> --
>                         Gleb.



-- 
Andy Lutomirski
AMA Capital Management, LLC

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Christopher Covington]

On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
> Hi all-
> 
> I would like to standardize on a very simple protocol by which a guest
> OS can obtain an RNG seed early in boot.
> 
> The main design requirements are:
> 
>  - The interface should be very easy to use.  Linux, at least, will
> want to use it extremely early in boot as part of kernel ASLR.  This
> means that PCI and ACPI will not work.

How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
Could hypervisors and simulators simply make sure these work?

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 11:30 AM, Christopher Covington
<cov@codeaurora.org> wrote:
> On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
>> Hi all-
>>
>> I would like to standardize on a very simple protocol by which a guest
>> OS can obtain an RNG seed early in boot.
>>
>> The main design requirements are:
>>
>>  - The interface should be very easy to use.  Linux, at least, will
>> want to use it extremely early in boot as part of kernel ASLR.  This
>> means that PCI and ACPI will not work.
>
> How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
> Could hypervisors and simulators simply make sure these work?
>

If RDRAND is available, then Linux, at least, will use it.  The rest
are too complicated for early use.  Linux on x86 plays some vaguely
clever games with rdtsc and poking at the i8254 port.

I think that these tricks are even less useful as a guest than they
are on metal, and we can use paravirt mechanisms to make guest early
boot rngs much stronger.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nadav Amit]

On Sep 19, 2014, at 9:42 PM, Andy Lutomirski <luto@amacapital.net> wrote:

> On Fri, Sep 19, 2014 at 11:30 AM, Christopher Covington
> <cov@codeaurora.org> wrote:
>> On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
>>> Hi all-
>>> 
>>> I would like to standardize on a very simple protocol by which a guest
>>> OS can obtain an RNG seed early in boot.
>>> 
>>> The main design requirements are:
>>> 
>>> - The interface should be very easy to use.  Linux, at least, will
>>> want to use it extremely early in boot as part of kernel ASLR.  This
>>> means that PCI and ACPI will not work.
>> 
>> How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
>> Could hypervisors and simulators simply make sure these work?
>> 
> 
> If RDRAND is available, then Linux, at least, will use it.  The rest
> are too complicated for early use.  Linux on x86 plays some vaguely
> clever games with rdtsc and poking at the i8254 port.
> 
> I think that these tricks are even less useful as a guest than they
> are on metal, and we can use paravirt mechanisms to make guest early
> boot rngs much stronger.

Sorry for interrupting, as I understand the discussion tries to be generic.

However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.

Nadav

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nadav Amit]

On Sep 19, 2014, at 9:42 PM, Andy Lutomirski <luto@amacapital.net> wrote:

> On Fri, Sep 19, 2014 at 11:30 AM, Christopher Covington
> <cov@codeaurora.org> wrote:
>> On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
>>> Hi all-
>>> 
>>> I would like to standardize on a very simple protocol by which a guest
>>> OS can obtain an RNG seed early in boot.
>>> 
>>> The main design requirements are:
>>> 
>>> - The interface should be very easy to use.  Linux, at least, will
>>> want to use it extremely early in boot as part of kernel ASLR.  This
>>> means that PCI and ACPI will not work.
>> 
>> How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
>> Could hypervisors and simulators simply make sure these work?
>> 
> 
> If RDRAND is available, then Linux, at least, will use it.  The rest
> are too complicated for early use.  Linux on x86 plays some vaguely
> clever games with rdtsc and poking at the i8254 port.
> 
> I think that these tricks are even less useful as a guest than they
> are on metal, and we can use paravirt mechanisms to make guest early
> boot rngs much stronger.

Sorry for interrupting, as I understand the discussion tries to be generic.

However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.

Nadav

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 1:21 PM, Nadav Amit <nadav.amit@gmail.com> wrote:
>
> On Sep 19, 2014, at 9:42 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>
>> On Fri, Sep 19, 2014 at 11:30 AM, Christopher Covington
>> <cov@codeaurora.org> wrote:
>>> On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
>>>> Hi all-
>>>>
>>>> I would like to standardize on a very simple protocol by which a guest
>>>> OS can obtain an RNG seed early in boot.
>>>>
>>>> The main design requirements are:
>>>>
>>>> - The interface should be very easy to use.  Linux, at least, will
>>>> want to use it extremely early in boot as part of kernel ASLR.  This
>>>> means that PCI and ACPI will not work.
>>>
>>> How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
>>> Could hypervisors and simulators simply make sure these work?
>>>
>>
>> If RDRAND is available, then Linux, at least, will use it.  The rest
>> are too complicated for early use.  Linux on x86 plays some vaguely
>> clever games with rdtsc and poking at the i8254 port.
>>
>> I think that these tricks are even less useful as a guest than they
>> are on metal, and we can use paravirt mechanisms to make guest early
>> boot rngs much stronger.
>
> Sorry for interrupting, as I understand the discussion tries to be generic.
>
> However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.

Because then guest user code will think that rdrand is there and will
try to use it, resulting in abysmal performance.

--Andy

>
> Nadav



-- 
Andy Lutomirski
AMA Capital Management, LLC
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Gleb Natapov]

On Fri, Sep 19, 2014 at 11:20:49AM -0700, Andy Lutomirski wrote:
> [cc: Alok Kataria at VMware]
> 
> On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb@kernel.org> wrote:
> > On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
> >> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb@kernel.org> wrote:
> >> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
> >> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
> >> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> >> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >> >> >>>>>
> >> >> >>>>> Linux detects what hypervior it runs on very early
> >> >> >>>>
> >> >> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
> >> >> >>>>
> >> >> >>> Still to early to do:
> >> >> >>>
> >> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >> >>>    if (h == KVMKVMKVM) {
> >> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >> >>>          rdmsr(kvm_rnd)
> >> >> >>>    else (h == HyperV) {
> >> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >> >>>         rdmsr(hv_rnd)
> >> >> >>>    else (h == XenXenXen) {
> >> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >> >>>         rdmsr(xen_rnd)
> >> >> >>>   }
> >> >> >>>
> >> >> >>
> >> >> >> If we need to do chase loops, especially not so...
> >> >> >>
> >> >> > What loops exactly? As a non native English speaker I fail to understand
> >> >> > if your answer is "yes" or "no" ;)
> >> >> >
> >> >>
> >> >> The above isn't actually the full algorithm used.
> >> >>
> >> > What part of actually algorithm cannot be implemented? Loop that searches
> >> > for KVM leaf in case KVM pretend to be HyperV (is this what you called
> >> > "chase loops"?)? First of all there is no need to implement it, if KVM
> >> > pretends to be HyperV use HyperV's way to obtain RNG, but what is the
> >> > problem with the loop?
> >> >
> >>
> >> It can be implemented, and I've done it.  But it's a mess.  Almost the
> >> very first thing we do in boot (even before decompressing the kernel)
> >> will be to scan a bunch of cpuid leaves looking for a hypervisor with
> >> an rng source that we can use for kASLR.  And we'll have to update
> >> that code and make it bigger every time another hypervisor adds
> >> exactly the same feature.
> > IMO implementing this feature is in hypervisor's best interest, so the task
> > of updating the code will scale by virtue of hypervisor's developers each
> > adding it for hypervisor he cares about.
> 
> I assume that you mean guest, not hypervisor.
> 
Yes, I mean guest support for hypervisor he cares about.

> >
> >>
> >> And then we have another copy of almost exactly the same code in the
> >> normal post-boot part of the kernel.
> >>
> >> We can certainly do this, but I'd much rather solve the problem once
> >> and let all of the hypervisors and guests opt in and immediately be
> >> compatible with each other.
> >>
> >> > I "forgot" VMware because I do not see VMware people to be CCed. They may
> >> > be even less excited about them being told _how_ this feature need to be
> >> > implemented (e.g implement HyperV leafs for the feature detection). I
> >> > do not want to and cannot speak for VMware, but my guess is that for
> >> > them it would be much easier to add an else clause for VMware in above
> >> > "if" then to coordinate with all hypervisor developers about MSR/cpuid
> >> > details. And since this is security feature implementing it for Linux
> >> > is in their best interest.
> >>
> >> Do you know any of them who should be cc'd?
> >>
> > No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may help.
> >
> > But VMware is an elephant in the room here. There are other hypervisors out there.
> > VirtualBox, bhyve...
> 
> Exactly.  The amount of effort to get everything to be compatible with
> everything scales quadratically in the number of hypervisors, and the
> probability that some combination is broken also increases.
> 
The effort is distributed equally among hypervisor developers. If they
want Linux to be more secure on their hypervisor they contribute guest
code. They do need to write hypervisor part anyway. On cpus with RDRAND
instruction this MSR is not even needed and some hypervisors may decide
that support for old cpus does not worth the effort. Unified interface
does not help if hypervisor does not implement it.

--
			Gleb.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 01:46 PM, Andy Lutomirski wrote:
>>
>> However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.
> 
> Because then guest user code will think that rdrand is there and will
> try to use it, resulting in abysmal performance.
> 

Yes, the presence of RDRAND implies a cheap and inexhaustible entropy
source.

	-hpa


_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Theodore Ts'o]

On Fri, Sep 19, 2014 at 09:40:42AM -0700, H. Peter Anvin wrote:
> 
> There is a huge disadvantage to the fact that CPUID is a user space
> instruction, though.

But if the goal is to provide something like getrandom(2) direct from
the Host OS, it's not necessarily harmful to allow the Guest ring 3
code to be able to fetch randomness in that way.  The hypervisor can
implement rate limiting to protect against the guest using this too
frequently, but this is something that you should be doing for guest
ring 0 code anyway, since from the POV of the hypervisor Guest ring 0
is not necessarily any more trusted than Guest ring 3.

       		       	    	    	       - Ted

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 3:05 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Sep 19, 2014 at 09:40:42AM -0700, H. Peter Anvin wrote:
>>
>> There is a huge disadvantage to the fact that CPUID is a user space
>> instruction, though.
>
> But if the goal is to provide something like getrandom(2) direct from
> the Host OS, it's not necessarily harmful to allow the Guest ring 3
> code to be able to fetch randomness in that way.  The hypervisor can
> implement rate limiting to protect against the guest using this too
> frequently, but this is something that you should be doing for guest
> ring 0 code anyway, since from the POV of the hypervisor Guest ring 0
> is not necessarily any more trusted than Guest ring 3.

On the other hand, the guest kernel might not want the guest ring 3 to
be able to get random numbers.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Nakajima, Jun]

On Fri, Sep 19, 2014 at 3:06 PM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Fri, Sep 19, 2014 at 3:05 PM, Theodore Ts'o <tytso@mit.edu> wrote:
>> On Fri, Sep 19, 2014 at 09:40:42AM -0700, H. Peter Anvin wrote:
>>>
>>> There is a huge disadvantage to the fact that CPUID is a user space
>>> instruction, though.
>>
>> But if the goal is to provide something like getrandom(2) direct from
>> the Host OS, it's not necessarily harmful to allow the Guest ring 3
>> code to be able to fetch randomness in that way.  The hypervisor can
>> implement rate limiting to protect against the guest using this too
>> frequently, but this is something that you should be doing for guest
>> ring 0 code anyway, since from the POV of the hypervisor Guest ring 0
>> is not necessarily any more trusted than Guest ring 3.
>
> On the other hand, the guest kernel might not want the guest ring 3 to
> be able to get random numbers.
>

But the RDSEED instruction, for example, is available in user-level.
And I'm not sure that the kernel can do something with that.

-- 
Jun
Intel Open Source Technology Center

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Theodore Ts'o]

On Fri, Sep 19, 2014 at 03:06:55PM -0700, Andy Lutomirski wrote:
> On Fri, Sep 19, 2014 at 3:05 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> > On Fri, Sep 19, 2014 at 09:40:42AM -0700, H. Peter Anvin wrote:
> >>
> >> There is a huge disadvantage to the fact that CPUID is a user space
> >> instruction, though.
> >
> > But if the goal is to provide something like getrandom(2) direct from
> > the Host OS, it's not necessarily harmful to allow the Guest ring 3
> > code to be able to fetch randomness in that way.  The hypervisor can
> > implement rate limiting to protect against the guest using this too
> > frequently, but this is something that you should be doing for guest
> > ring 0 code anyway, since from the POV of the hypervisor Guest ring 0
> > is not necessarily any more trusted than Guest ring 3.
> 
> On the other hand, the guest kernel might not want the guest ring 3 to
> be able to get random numbers.

Um, why?

We're talking about using this to seed the RNG, and not something that
the guest kernel would be using continuously.  So what's the problem
with letting the guest ring get random numbers from the host?

     	     	       	    	       	       - Ted

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 3:57 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Sep 19, 2014 at 03:06:55PM -0700, Andy Lutomirski wrote:
>> On Fri, Sep 19, 2014 at 3:05 PM, Theodore Ts'o <tytso@mit.edu> wrote:
>> > On Fri, Sep 19, 2014 at 09:40:42AM -0700, H. Peter Anvin wrote:
>> >>
>> >> There is a huge disadvantage to the fact that CPUID is a user space
>> >> instruction, though.
>> >
>> > But if the goal is to provide something like getrandom(2) direct from
>> > the Host OS, it's not necessarily harmful to allow the Guest ring 3
>> > code to be able to fetch randomness in that way.  The hypervisor can
>> > implement rate limiting to protect against the guest using this too
>> > frequently, but this is something that you should be doing for guest
>> > ring 0 code anyway, since from the POV of the hypervisor Guest ring 0
>> > is not necessarily any more trusted than Guest ring 3.
>>
>> On the other hand, the guest kernel might not want the guest ring 3 to
>> be able to get random numbers.
>
> Um, why?

To force deterministic execution.

I incorrectly thought that the kernel could switch RDRAND on and off.
It turns out that a hypervisor can do this, but not the kernel.  Also,
determinism is lost anyway because of TSX, which *also* can't be
turned on and off.

>
> We're talking about using this to seed the RNG, and not something that
> the guest kernel would be using continuously.  So what's the problem
> with letting the guest ring get random numbers from the host?

I object to preventing guest kernels from limiting the privileges of
their own userspace.  Letting guest CPL3 do this is essentially
setting guest policy in the hypervisor, which I dislike if we can
avoid it.

Admittedly, in this case, control of RNG availability in guest
userspace may be a lost cause regardless.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 04:12 PM, Andy Lutomirski wrote:
> 
> To force deterministic execution.
> 
> I incorrectly thought that the kernel could switch RDRAND on and off.
> It turns out that a hypervisor can do this, but not the kernel.  Also,
> determinism is lost anyway because of TSX, which *also* can't be
> turned on and off.
> 

Actually, a much bigger reason is because it lets rogue guest *user
space*, even will a well-behaved guest OS, do something potentially
harmful to the host.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 04:12 PM, Andy Lutomirski wrote:
> 
> To force deterministic execution.
> 
> I incorrectly thought that the kernel could switch RDRAND on and off.
> It turns out that a hypervisor can do this, but not the kernel.  Also,
> determinism is lost anyway because of TSX, which *also* can't be
> turned on and off.
> 

Actually, a much bigger reason is because it lets rogue guest *user
space*, even will a well-behaved guest OS, do something potentially
harmful to the host.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Theodore Ts'o]

On Fri, Sep 19, 2014 at 04:29:53PM -0700, H. Peter Anvin wrote:
> 
> Actually, a much bigger reason is because it lets rogue guest *user
> space*, even will a well-behaved guest OS, do something potentially
> harmful to the host.

Right, but if the host kernel is dependent on the guest OS for
security, the game is over.  The Guest Kernel must NEVER been able to
do anything harmful to the host.  If it can, it is a severe security
bug in KVM that must be fixed ASAP.

						- Ted

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Andy Lutomirski]

On Fri, Sep 19, 2014 at 4:35 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Fri, Sep 19, 2014 at 04:29:53PM -0700, H. Peter Anvin wrote:
>>
>> Actually, a much bigger reason is because it lets rogue guest *user
>> space*, even will a well-behaved guest OS, do something potentially
>> harmful to the host.
>
> Right, but if the host kernel is dependent on the guest OS for
> security, the game is over.  The Guest Kernel must NEVER been able to
> do anything harmful to the host.  If it can, it is a severe security
> bug in KVM that must be fixed ASAP.

Nonetheless, I suspect that some OS kernel author, somewhere, will
object to having a hypervisor that exposes new capabilities to guest
CPL 3 without requiring the guest to opt in, if for no other reason
than that it slightly increases the attack surface.

I certainly object on these grounds.

--Andy

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/19/2014 04:35 PM, Theodore Ts'o wrote:
> On Fri, Sep 19, 2014 at 04:29:53PM -0700, H. Peter Anvin wrote:
>>
>> Actually, a much bigger reason is because it lets rogue guest *user
>> space*, even will a well-behaved guest OS, do something potentially
>> harmful to the host.
> 
> Right, but if the host kernel is dependent on the guest OS for
> security, the game is over.  The Guest Kernel must NEVER been able to
> do anything harmful to the host.  If it can, it is a severe security
> bug in KVM that must be fixed ASAP.
> 

"Security" and "resource well-behaved" are two different things.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Paolo Bonzini]

Il 19/09/2014 22:46, Andy Lutomirski ha scritto:
>> >
>> > However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.
> Because then guest user code will think that rdrand is there and will
> try to use it, resulting in abysmal performance.

KVM could expose a CPUID leaf that says "RDRAND is not there, but if you
execute it the hypervisor will try to do something slow but sane".

Paolo
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Alok Kataria]

Hi Andy,

On Fri, 2014-09-19 at 11:20 -0700, Andy Lutomirski wrote:
> [cc: Alok Kataria at VMware]
> 
> On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb@kernel.org> wrote:
> > On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
> >> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb@kernel.org> wrote:
> >> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
> >> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin wrote:
> >> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter Anvin wrote:
> >> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >> >> >>>>>
> >> >> >>>>> Linux detects what hypervior it runs on very early
> >> >> >>>>
> >> >> >>>> Not anywhere close to early enough.  We're talking for uses like kASLR.
> >> >> >>>>
> >> >> >>> Still to early to do:
> >> >> >>>
> >> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >> >>>    if (h == KVMKVMKVM) {
> >> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >> >>>          rdmsr(kvm_rnd)
> >> >> >>>    else (h == HyperV) {
> >> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >> >>>         rdmsr(hv_rnd)
> >> >> >>>    else (h == XenXenXen) {
> >> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >> >>>         rdmsr(xen_rnd)
> >> >> >>>   }
> >> >> >>>
> >> >> >>
> >> >> >> If we need to do chase loops, especially not so...
> >> >> >>
> >> >> > What loops exactly? As a non native English speaker I fail to understand
> >> >> > if your answer is "yes" or "no" ;)
> >> >> >
> >> >>
> >> >> The above isn't actually the full algorithm used.
> >> >>
> >> > What part of actually algorithm cannot be implemented? Loop that searches
> >> > for KVM leaf in case KVM pretend to be HyperV (is this what you called
> >> > "chase loops"?)? First of all there is no need to implement it, if KVM
> >> > pretends to be HyperV use HyperV's way to obtain RNG, but what is the
> >> > problem with the loop?
> >> >
> >>
> >> It can be implemented, and I've done it.  But it's a mess.  Almost the
> >> very first thing we do in boot (even before decompressing the kernel)
> >> will be to scan a bunch of cpuid leaves looking for a hypervisor with
> >> an rng source that we can use for kASLR.  And we'll have to update
> >> that code and make it bigger every time another hypervisor adds
> >> exactly the same feature.
> > IMO implementing this feature is in hypervisor's best interest, so the task
> > of updating the code will scale by virtue of hypervisor's developers each
> > adding it for hypervisor he cares about.
> 
> I assume that you mean guest, not hypervisor.
> 
> >
> >>
> >> And then we have another copy of almost exactly the same code in the
> >> normal post-boot part of the kernel.
> >>
> >> We can certainly do this, but I'd much rather solve the problem once
> >> and let all of the hypervisors and guests opt in and immediately be
> >> compatible with each other.
> >>
> >> > I "forgot" VMware because I do not see VMware people to be CCed. They may
> >> > be even less excited about them being told _how_ this feature need to be
> >> > implemented (e.g implement HyperV leafs for the feature detection). I
> >> > do not want to and cannot speak for VMware, but my guess is that for
> >> > them it would be much easier to add an else clause for VMware in above
> >> > "if" then to coordinate with all hypervisor developers about MSR/cpuid
> >> > details. And since this is security feature implementing it for Linux
> >> > is in their best interest.
> >>
> >> Do you know any of them who should be cc'd?
> >>
> > No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may help.
> >
> > But VMware is an elephant in the room here. There are other hypervisors out there.
> > VirtualBox, bhyve...
> 
> Exactly.  The amount of effort to get everything to be compatible with
> everything scales quadratically in the number of hypervisors, and the
> probability that some combination is broken also increases.
> 
> If we can get everyone to back something common here then this problem
> goes away.

There was a similar attempt few years back [1], to standardize on the
hypervisor cpuid space. Though a few of them were interested, getting
all hypervisor vendors to agree (actually even discuss this) turned out
to be a futile exercise. Don't mean to discourage you, but what I
learned from that attempt was that it's very difficult to standardize
unless the hardware vendors are proposing it.

In anycase can you point me to a mail which discusses the specifics of
the interface you are proposing ? 

Alok

[1] - http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22643
      https://lkml.org/lkml/2008/9/26/351

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Christopher Covington]

On 09/19/2014 05:46 PM, H. Peter Anvin wrote:
> On 09/19/2014 01:46 PM, Andy Lutomirski wrote:
>>>
>>> However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.
>>
>> Because then guest user code will think that rdrand is there and will
>> try to use it, resulting in abysmal performance.
>>
> 
> Yes, the presence of RDRAND implies a cheap and inexhaustible entropy
> source.

A guest kernel couldn't make it look like RDRAND is not present to guest
userspace?

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[Christopher Covington]

On 09/19/2014 02:42 PM, Andy Lutomirski wrote:
> On Fri, Sep 19, 2014 at 11:30 AM, Christopher Covington
> <cov@codeaurora.org> wrote:
>> On 09/17/2014 10:50 PM, Andy Lutomirski wrote:
>>> Hi all-
>>>
>>> I would like to standardize on a very simple protocol by which a guest
>>> OS can obtain an RNG seed early in boot.
>>>
>>> The main design requirements are:
>>>
>>>  - The interface should be very easy to use.  Linux, at least, will
>>> want to use it extremely early in boot as part of kernel ASLR.  This
>>> means that PCI and ACPI will not work.
>>
>> How do non-virtual systems get entropy this early? RDRAND/Padlock? Truerand?
>> Could hypervisors and simulators simply make sure these work?
>>
> 
> If RDRAND is available, then Linux, at least, will use it.  The rest
> are too complicated for early use.  Linux on x86 plays some vaguely
> clever games with rdtsc and poking at the i8254 port.

I just wanted to check that it couldn't be as simple as giving one or both of
the timers random initial values.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/22/2014 06:31 AM, Christopher Covington wrote:
> On 09/19/2014 05:46 PM, H. Peter Anvin wrote:
>> On 09/19/2014 01:46 PM, Andy Lutomirski wrote:
>>>>
>>>> However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.
>>>
>>> Because then guest user code will think that rdrand is there and will
>>> try to use it, resulting in abysmal performance.
>>>
>>
>> Yes, the presence of RDRAND implies a cheap and inexhaustible entropy
>> source.
> 
> A guest kernel couldn't make it look like RDRAND is not present to guest
> userspace?
> 

It could, but how would you enumerate that?  A new "RDRAND-CPL-0" CPUID
bit pretty much would be required.

	-hpa


_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

On 09/22/2014 07:17 AM, H. Peter Anvin wrote:
> 
> It could, but how would you enumerate that?  A new "RDRAND-CPL-0" CPUID
> bit pretty much would be required.
> 

Note that there are two things that differ: the CPL 0-ness and the
performance/exhaustibility attributes.

	-hpa

Re: Standardizing an MSR or other hypercall to get an RNG seed?

[H. Peter Anvin]

Not really, no.

Sent from my tablet, pardon any formatting problems.

> On Sep 22, 2014, at 06:31, Christopher Covington <cov@codeaurora.org> wrote:
> 
>> On 09/19/2014 05:46 PM, H. Peter Anvin wrote:
>> On 09/19/2014 01:46 PM, Andy Lutomirski wrote:
>>>> 
>>>> However, it sounds to me that at least for KVM, it is very easy just to emulate the RDRAND instruction. The hypervisor would report to the guest that RDRAND is supported in CPUID and the emulate the instruction when guest executes it. KVM already traps guest #UD (which would occur if RDRAND executed while it is not supported) - so this scheme wouldn’t introduce additional overhead over RDMSR.
>>> 
>>> Because then guest user code will think that rdrand is there and will
>>> try to use it, resulting in abysmal performance.
>> 
>> Yes, the presence of RDRAND implies a cheap and inexhaustible entropy
>> source.
> 
> A guest kernel couldn't make it look like RDRAND is not present to guest
> userspace?
> 
> Christopher
> 
> -- 
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by the Linux Foundation.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization