[PATCH 1/2] x86/random: Retry on RDSEED failure

[PATCH 1/2] x86/random: Retry on RDSEED failure

[Kirill A. Shutemov]

The function rdrand_long() retries 10 times before returning failure to
the caller. On the other hand, rdseed_long() gives up on the first
failure.

According to the Intel SDM, both instructions should follow the same
retry approach. This information can be found in the section titled
"Random Number Generator Instructions".

To align the behavior of rdseed_long() with rdrand_long(), it should be
modified to retry 10 times before giving up.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/archrandom.h | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
index 02bae8e0758b..918c5880de9e 100644
--- a/arch/x86/include/asm/archrandom.h
+++ b/arch/x86/include/asm/archrandom.h
@@ -33,11 +33,19 @@ static inline bool __must_check rdrand_long(unsigned long *v)
 
 static inline bool __must_check rdseed_long(unsigned long *v)
 {
+	unsigned int retry = RDRAND_RETRY_LOOPS;
 	bool ok;
-	asm volatile("rdseed %[out]"
-		     CC_SET(c)
-		     : CC_OUT(c) (ok), [out] "=r" (*v));
-	return ok;
+
+	do {
+		asm volatile("rdseed %[out]"
+			     CC_SET(c)
+			     : CC_OUT(c) (ok), [out] "=r" (*v));
+
+		if (ok)
+			return true;
+	} while (--retry);
+
+	return false;
 }
 
 /*
-- 
2.43.0

[PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Kirill A. Shutemov]

RDRAND and RDSEED instructions rarely fail. Ten retries should be
sufficient to account for occasional failures.

If the instruction fails more than ten times, it is likely that the
hardware is broken or someone is attempting to exceed the rate at which
the random number generator hardware can provide random numbers.

Issue a warning if ten retries were not enough.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 arch/x86/include/asm/archrandom.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
index 918c5880de9e..fc8d837fb3b9 100644
--- a/arch/x86/include/asm/archrandom.h
+++ b/arch/x86/include/asm/archrandom.h
@@ -13,6 +13,12 @@
 #include <asm/processor.h>
 #include <asm/cpufeature.h>
 
+#ifdef KASLR_COMPRESSED_BOOT
+#define rd_warn(msg) warn(msg)
+#else
+#define rd_warn(msg) WARN_ONCE(1, msg)
+#endif
+
 #define RDRAND_RETRY_LOOPS	10
 
 /* Unconditional execution of RDRAND and RDSEED */
@@ -28,6 +34,9 @@ static inline bool __must_check rdrand_long(unsigned long *v)
 		if (ok)
 			return true;
 	} while (--retry);
+
+	rd_warn("RDRAND failed\n");
+
 	return false;
 }
 
@@ -45,6 +54,8 @@ static inline bool __must_check rdseed_long(unsigned long *v)
 			return true;
 	} while (--retry);
 
+	rd_warn("RDSEED failed\n");
+
 	return false;
 }
 
-- 
2.43.0

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

Hi Kirill,

Picking up from my last email on patch 1/2:

On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
<kirill.shutemov@linux.intel.com> wrote:
> RDRAND and RDSEED instructions rarely fail. Ten retries should be
> sufficient to account for occasional failures.
>
> If the instruction fails more than ten times, it is likely that the
> hardware is broken or someone is attempting to exceed the rate at which
> the random number generator hardware can provide random numbers.

You're the Intel employee so you can find out about this with much
more assurance than me, but I understand the sentence above to be _way
more_ true for RDRAND than for RDSEED. If your informed opinion is,
"RDRAND failing can only be due to totally broken hardware" then a
WARN_ON seems like an appropriate solution, consistent with what other
drivers do for totally broken hardware. I'm less convinced that this
is the case also for RDSEED, but you know better than me.

However, there's one potentially concerning aspect to consider: if the
statement is "RDRAND only fails when the hardware fails", that's fine,
but if the statement is "RDRAND only fails when the hardware fails or
a user hammers on RDRAND in a busy loop," then this seems like a
potential DoS vector from userspace, since RDRAND is not a privileged
instruction. Unless there's different pools and rate limiting and
hardware and such depending on which ring the instruction is called
from? But I've never read about that. What's your feeling on this
concern?

And if the DoS thing _is_ a concern, and the use case for this WARN_ON
in the first place is the trusted computing scenario, so we basically
only care about early boot, then one addendum would be to only warn if
we're in early boot, which would work because seeding via RDRAND is
attempted pretty early on in init.c.

Jason

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> Hi Kirill,
> 
> Picking up from my last email on patch 1/2:
> 
> On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
> <kirill.shutemov@linux.intel.com> wrote:
> > RDRAND and RDSEED instructions rarely fail. Ten retries should be
> > sufficient to account for occasional failures.
> >
> > If the instruction fails more than ten times, it is likely that the
> > hardware is broken or someone is attempting to exceed the rate at which
> > the random number generator hardware can provide random numbers.
> 
> You're the Intel employee so you can find out about this with much
> more assurance than me, but I understand the sentence above to be _way
> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> "RDRAND failing can only be due to totally broken hardware"

No, this is not the case per Intel SDM. I think we can live under a simple
assumption that both of these instructions can fail not just due to broken
HW, but also due to enough pressure put into the whole DRBG construction
that supplies random numbers via RDRAND/RDSEED. 

 then a
> WARN_ON seems like an appropriate solution, consistent with what other
> drivers do for totally broken hardware. I'm less convinced that this
> is the case also for RDSEED, but you know better than me.

I do agree that due to internal structure of DRBG it is easier to create
a situation when RDSEED will fail. But for the purpose of Linux RNG and 
confidential computing it actually doesn’t make a difference if we get an
output from RDRAND or RDSEED, as soon as we get either of them. 
Problems only start imo when both of them are made to fail. 

> 
> However, there's one potentially concerning aspect to consider: if the
> statement is "RDRAND only fails when the hardware fails", that's fine,
> but if the statement is "RDRAND only fails when the hardware fails or
> a user hammers on RDRAND in a busy loop," then this seems like a
> potential DoS vector from userspace, since RDRAND is not a privileged
> instruction. Unless there's different pools and rate limiting and
> hardware and such depending on which ring the instruction is called
> from? But I've never read about that. What's your feeling on this
> concern?

RDRAND can fail with enough load as I already said above. 
I am also not aware about any ring separation or anything like this for
RDRAND/RDSEED instructions. 

I guess your concern about DoS here is for the case when we don’t
trust the host/VMM *and* assume malicious userspace, correct? 
Because in non-confidential computing case, the Linux RNG in such
case will just use non-RDRAND fallbacks, no DoS will happen and we
should have enough entropy that is outside of userspace control.

I guess this is indeed difficult situation because we don’t have any other
entropy sources anymore (unless assume some special HW). 
But you bring a very valid point that in this case we make it easier for userspace
to make a DoS to the kernel if we require RDRAND/RDSEED to 
succeed, which is not acceptable (with exception of early boot when
we don’t have the userspace problem). 

> 
> And if the DoS thing _is_ a concern, and the use case for this WARN_ON
> in the first place is the trusted computing scenario, so we basically
> only care about early boot, then one addendum would be to only warn if
> we're in early boot, which would work because seeding via RDRAND is
> attempted pretty early on in init.c.

I don’t think we are only concerned with initial early boot and initial seeding. 
What about periodic reseeding of ChaCha CSPRNG? If you don’t get
RDRAND/RDSEED output during this step, don’t we formally loose the forward
prediction resistance property of Linux RNG assuming this is the only source
of entropy that is outside of attacker control? 

Best Regards,
Elena. 

> 
> Jason

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 2:45 PM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
> No, this is not the case per Intel SDM. I think we can live under a simple
> assumption that both of these instructions can fail not just due to broken
> HW, but also due to enough pressure put into the whole DRBG construction
> that supplies random numbers via RDRAND/RDSEED.

Yea, thought so.

> I guess your concern about DoS here is for the case when we don’t
> trust the host/VMM *and* assume malicious userspace, correct?
> Because in non-confidential computing case, the Linux RNG in such
> case will just use non-RDRAND fallbacks, no DoS will happen and we
> should have enough entropy that is outside of userspace control.

Don't think about the RNG for just one second. The basic principle is
simpler: if you have a
`WARN_ON(unprivd_userspace_triggerable_condition)`, that's usually
considered a DoS - panic_on_warn and such.

> >
> > And if the DoS thing _is_ a concern, and the use case for this WARN_ON
> > in the first place is the trusted computing scenario, so we basically
> > only care about early boot, then one addendum would be to only warn if
> > we're in early boot, which would work because seeding via RDRAND is
> > attempted pretty early on in init.c.
>
> I don’t think we are only concerned with initial early boot and initial seeding.
> What about periodic reseeding of ChaCha CSPRNG? If you don’t get
> RDRAND/RDSEED output during this step, don’t we formally loose the forward
> prediction resistance property of Linux RNG assuming this is the only source
> of entropy that is outside of attacker control?

If you never add new material, and you have the initial seed, then
it's deterministic. But you still mostly can't backtrack if the state
leaks at some future point in time.

Jason

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> On Tue, Jan 30, 2024 at 2:45 PM Reshetova, Elena
> <elena.reshetova@intel.com> wrote:
> > No, this is not the case per Intel SDM. I think we can live under a simple
> > assumption that both of these instructions can fail not just due to broken
> > HW, but also due to enough pressure put into the whole DRBG construction
> > that supplies random numbers via RDRAND/RDSEED.
> 
> Yea, thought so.
> 
> > I guess your concern about DoS here is for the case when we don’t
> > trust the host/VMM *and* assume malicious userspace, correct?
> > Because in non-confidential computing case, the Linux RNG in such
> > case will just use non-RDRAND fallbacks, no DoS will happen and we
> > should have enough entropy that is outside of userspace control.
> 
> Don't think about the RNG for just one second. The basic principle is
> simpler: if you have a
> `WARN_ON(unprivd_userspace_triggerable_condition)`, that's usually
> considered a DoS - panic_on_warn and such.

Ok, agree, you do bring a valid point that we should not create new 
DoS attack vectors from  userspace in such cases. 

> 
> > >
> > > And if the DoS thing _is_ a concern, and the use case for this WARN_ON
> > > in the first place is the trusted computing scenario, so we basically
> > > only care about early boot, then one addendum would be to only warn if
> > > we're in early boot, which would work because seeding via RDRAND is
> > > attempted pretty early on in init.c.
> >
> > I don’t think we are only concerned with initial early boot and initial seeding.
> > What about periodic reseeding of ChaCha CSPRNG? If you don’t get
> > RDRAND/RDSEED output during this step, don’t we formally loose the forward
> > prediction resistance property of Linux RNG assuming this is the only source
> > of entropy that is outside of attacker control?
> 
> If you never add new material, and you have the initial seed, then
> it's deterministic. But you still mostly can't backtrack if the state
> leaks at some future point in time.

I am not talking about backtrack resistance, i.e. when attacker learns about 
RNG state and then can recover the past output. I was talking about an attacker
learning the RNG state at some point of time (RNG compromise) and
then for RNG being able to recover over time from this state to a secure state using 
fresh entropy input that is outside of attacker control/observance.
Does Linux RNG aim to provide this property? Do people care about this? 
If noone cares about this one and Linux RNG doesn’t aim to provide it anyhow,
then I agree that we should just ensure that early entropy collection includes
RDRAND/RDSEED input for confidential VMs one way or another.

Best Regards,
Elena.

> 
> Jason

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 02:55:08PM +0000, Reshetova, Elena wrote:
> 
> 
> > On Tue, Jan 30, 2024 at 2:45 PM Reshetova, Elena
> > <elena.reshetova@intel.com> wrote:
> > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > assumption that both of these instructions can fail not just due to broken
> > > HW, but also due to enough pressure put into the whole DRBG construction
> > > that supplies random numbers via RDRAND/RDSEED.
> > 
> > Yea, thought so.
> > 
> > > I guess your concern about DoS here is for the case when we don’t
> > > trust the host/VMM *and* assume malicious userspace, correct?
> > > Because in non-confidential computing case, the Linux RNG in such
> > > case will just use non-RDRAND fallbacks, no DoS will happen and we
> > > should have enough entropy that is outside of userspace control.
> > 
> > Don't think about the RNG for just one second. The basic principle is
> > simpler: if you have a
> > `WARN_ON(unprivd_userspace_triggerable_condition)`, that's usually
> > considered a DoS - panic_on_warn and such.
> 
> Ok, agree, you do bring a valid point that we should not create new 
> DoS attack vectors from  userspace in such cases. 
> 
> > 
> > > >
> > > > And if the DoS thing _is_ a concern, and the use case for this WARN_ON
> > > > in the first place is the trusted computing scenario, so we basically
> > > > only care about early boot, then one addendum would be to only warn if
> > > > we're in early boot, which would work because seeding via RDRAND is
> > > > attempted pretty early on in init.c.
> > >
> > > I don’t think we are only concerned with initial early boot and initial seeding.
> > > What about periodic reseeding of ChaCha CSPRNG? If you don’t get
> > > RDRAND/RDSEED output during this step, don’t we formally loose the forward
> > > prediction resistance property of Linux RNG assuming this is the only source
> > > of entropy that is outside of attacker control?
> > 
> > If you never add new material, and you have the initial seed, then
> > it's deterministic. But you still mostly can't backtrack if the state
> > leaks at some future point in time.
> 
> I am not talking about backtrack resistance, i.e. when attacker learns about 
> RNG state and then can recover the past output. I was talking about an attacker
> learning the RNG state at some point of time (RNG compromise) and
> then for RNG being able to recover over time from this state to a secure state using 
> fresh entropy input that is outside of attacker control/observance.
> Does Linux RNG aim to provide this property? Do people care about this? 
> If noone cares about this one and Linux RNG doesn’t aim to provide it anyhow,
> then I agree that we should just ensure that early entropy collection includes
> RDRAND/RDSEED input for confidential VMs one way or another.

That's the first thing I mentioned -- "If you never add new material,
and you have the initial seed, then it's deterministic." The property
you mention is a good one to have and Linux usually has it. 


> 
> Best Regards,
> Elena.
> 
> > 
> > Jason

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dave Hansen]

On 1/30/24 05:45, Reshetova, Elena wrote:
>> You're the Intel employee so you can find out about this with much
>> more assurance than me, but I understand the sentence above to be _way
>> more_ true for RDRAND than for RDSEED. If your informed opinion is,
>> "RDRAND failing can only be due to totally broken hardware"
> No, this is not the case per Intel SDM. I think we can live under a simple
> assumption that both of these instructions can fail not just due to broken
> HW, but also due to enough pressure put into the whole DRBG construction
> that supplies random numbers via RDRAND/RDSEED. 

I don't think the SDM is the right thing to look at for guidance here.

Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
to be exceedingly rare by design.  If they're not, we're going to get
our trusty torches and pitchforks and go after the folks who built the
broken hardware.

Repeat after me:

	Regular RDRAND/RDSEED failures only occur on broken hardware

If it's nice hardware that's gone bad, then we WARN() and try to make
the best of it.  If it turns out that WARN() was because of a broken
hardware _design_ then we go sharpen the pitchforks.

Anybody disagree?

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 6:32 PM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 1/30/24 05:45, Reshetova, Elena wrote:
> >> You're the Intel employee so you can find out about this with much
> >> more assurance than me, but I understand the sentence above to be _way
> >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> >> "RDRAND failing can only be due to totally broken hardware"
> > No, this is not the case per Intel SDM. I think we can live under a simple
> > assumption that both of these instructions can fail not just due to broken
> > HW, but also due to enough pressure put into the whole DRBG construction
> > that supplies random numbers via RDRAND/RDSEED.
>
> I don't think the SDM is the right thing to look at for guidance here.
>
> Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> to be exceedingly rare by design.  If they're not, we're going to get
> our trusty torches and pitchforks and go after the folks who built the
> broken hardware.
>
> Repeat after me:
>
>         Regular RDRAND/RDSEED failures only occur on broken hardware
>
> If it's nice hardware that's gone bad, then we WARN() and try to make
> the best of it.  If it turns out that WARN() was because of a broken
> hardware _design_ then we go sharpen the pitchforks.
>
> Anybody disagree?

Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
in a busy loop. So at the very least, your statement holds true only
for RDRAND.

But, anyway, if the statement "RDRAND failures only occur on broken
hardware" is true, then a WARN() in the failure path there presents no
DoS potential of any kind, and so that's a straightforward conclusion
to this discussion. However, that really hinges on  "RDRAND failures
only occur on broken hardware" being a true statement.

Also, I don't know how much heavy lifting the word "regular" was doing
in your original statement. Because my example shows that that
irregular RDSEED usage from malicious users can hinder regular users.
If that also applies to RDRAND, the "regular" makes the statement not
as useful for taking conclusive action here.

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dave Hansen]

On 1/30/24 09:49, Jason A. Donenfeld wrote:
>> Anybody disagree?
> Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> in a busy loop. So at the very least, your statement holds true only
> for RDRAND.

Well, darn. :)

Any chance you could share some more information about the environment
where you're seeing this?  It'd be good to reconcile what you're seeing
with how the hardware is expected to behave.

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[H. Peter Anvin]

On January 30, 2024 9:58:09 AM PST, Dave Hansen <dave.hansen@intel.com> wrote:
>On 1/30/24 09:49, Jason A. Donenfeld wrote:
>>> Anybody disagree?
>> Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
>> in a busy loop. So at the very least, your statement holds true only
>> for RDRAND.
>
>Well, darn. :)
>
>Any chance you could share some more information about the environment
>where you're seeing this?  It'd be good to reconcile what you're seeing
>with how the hardware is expected to behave.

What CPU is this and could you clarify exactly how you run your busy loop?

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 7:16 PM H. Peter Anvin <hpa@zytor.com> wrote:
>
> On January 30, 2024 9:58:09 AM PST, Dave Hansen <dave.hansen@intel.com> wrote:
> >On 1/30/24 09:49, Jason A. Donenfeld wrote:
> >>> Anybody disagree?
> >> Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> >> in a busy loop. So at the very least, your statement holds true only
> >> for RDRAND.
> >
> >Well, darn. :)
> >
> >Any chance you could share some more information about the environment
> >where you're seeing this?  It'd be good to reconcile what you're seeing
> >with how the hardware is expected to behave.
>
> What CPU is this and could you clarify exactly how you run your busy loop?

That is already in this thread already. Maybe catch up on the whole
thing and then jump back in?
https://lore.kernel.org/all/Zbjw5hRHr_E6k18r@zx2c4.com/

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 6:58 PM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 1/30/24 09:49, Jason A. Donenfeld wrote:
> >> Anybody disagree?
> > Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> > in a busy loop. So at the very least, your statement holds true only
> > for RDRAND.
>
> Well, darn. :)
>
> Any chance you could share some more information about the environment
> where you're seeing this?  It'd be good to reconcile what you're seeing
> with how the hardware is expected to behave.

That is already in this thread already. Maybe catch up on the whole
thing and then jump back in?
https://lore.kernel.org/all/Zbjw5hRHr_E6k18r@zx2c4.com/

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dave Hansen]

On 1/30/24 10:23, Jason A. Donenfeld wrote:
>> Any chance you could share some more information about the environment
>> where you're seeing this?  It'd be good to reconcile what you're seeing
>> with how the hardware is expected to behave.
> That is already in this thread already. Maybe catch up on the whole
> thing and then jump back in?
> https://lore.kernel.org/all/Zbjw5hRHr_E6k18r@zx2c4.com/

Gah, sorry about that.  I can reproduce what you're seeing, and it does
seem widespread.  Let me do some digging and see where we got our wires
crossed.

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Daniel P. Berrangé]

On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
> On Tue, Jan 30, 2024 at 6:32 PM Dave Hansen <dave.hansen@intel.com> wrote:
> >
> > On 1/30/24 05:45, Reshetova, Elena wrote:
> > >> You're the Intel employee so you can find out about this with much
> > >> more assurance than me, but I understand the sentence above to be _way
> > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> > >> "RDRAND failing can only be due to totally broken hardware"
> > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > assumption that both of these instructions can fail not just due to broken
> > > HW, but also due to enough pressure put into the whole DRBG construction
> > > that supplies random numbers via RDRAND/RDSEED.
> >
> > I don't think the SDM is the right thing to look at for guidance here.
> >
> > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> > to be exceedingly rare by design.  If they're not, we're going to get
> > our trusty torches and pitchforks and go after the folks who built the
> > broken hardware.
> >
> > Repeat after me:
> >
> >         Regular RDRAND/RDSEED failures only occur on broken hardware
> >
> > If it's nice hardware that's gone bad, then we WARN() and try to make
> > the best of it.  If it turns out that WARN() was because of a broken
> > hardware _design_ then we go sharpen the pitchforks.
> >
> > Anybody disagree?
> 
> Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> in a busy loop. So at the very least, your statement holds true only
> for RDRAND.
> 
> But, anyway, if the statement "RDRAND failures only occur on broken
> hardware" is true, then a WARN() in the failure path there presents no
> DoS potential of any kind, and so that's a straightforward conclusion
> to this discussion. However, that really hinges on  "RDRAND failures
> only occur on broken hardware" being a true statement.

There's a useful comment here from an Intel engineer

https://web.archive.org/web/20190219074642/https://software.intel.com/en-us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed

  "RDRAND is, indeed, faster than RDSEED because it comes
   from a hardware-based pseudorandom number generator.
   One seed value (effectively, the output from one RDSEED
   command) can provide up to 511 128-bit random values
   before forcing a reseed"

We know we can exhaust RDSEED directly pretty trivially. Making your
test program run in parallel across 20 cpus, I got a mere 3% success
rate from RDSEED.

If RDRAND is reseeding every 511 values, RDRAND output would have
to be consumed significantly faster than RDSEED in order that the
reseed will happen frequently enough to exhaust the seeds.

This looks pretty hard, but maybe with a large enough CPU count
this will be possible in extreme load ?

So I'm not convinced we can blindly wave away RDRAND failures as
guaranteed to mean broken hardware.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 7:06 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> So I'm not convinced we can blindly wave away RDRAND failures as
> guaranteed to mean broken hardware.

Indeed, and now I'm further disturbed by the @intel.com people on the
thread making these claims that are demonstratively false.

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 7:06 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
> > On Tue, Jan 30, 2024 at 6:32 PM Dave Hansen <dave.hansen@intel.com> wrote:
> > >
> > > On 1/30/24 05:45, Reshetova, Elena wrote:
> > > >> You're the Intel employee so you can find out about this with much
> > > >> more assurance than me, but I understand the sentence above to be _way
> > > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> > > >> "RDRAND failing can only be due to totally broken hardware"
> > > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > > assumption that both of these instructions can fail not just due to broken
> > > > HW, but also due to enough pressure put into the whole DRBG construction
> > > > that supplies random numbers via RDRAND/RDSEED.
> > >
> > > I don't think the SDM is the right thing to look at for guidance here.
> > >
> > > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> > > to be exceedingly rare by design.  If they're not, we're going to get
> > > our trusty torches and pitchforks and go after the folks who built the
> > > broken hardware.
> > >
> > > Repeat after me:
> > >
> > >         Regular RDRAND/RDSEED failures only occur on broken hardware
> > >
> > > If it's nice hardware that's gone bad, then we WARN() and try to make
> > > the best of it.  If it turns out that WARN() was because of a broken
> > > hardware _design_ then we go sharpen the pitchforks.
> > >
> > > Anybody disagree?
> >
> > Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> > in a busy loop. So at the very least, your statement holds true only
> > for RDRAND.
> >
> > But, anyway, if the statement "RDRAND failures only occur on broken
> > hardware" is true, then a WARN() in the failure path there presents no
> > DoS potential of any kind, and so that's a straightforward conclusion
> > to this discussion. However, that really hinges on  "RDRAND failures
> > only occur on broken hardware" being a true statement.
>
> There's a useful comment here from an Intel engineer
>
> https://web.archive.org/web/20190219074642/https://software.intel.com/en-us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed
>
>   "RDRAND is, indeed, faster than RDSEED because it comes
>    from a hardware-based pseudorandom number generator.
>    One seed value (effectively, the output from one RDSEED
>    command) can provide up to 511 128-bit random values
>    before forcing a reseed"
>
> We know we can exhaust RDSEED directly pretty trivially. Making your
> test program run in parallel across 20 cpus, I got a mere 3% success
> rate from RDSEED.
>
> If RDRAND is reseeding every 511 values, RDRAND output would have
> to be consumed significantly faster than RDSEED in order that the
> reseed will happen frequently enough to exhaust the seeds.
>
> This looks pretty hard, but maybe with a large enough CPU count
> this will be possible in extreme load ?

So what this suggests is that the guest-guest DoS caused by looping
forever (or panic-on-warn'ing) is at least possible on large enough
hardware for some non-zero amount of time, depending on whatever hard
to hit environmental factors.

Another approach would be to treat this as a hardware flaw, in that
the RDRAND does not provide a universally reliable interface, and so
something like CoCo doesn't work with the current design, and so Intel
should issue some microcode updates that gives some separated pools
and separated rate limiting on a per-VMX ring 0 basis. Or something
like that. I dunno; maybe it's unrealistic to hope Intel will repair
their interface. But I think we've got to acknowledge that it's sort
of broken/irreliable.

Jason

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[H. Peter Anvin]

On January 30, 2024 10:05:59 AM PST, "Daniel P. Berrangé" <berrange@redhat.com> wrote:
>On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
>> On Tue, Jan 30, 2024 at 6:32 PM Dave Hansen <dave.hansen@intel.com> wrote:
>> >
>> > On 1/30/24 05:45, Reshetova, Elena wrote:
>> > >> You're the Intel employee so you can find out about this with much
>> > >> more assurance than me, but I understand the sentence above to be _way
>> > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
>> > >> "RDRAND failing can only be due to totally broken hardware"
>> > > No, this is not the case per Intel SDM. I think we can live under a simple
>> > > assumption that both of these instructions can fail not just due to broken
>> > > HW, but also due to enough pressure put into the whole DRBG construction
>> > > that supplies random numbers via RDRAND/RDSEED.
>> >
>> > I don't think the SDM is the right thing to look at for guidance here.
>> >
>> > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
>> > to be exceedingly rare by design.  If they're not, we're going to get
>> > our trusty torches and pitchforks and go after the folks who built the
>> > broken hardware.
>> >
>> > Repeat after me:
>> >
>> >         Regular RDRAND/RDSEED failures only occur on broken hardware
>> >
>> > If it's nice hardware that's gone bad, then we WARN() and try to make
>> > the best of it.  If it turns out that WARN() was because of a broken
>> > hardware _design_ then we go sharpen the pitchforks.
>> >
>> > Anybody disagree?
>> 
>> Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
>> in a busy loop. So at the very least, your statement holds true only
>> for RDRAND.
>> 
>> But, anyway, if the statement "RDRAND failures only occur on broken
>> hardware" is true, then a WARN() in the failure path there presents no
>> DoS potential of any kind, and so that's a straightforward conclusion
>> to this discussion. However, that really hinges on  "RDRAND failures
>> only occur on broken hardware" being a true statement.
>
>There's a useful comment here from an Intel engineer
>
>https://web.archive.org/web/20190219074642/https://software.intel.com/en-us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed
>
>  "RDRAND is, indeed, faster than RDSEED because it comes
>   from a hardware-based pseudorandom number generator.
>   One seed value (effectively, the output from one RDSEED
>   command) can provide up to 511 128-bit random values
>   before forcing a reseed"
>
>We know we can exhaust RDSEED directly pretty trivially. Making your
>test program run in parallel across 20 cpus, I got a mere 3% success
>rate from RDSEED.
>
>If RDRAND is reseeding every 511 values, RDRAND output would have
>to be consumed significantly faster than RDSEED in order that the
>reseed will happen frequently enough to exhaust the seeds.
>
>This looks pretty hard, but maybe with a large enough CPU count
>this will be possible in extreme load ?
>
>So I'm not convinced we can blindly wave away RDRAND failures as
>guaranteed to mean broken hardware.
>
>With regards,
>Daniel

The general approach has been "don't credit entropy and try again on the next interrupt." We can, of course, be much more aggressive during boot.

We only need 256-512 bits for the kernel random pool to be equivalent to breaking mainstream crypto primitives even if it is a PRNG only from that point on (which is extremely unlikely.) The Linux PRNG has a very large state, which helps buffer entropy variations.

Again, applications *should* be using /dev/[u]random as appropriate, and if they opt to use lower level primitives in user space they need to implement them correctly – there is literally nothing the kernel can do at that point.

If the probability of success is 3% per your CPU that is still 2 bits of true entropy per invocation. However, the probability of failure after 16 loops is over 60%. I think this validates the concept of continuing to poll periodically rather than looping in time critical paths.

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
> > On Tue, Jan 30, 2024 at 6:32 PM Dave Hansen <dave.hansen@intel.com> wrote:
> > >
> > > On 1/30/24 05:45, Reshetova, Elena wrote:
> > > >> You're the Intel employee so you can find out about this with much
> > > >> more assurance than me, but I understand the sentence above to be _way
> > > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> > > >> "RDRAND failing can only be due to totally broken hardware"
> > > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > > assumption that both of these instructions can fail not just due to broken
> > > > HW, but also due to enough pressure put into the whole DRBG construction
> > > > that supplies random numbers via RDRAND/RDSEED.
> > >
> > > I don't think the SDM is the right thing to look at for guidance here.
> > >
> > > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> > > to be exceedingly rare by design.  If they're not, we're going to get
> > > our trusty torches and pitchforks and go after the folks who built the
> > > broken hardware.
> > >
> > > Repeat after me:
> > >
> > >         Regular RDRAND/RDSEED failures only occur on broken hardware
> > >
> > > If it's nice hardware that's gone bad, then we WARN() and try to make
> > > the best of it.  If it turns out that WARN() was because of a broken
> > > hardware _design_ then we go sharpen the pitchforks.
> > >
> > > Anybody disagree?
> >
> > Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> > in a busy loop. So at the very least, your statement holds true only
> > for RDRAND.
> >
> > But, anyway, if the statement "RDRAND failures only occur on broken
> > hardware" is true, then a WARN() in the failure path there presents no
> > DoS potential of any kind, and so that's a straightforward conclusion
> > to this discussion. However, that really hinges on  "RDRAND failures
> > only occur on broken hardware" being a true statement.
> 
> There's a useful comment here from an Intel engineer
> 
> https://web.archive.org/web/20190219074642/https://software.intel.com/en-
> us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed
> 
>   "RDRAND is, indeed, faster than RDSEED because it comes
>    from a hardware-based pseudorandom number generator.
>    One seed value (effectively, the output from one RDSEED
>    command) can provide up to 511 128-bit random values
>    before forcing a reseed"
> 
> We know we can exhaust RDSEED directly pretty trivially. Making your
> test program run in parallel across 20 cpus, I got a mere 3% success
> rate from RDSEED.
> 
> If RDRAND is reseeding every 511 values, RDRAND output would have
> to be consumed significantly faster than RDSEED in order that the
> reseed will happen frequently enough to exhaust the seeds.
> 
> This looks pretty hard, but maybe with a large enough CPU count
> this will be possible in extreme load ?
> 
> So I'm not convinced we can blindly wave away RDRAND failures as
> guaranteed to mean broken hardware.

This matches both my understanding (I do have cryptography background
and understanding how cryptographic RNGs work)
and official public docs that Intel published on this matter. 
Given that the physical entropy source is limited anyhow, and by giving
enough pressure on the whole construction you should be able to
make RDRAND fail because if the intermediate AES-CBC MAC extractor/
conditioner is not getting its min entropy input rate, it wont
produce a proper seed for AES CTR DRBG. 
Of course exact details/numbers can wary between different generations of 
Intel DRNG implementation, and the platforms where it is running on,
so be careful to sticking to concrete numbers. 

That said, I have taken an AR to follow up internally on what can be done
to improve our situation with RDRAND/RDSEED. But I would still like to
finish the discussion on what people think should be done in the
meanwhile keeping in mind that the problem is not intel specific, despite us
intel people bringing it for public discussion first. The old saying is still here:
"Please don’t shoot the messenger" )) We are actually trying to be open 
about these things and create a public discussion. 

Best Regards,
Elena. 

> 
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Wed, Jan 31, 2024 at 08:16:56AM +0000, Reshetova, Elena wrote:

Good morning, I hope the week is going well for everyone.

> > On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
> > > On Tue, Jan 30, 2024 at 6:32???PM Dave Hansen <dave.hansen@intel.com> wrote:
> > > >
> > > > On 1/30/24 05:45, Reshetova, Elena wrote:
> > > > >> You're the Intel employee so you can find out about this with much
> > > > >> more assurance than me, but I understand the sentence above to be _way
> > > > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> > > > >> "RDRAND failing can only be due to totally broken hardware"
> > > > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > > > assumption that both of these instructions can fail not just due to broken
> > > > > HW, but also due to enough pressure put into the whole DRBG construction
> > > > > that supplies random numbers via RDRAND/RDSEED.
> > > >
> > > > I don't think the SDM is the right thing to look at for guidance here.
> > > >
> > > > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> > > > to be exceedingly rare by design.  If they're not, we're going to get
> > > > our trusty torches and pitchforks and go after the folks who built the
> > > > broken hardware.
> > > >
> > > > Repeat after me:
> > > >
> > > >         Regular RDRAND/RDSEED failures only occur on broken hardware
> > > >
> > > > If it's nice hardware that's gone bad, then we WARN() and try to make
> > > > the best of it.  If it turns out that WARN() was because of a broken
> > > > hardware _design_ then we go sharpen the pitchforks.
> > > >
> > > > Anybody disagree?
> > >
> > > Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> > > in a busy loop. So at the very least, your statement holds true only
> > > for RDRAND.
> > >
> > > But, anyway, if the statement "RDRAND failures only occur on broken
> > > hardware" is true, then a WARN() in the failure path there presents no
> > > DoS potential of any kind, and so that's a straightforward conclusion
> > > to this discussion. However, that really hinges on  "RDRAND failures
> > > only occur on broken hardware" being a true statement.
> > 
> > There's a useful comment here from an Intel engineer
> > 
> > https://web.archive.org/web/20190219074642/https://software.intel.com/en-
> > us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed
> > 
> >   "RDRAND is, indeed, faster than RDSEED because it comes
> >    from a hardware-based pseudorandom number generator.
> >    One seed value (effectively, the output from one RDSEED
> >    command) can provide up to 511 128-bit random values
> >    before forcing a reseed"
> > 
> > We know we can exhaust RDSEED directly pretty trivially. Making your
> > test program run in parallel across 20 cpus, I got a mere 3% success
> > rate from RDSEED.
> > 
> > If RDRAND is reseeding every 511 values, RDRAND output would have
> > to be consumed significantly faster than RDSEED in order that the
> > reseed will happen frequently enough to exhaust the seeds.
> > 
> > This looks pretty hard, but maybe with a large enough CPU count
> > this will be possible in extreme load ?
> > 
> > So I'm not convinced we can blindly wave away RDRAND failures as
> > guaranteed to mean broken hardware.

> This matches both my understanding (I do have cryptography
> background and understanding how cryptographic RNGs work) and
> official public docs that Intel published on this matter.  Given
> that the physical entropy source is limited anyhow, and by giving
> enough pressure on the whole construction you should be able to make
> RDRAND fail because if the intermediate AES-CBC MAC extractor/
> conditioner is not getting its min entropy input rate, it wont
> produce a proper seed for AES CTR DRBG.  Of course exact
> details/numbers can wary between different generations of Intel DRNG
> implementation, and the platforms where it is running on, so be
> careful to sticking to concrete numbers.
>
> That said, I have taken an AR to follow up internally on what can be
> done to improve our situation with RDRAND/RDSEED. But I would still
> like to finish the discussion on what people think should be done in
> the meanwhile keeping in mind that the problem is not intel
> specific, despite us intel people bringing it for public discussion
> first. The old saying is still here: "Please don't shoot the
> messenger" )) We are actually trying to be open about these things
> and create a public discussion.

Based on Dave Hansen's comments above, it appears that the COCO
community needs to break out the oil and whetstones and hone the tips
of their pitchforks.. :-)

The positive issue in all of this is that, to date, TDX hardware has
not seen significant public availability.  I suspect that when that
happens, if this problem isn't corrected, there will be the usual
flood of papers demonstrating quasi-practical lab attacks that stem
from the fruits of a poisonable random number source.

The problem reproduces pretty easily, albeit on somewhat dated
hardware.

One of our lab machines, that reports a model name of 'Intel(R)
Core(TM) i5-6500 CPU @ 3.20GHz', shows a consistent failure rate of
65% for RDSEED on a single-threaded test.  It gets worse when more
simultaneous demand is placed on the hardware randomness source, as
was demonstrated elsewhere.

Corrupted randomness has the potential to strike pretty deeply into
the TDX attestation architecture, given the need to generate signed
attestation reports and the SGX/TDX key generation architecture that
requires 'wearout protection'.  Beyond that, there is the subsequent
need to conduct userspace attestation, currently by IMA as I believe
is the intent, that in turn requires cryptography with undeniable
integrity.

At this point, given that all this is about confidentiality, that in
turn implies a trusted platform, there is only one option, panic and
hard fail the boot if there is any indication that the hardware has
not been able to provide sound instruction based randomness.  Doing
anything else breaks the 'contract' that a user is pushing a workload
into a trusted/confidential environment.

RDSEED is the root of hardware instruction based randomness and its
randomness comes from quantum noise across a diode junction
(simplistically).  The output of RDSEED drives the AES derived RDRAND
randomness.

Additional clarification from inside of Intel on this is needed, but
the problem would appear to be that the above infrastructure
(RDSEED/RDRAND) is part of the 'Uncore' architecture, rather than
being core specific.  This creates an incestuous relationship across
all of the cores sharing a resource, that as in the past, creates
security issues.

This issue was easily anticipated and foreshadowed by the
demonstration of the CVE-2020-0543/CrossTalk vulnerability.

If the above interpretion is correct, a full fix should be 'straight
forward', for some definition of 'straight forward'... :-)

On TDX capable hardare, the RDSEED noise source needs to come from a
piece of core specific silicon.  If the boot of a TDX VM is core
locked, this would create an environment where a socket based sibling
adversary would be unable to compromise the root of the randomness
source.

Once the Linux random number generator is properly seeded, the issue
should be over, given that by now, everyone has agreed that a properly
initialized Linux RNG cannot 'run out' of randomness.

Given that it seems pretty clear that timing and other 'noise'
information in a COCO environment can't be trusted, having core
specific randomness would be a win for the overall cryptographic
health of VM's that are running in a COCO environment.

Once again, an attack doesn't need to be practical, only demonstrable.
Once demonstrated, faith is lost in the technology, SGX clearly
demonstrated that, as did the micro-architectural attacks.

Both SGX and TDX operate from the notion of 'you trust Intel and the
silicon', so the fix is for Intel to implement a secure silicon based
source of randomness.

AMD will probably need the same thing.

> Best Regards,
> Elena. 

Hopefuly the above is helpful in furthering these discussions.

Have a good remainder of the week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

On Wed, Jan 31, 2024 at 9:17 AM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
> This matches both my understanding (I do have cryptography background
> and understanding how cryptographic RNGs work)
> and official public docs that Intel published on this matter.
> Given that the physical entropy source is limited anyhow, and by giving
> enough pressure on the whole construction you should be able to
> make RDRAND fail because if the intermediate AES-CBC MAC extractor/
> conditioner is not getting its min entropy input rate, it wont
> produce a proper seed for AES CTR DRBG.
> Of course exact details/numbers can wary between different generations of
> Intel DRNG implementation, and the platforms where it is running on,
> so be careful to sticking to concrete numbers.

Alright, so RDRAND is not reliable. The question for us now is: do we
want RDRAND unreliability to translate to another form of
unreliability elsewhere, e.g. DoS/infiniteloop/latency/WARN_ON()? Or
would it be better to declare the hardware simply broken and ask Intel
to fix it? (I don't know the answer to that question.)

> That said, I have taken an AR to follow up internally on what can be done
> to improve our situation with RDRAND/RDSEED.

Specifying this is an interesting question. What exactly might our
requirements be for a "non-broken" RDRAND? It seems like we have two
basic ones:

- One VMX (or host) context can't DoS another one.
- Ring 3 can't DoS ring 0.

I don't know whether that'd be implemented with context-tied rate
limiting or more state or what. But I think, short of just making
RDRAND never fail, that's basically what's needed.

Jason

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> On Wed, Jan 31, 2024 at 9:17 AM Reshetova, Elena
> <elena.reshetova@intel.com> wrote:
> > This matches both my understanding (I do have cryptography background
> > and understanding how cryptographic RNGs work)
> > and official public docs that Intel published on this matter.
> > Given that the physical entropy source is limited anyhow, and by giving
> > enough pressure on the whole construction you should be able to
> > make RDRAND fail because if the intermediate AES-CBC MAC extractor/
> > conditioner is not getting its min entropy input rate, it wont
> > produce a proper seed for AES CTR DRBG.
> > Of course exact details/numbers can wary between different generations of
> > Intel DRNG implementation, and the platforms where it is running on,
> > so be careful to sticking to concrete numbers.
> 
> Alright, so RDRAND is not reliable. 

Correction here: "... not reliable *in theory*". Because in practice it
all depends on amount of pressure you are able to put on the overall 
construction, which goes into concrete numbers I warned about.
That would depend on the number of available cores, and some other
platform specific factors. I will work on getting this clarified externally
so that there is no confusion. 


The question for us now is: do we
> want RDRAND unreliability to translate to another form of
> unreliability elsewhere, e.g. DoS/infiniteloop/latency/WARN_ON()? Or
> would it be better to declare the hardware simply broken and ask Intel
> to fix it? (I don't know the answer to that question.)
> 
> > That said, I have taken an AR to follow up internally on what can be done
> > to improve our situation with RDRAND/RDSEED.
> 
> Specifying this is an interesting question. What exactly might our
> requirements be for a "non-broken" RDRAND? It seems like we have two
> basic ones:
> 
> - One VMX (or host) context can't DoS another one.
> - Ring 3 can't DoS ring 0.
> 
> I don't know whether that'd be implemented with context-tied rate
> limiting or more state or what. But I think, short of just making
> RDRAND never fail, that's basically what's needed.

I agree.

Best Regards,
Elena.

> 
> Jason

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Wed, Jan 31, 2024 at 02:06:13PM +0100, Jason A. Donenfeld wrote:

Hi again to everyone, beautiful day here in North Dakota.

> On Wed, Jan 31, 2024 at 9:17???AM Reshetova, Elena
> <elena.reshetova@intel.com> wrote:
> > This matches both my understanding (I do have cryptography background
> > and understanding how cryptographic RNGs work)
> > and official public docs that Intel published on this matter.
> > Given that the physical entropy source is limited anyhow, and by giving
> > enough pressure on the whole construction you should be able to
> > make RDRAND fail because if the intermediate AES-CBC MAC extractor/
> > conditioner is not getting its min entropy input rate, it wont
> > produce a proper seed for AES CTR DRBG.
> > Of course exact details/numbers can wary between different generations of
> > Intel DRNG implementation, and the platforms where it is running on,
> > so be careful to sticking to concrete numbers.

> Alright, so RDRAND is not reliable. The question for us now is: do
> we want RDRAND unreliability to translate to another form of
> unreliability elsewhere, e.g. DoS/infiniteloop/latency/WARN_ON()? Or
> would it be better to declare the hardware simply broken and ask
> Intel to fix it? (I don't know the answer to that question.)

I think it would demonstrate a lack of appropriate engineering
diligence on the part of our community to declare RDRAND 'busted' at
this point.

While it appeares to be trivially easy to force RDSEED into depletion,
there does not seem to be a suggestion, at least in the open
literature, that this directly or easily translates into stalling
output from RDRAND in any type of relevant adversarial fashion.

If this were the case, given what CVE's seem to be worth on a resume,
someone would have rented a cloud machine and come up with a POC
against RDRAND in a multi-tenant environment and then promptly put up
a web-site called 'Random Starve' or something equally ominous.

This is no doubt secondary to the 1022x amplication factor inherent in
the 'Bull Mountain' architecture.

I'm a bit surprised that no one from the Intel side of this
conversation didn't pitch this over the wall as soon as this
conversation came up, but I would suggest that everyone concerned
about this issue give the following a thorough read:

https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html

Relevant highlights:

- As I suggested in my earlier e-mail, random number generation is a
  socket based resource, hence an adversarial domain limited to only
  the cores on a common socket.

- There is a maximum randomness throughput rate of 800 MB/s over all
  cores sharing common random number infrastructure.  Single thread
  throughput rates of 70-200 MB/s are demonstratable.

- A failure of RDRAND over 10 re-tries is 'astronomically' small, with
  no definition of astronomical provided, one would assume really
  small, given they are using the word astronomical.

> > That said, I have taken an AR to follow up internally on what can be done
> > to improve our situation with RDRAND/RDSEED.

I think I can save you some time Elena.

> Specifying this is an interesting question. What exactly might our
> requirements be for a "non-broken" RDRAND? It seems like we have two
> basic ones:
> 
> - One VMX (or host) context can't DoS another one.
> - Ring 3 can't DoS ring 0.
> 
> I don't know whether that'd be implemented with context-tied rate
> limiting or more state or what. But I think, short of just making
> RDRAND never fail, that's basically what's needed.

I think we probably have that, for all intents and purposes, given
that we embrace the following methodogy:

- Use RDRAND exclusively.

- Be willing to take 10 swings at the plate.

- Given the somewhat demanding requirements for TDX/COCO, fail and
  either deadlock or panic after 10 swings since that would seem to
  suggest the hardware is broken, ie. RMA time.

Either deadlock or panic would be appropriate.  The objective in the
COCO environment is to get the person who clicked on the 'Enable Azure
Confidential' checkbox, or its equivalent, on their cloud dashboard,
to call the HelpDesk and ask them why their confidential application
won't come up.

After the user confirms to the HelpDesk that their computer is plugged
in, the problem will get fixed.  Either the broken hardware will be
identified and idled out or the mighty sword of vengeance will be
summoned down on whoever has all of the other cores on the socket
pegged.

Final thoughts:

- RDSEED is probably a poor thing to be using.

- There may be a reasonable argument that RDSEED shouldn't have been
  exposed above ring 0, but that ship has sailed.  Brownie points
  moving forward for an RDsomething that is ring 0 and has guaranteed
  access to some amount of functionally reasonable entropy.

- Intel and AMD are already doing a lot of 'special' stuff with their
  COCO hardware in order to defy the long standing adage of: 'You
  can't have security without physical security'.  Access to per core thermal
  noise, as I suggested, is probably a big lift but clever engineers can
  probably cook up some type of fairness doctrine for randomness in
  TDX or SEV_SNP, given the particular importance of instruction based
  randomness in COCO.

- Perfection is the enemy of good.

> Jason

Have a good day.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Theodore Ts'o]

On Wed, Jan 31, 2024 at 02:35:32PM -0600, Dr. Greg wrote:
> I think it would demonstrate a lack of appropriate engineering
> diligence on the part of our community to declare RDRAND 'busted' at
> this point.
> 
> While it appeares to be trivially easy to force RDSEED into depletion,
> there does not seem to be a suggestion, at least in the open
> literature, that this directly or easily translates into stalling
> output from RDRAND in any type of relevant adversarial fashion.
> 
> If this were the case, given what CVE's seem to be worth on a resume,
> someone would have rented a cloud machine and come up with a POC
> against RDRAND in a multi-tenant environment and then promptly put up
> a web-site called 'Random Starve' or something equally ominous.

I suspect the reason why DOS attacks aren't happening in practice, is
because of concerns over the ability to trust the RDRAND (how do you
prove that the NSA didn't put a backdoor into the hardware with
Intel's acquisence --- after all, the NSA absolutely positively didn't
encourage the kneecaping of WEP and absolutely didn't put a trapdoor
into DUAL_EC_DRBG...)  since it can not externally audited and verfied
by a third party, in contrast to the source code for the /dev/random
driver or the RNG used in OpenSSL.

As a result, most random number generators use RDRAND in combination
with other techniques.  If RDRAND is absolutely trustworthy, the extra
sources won't hurt --- and if it isn't trustworthy mixing in other
sources will likely make things harder for Fort Meade.  And even if
these other sources might be observable for someone who can listen in
on the inter-packet arrival times on the LAN (for example), it might
not be so easy for an analyst sitting at their desk in Fort Meade.

And once you do _that_, you don't need to necessarily loop on RDRAND,
because it's one of multiple sources of entropies that are getting
mixed togethwer.  Hence, even if someone drives RDRAND into depletion,
if they are using getrandom(2), it's not a big deal.

There's a special case with Confidential Compute VM's, since the
assumption is that you want to protect against even a malicious
hypervisor who could theoretically control all other sources of timing
uncertainty.  And so, yes, in that case, the only thing we can do is
Panic if RDRAND fails.

   	    	  		     	 - Ted

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Wed, Jan 31, 2024 at 11:47:35PM -0500, Theodore Ts'o wrote:
> On Wed, Jan 31, 2024 at 02:35:32PM -0600, Dr. Greg wrote:
> > I think it would demonstrate a lack of appropriate engineering
> > diligence on the part of our community to declare RDRAND 'busted' at
> > this point.
> > 
> > While it appeares to be trivially easy to force RDSEED into depletion,
> > there does not seem to be a suggestion, at least in the open
> > literature, that this directly or easily translates into stalling
> > output from RDRAND in any type of relevant adversarial fashion.
> > 
> > If this were the case, given what CVE's seem to be worth on a resume,
> > someone would have rented a cloud machine and come up with a POC
> > against RDRAND in a multi-tenant environment and then promptly put up
> > a web-site called 'Random Starve' or something equally ominous.

> I suspect the reason why DOS attacks aren't happening in practice, is
> because of concerns over the ability to trust the RDRAND (how do you
> prove that the NSA didn't put a backdoor into the hardware with
> Intel's acquisence --- after all, the NSA absolutely positively didn't
> encourage the kneecaping of WEP and absolutely didn't put a trapdoor
> into DUAL_EC_DRBG...)  since it can not externally audited and verfied
> by a third party, in contrast to the source code for the /dev/random
> driver or the RNG used in OpenSSL.
> 
> As a result, most random number generators use RDRAND in combination
> with other techniques.  If RDRAND is absolutely trustworthy, the extra
> sources won't hurt --- and if it isn't trustworthy mixing in other
> sources will likely make things harder for Fort Meade.  And even if
> these other sources might be observable for someone who can listen in
> on the inter-packet arrival times on the LAN (for example), it might
> not be so easy for an analyst sitting at their desk in Fort Meade.
> 
> And once you do _that_, you don't need to necessarily loop on RDRAND,
> because it's one of multiple sources of entropies that are getting
> mixed togethwer.  Hence, even if someone drives RDRAND into depletion,
> if they are using getrandom(2), it's not a big deal.

All well taken points, the Linux RNG and associated community has
benefited from your and Jason's concerns and work on all of this.

However, whether or not DOS attacks based on RNG depletion are
happening in the wild, and the reasons they are not, are orthogonal to
whether or not they can be proven to exist, which was the point I was
trying to make.

Demonstrating a vulnerability in something as critical as Intel's RNG
implementation would be a big motivation for some research group.  The
fact that hasn't occurred would seem to suggest that the RDRAND
resource depletion we are concerned with is not adversarially
exploitable.

I suspect that the achievable socket core count cannot effectively
overwhelm the 1022x amplification factor inherent in the design of the
RDSEED based seeding of RDRAND.

We will see if Elena can come up with what Intel engineering's
definition of 'astronomical' is.. :-)

> There's a special case with Confidential Compute VM's, since the
> assumption is that you want to protect against even a malicious
> hypervisor who could theoretically control all other sources of
> timing uncertainty.  And so, yes, in that case, the only thing we
> can do is Panic if RDRAND fails.

Indeed.

The bigger question, which I will respond to Elena with, is how much
this issue calls the entire question of confidential computing into
question.

>    	    	  		     	 - Ted

Have a good day, it has been a long time since you and I were standing
around with Phil Hughes in the Galleria in Atlanta argueing about
whether or not Active Directory was going to dominate enterprise
computing.

It does seem to have gained some significant amount of traction... :-)

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Daniel P. Berrangé]

On Thu, Feb 01, 2024 at 03:54:51AM -0600, Dr. Greg wrote:
> I suspect that the achievable socket core count cannot effectively
> overwhelm the 1022x amplification factor inherent in the design of the
> RDSEED based seeding of RDRAND.

In testing I could get RDSEED down to < 3% success rate when
running on 20 cores in parallel on a laptop class i7. If that
failure rate can be improved by a little more than one order
of magnitude to 0.1% we're starting to get to the point where
it might be enough to make RDRAND re-seed fail.

Intel's Sierra Forest CPUs are said to have a variant with 288
cores per socket, which is an order of magnitude larger. It is
conceivable this might be large enough to demonstrate RDRAND
failure in extreme load. Then again who knows what else has
changed that might alter the equation, maybe the DRBG is also
better / faster. Only real world testing can say for sure.
One thing is certain though, core counts per socket keep going
up, so the potential worst case load on RDSEED will increase...

> We will see if Elena can come up with what Intel engineering's
> definition of 'astronomical' is.. :-)
> 
> > There's a special case with Confidential Compute VM's, since the
> > assumption is that you want to protect against even a malicious
> > hypervisor who could theoretically control all other sources of
> > timing uncertainty.  And so, yes, in that case, the only thing we
> > can do is Panic if RDRAND fails.
> 
> Indeed.
> 
> The bigger question, which I will respond to Elena with, is how much
> this issue calls the entire question of confidential computing into
> question.

A denial of service (from a panic on RDRAND fail) doesn't undermine
confidental computing. Guest data confidentiality is maintained by
panicing on RDRAND failure and DoS protection isn't a threat that CC
claims to be able to mitigate in general.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Thu, Feb 01, 2024 at 11:08:09AM +0000, Daniel P. Berrang?? wrote:

Hi Dan, thanks for the thoughts.

> On Thu, Feb 01, 2024 at 03:54:51AM -0600, Dr. Greg wrote:
> > I suspect that the achievable socket core count cannot effectively
> > overwhelm the 1022x amplification factor inherent in the design of the
> > RDSEED based seeding of RDRAND.

> In testing I could get RDSEED down to < 3% success rate when
> running on 20 cores in parallel on a laptop class i7. If that
> failure rate can be improved by a little more than one order
> of magnitude to 0.1% we're starting to get to the point where
> it might be enough to make RDRAND re-seed fail.
> 
> Intel's Sierra Forest CPUs are said to have a variant with 288
> cores per socket, which is an order of magnitude larger. It is
> conceivable this might be large enough to demonstrate RDRAND
> failure in extreme load. Then again who knows what else has
> changed that might alter the equation, maybe the DRBG is also
> better / faster. Only real world testing can say for sure.
> One thing is certain though, core counts per socket keep going
> up, so the potential worst case load on RDSEED will increase...

Indeed, that would seem to be the important and operative question
that Intel could answer, maybe Dave and Elena will be able to provide
some guidance.

Until someone can actually demonstrate a sustained RDRAND depletion
attack we don't have an issue, only a lot of wringing of hands and
other handwaving on what we should do.

The thing that intrigues me is that we have two AMD engineers
following this, do you guys have any comments, reflections?  Unless I
misunderstand, SEV-SNP has the same challenges and issues.

As of late you guys have been delivering higher core counts that would
make your platform more susceptible.  Does your hardware design not
have a socket common RNG architecture that makes RDSEED vulnerable to
socket adversarial depletion?  Is this a complete non-issue in
practice?

Big opportunity here to proclaim: "Just buy AMD"... :-)

> > We will see if Elena can come up with what Intel engineering's
> > definition of 'astronomical' is.. :-)
> > 
> > > There's a special case with Confidential Compute VM's, since the
> > > assumption is that you want to protect against even a malicious
> > > hypervisor who could theoretically control all other sources of
> > > timing uncertainty.  And so, yes, in that case, the only thing we
> > > can do is Panic if RDRAND fails.
> > 
> > Indeed.
> > 
> > The bigger question, which I will respond to Elena with, is how much
> > this issue calls the entire question of confidential computing into
> > question.

> A denial of service (from a panic on RDRAND fail) doesn't undermine
> confidental computing. Guest data confidentiality is maintained by
> panicing on RDRAND failure and DoS protection isn't a threat that CC
> claims to be able to mitigate in general.

Yes, if there is a problem with RDRAND we have a CoCo solution, full
stop.

The issue that I was raising with Elena is more generic, to wit:

Her expressed concern is that a code construct looking something like this,
rdrand() returning 0 on success:

for (i= 0; i < 9; ++i)
	if (!rdrand(&seed))
		break;
	sleep(some time);
}
if (i == 9)
	BUG("No entropy");

do_something_with(seed);

Could be sufficiently manipulated by a malicious hypervisor in a TDX
environment so as to compromise its functionality.

If this level of control is indeed possible, given the long history of
timing and side-channel attacks against cryptography, this would seem
to pose significant questions as to whether or not CoCo can deliver on
its stated goals.

> With regards,
> Daniel

Have a good evening.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> > > The bigger question, which I will respond to Elena with, is how much
> > > this issue calls the entire question of confidential computing into
> > > question.
> 
> > A denial of service (from a panic on RDRAND fail) doesn't undermine
> > confidental computing. Guest data confidentiality is maintained by
> > panicing on RDRAND failure and DoS protection isn't a threat that CC
> > claims to be able to mitigate in general.
> 
> Yes, if there is a problem with RDRAND we have a CoCo solution, full
> stop.
> 
> The issue that I was raising with Elena is more generic, to wit:
> 
> Her expressed concern is that a code construct looking something like this,
> rdrand() returning 0 on success:
> 
> for (i= 0; i < 9; ++i)
> 	if (!rdrand(&seed))
> 		break;
> 	sleep(some time);
> }
> if (i == 9)
> 	BUG("No entropy");
> 
> do_something_with(seed);
> 
> Could be sufficiently manipulated by a malicious hypervisor in a TDX
> environment so as to compromise its functionality.

This is not what I had in mind. How does the above can be manipulated
by a malicious hypervisor? If the above construction can be
logically manipulated we have other issues than rdrand, this is imo
already a control flow manipulation attack that you are stating here. 

What a malicious hypervisor can *in theory* do is to insert the execution
delays and make the above loop fail even if we assume that the probability
of falling the 10 retry loop is negligible in normal cases (assuming tightness
or other timing requirements). But again, this is theoretical at this point.
But if the SW refuses to proceed and panics in such cases, we have a DoS
as we already discussed.

Best Regards,
Elena.


> 
> If this level of control is indeed possible, given the long history of
> timing and side-channel attacks against cryptography, this would seem
> to pose significant questions as to whether or not CoCo can deliver on
> its stated goals.
> 
> > With regards,
> > Daniel
> 
> Have a good evening.
> 
> As always,
> Dr. Greg
> 
> The Quixote Project - Flailing at the Travails of Cybersecurity
>               https://github.com/Quixote-Project

RE: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Reshetova, Elena]

> On Wed, Jan 31, 2024 at 02:06:13PM +0100, Jason A. Donenfeld wrote:
> 
> Hi again to everyone, beautiful day here in North Dakota.
> 
> > On Wed, Jan 31, 2024 at 9:17???AM Reshetova, Elena
> > <elena.reshetova@intel.com> wrote:
> > > This matches both my understanding (I do have cryptography background
> > > and understanding how cryptographic RNGs work)
> > > and official public docs that Intel published on this matter.
> > > Given that the physical entropy source is limited anyhow, and by giving
> > > enough pressure on the whole construction you should be able to
> > > make RDRAND fail because if the intermediate AES-CBC MAC extractor/
> > > conditioner is not getting its min entropy input rate, it wont
> > > produce a proper seed for AES CTR DRBG.
> > > Of course exact details/numbers can wary between different generations of
> > > Intel DRNG implementation, and the platforms where it is running on,
> > > so be careful to sticking to concrete numbers.
> 
> > Alright, so RDRAND is not reliable. The question for us now is: do
> > we want RDRAND unreliability to translate to another form of
> > unreliability elsewhere, e.g. DoS/infiniteloop/latency/WARN_ON()? Or
> > would it be better to declare the hardware simply broken and ask
> > Intel to fix it? (I don't know the answer to that question.)
> 
> I think it would demonstrate a lack of appropriate engineering
> diligence on the part of our community to declare RDRAND 'busted' at
> this point.
> 
> While it appeares to be trivially easy to force RDSEED into depletion,
> there does not seem to be a suggestion, at least in the open
> literature, that this directly or easily translates into stalling
> output from RDRAND in any type of relevant adversarial fashion.
> 
> If this were the case, given what CVE's seem to be worth on a resume,
> someone would have rented a cloud machine and come up with a POC
> against RDRAND in a multi-tenant environment and then promptly put up
> a web-site called 'Random Starve' or something equally ominous.
> 
> This is no doubt secondary to the 1022x amplication factor inherent in
> the 'Bull Mountain' architecture.
> 
> I'm a bit surprised that no one from the Intel side of this
> conversation didn't pitch this over the wall as soon as this
> conversation came up, but I would suggest that everyone concerned
> about this issue give the following a thorough read:
> 
> https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-
> random-number-generator-drng-software-implementation-guide.html
> 
> Relevant highlights:
> 
> - As I suggested in my earlier e-mail, random number generation is a
>   socket based resource, hence an adversarial domain limited to only
>   the cores on a common socket.
> 
> - There is a maximum randomness throughput rate of 800 MB/s over all
>   cores sharing common random number infrastructure.  Single thread
>   throughput rates of 70-200 MB/s are demonstratable.
> 
> - A failure of RDRAND over 10 re-tries is 'astronomically' small, with
>   no definition of astronomical provided, one would assume really
>   small, given they are using the word astronomical.

As I said, I want to investigate this properly before stating anything.
In a CoCo VM we cannot guarantee that a victim guest is able to execute
this 10 re-try loop (there is also a tightness requirement listed in official 
guide that is not further specified) without interruption since all guest
scheduling is under the host control. Again, this is the angle that was not
present before and I want to make sure we are protected against this case. 

> 
> > > That said, I have taken an AR to follow up internally on what can be done
> > > to improve our situation with RDRAND/RDSEED.
> 
> I think I can save you some time Elena.
> 
> > Specifying this is an interesting question. What exactly might our
> > requirements be for a "non-broken" RDRAND? It seems like we have two
> > basic ones:
> >
> > - One VMX (or host) context can't DoS another one.
> > - Ring 3 can't DoS ring 0.
> >
> > I don't know whether that'd be implemented with context-tied rate
> > limiting or more state or what. But I think, short of just making
> > RDRAND never fail, that's basically what's needed.
> 
> I think we probably have that, for all intents and purposes, given
> that we embrace the following methodogy:
> 
> - Use RDRAND exclusively.
> 
> - Be willing to take 10 swings at the plate.
> 
> - Given the somewhat demanding requirements for TDX/COCO, fail and
>   either deadlock or panic after 10 swings since that would seem to
>   suggest the hardware is broken, ie. RMA time.

Again, my worry here that a CoCo guest is not in control of its own scheduling
and this might make an impact on the above statement, i.e. it might
theoretical be possible to cause this without physically broken HW. 

Best Regards,
Elena.

> 
> Either deadlock or panic would be appropriate.  The objective in the
> COCO environment is to get the person who clicked on the 'Enable Azure
> Confidential' checkbox, or its equivalent, on their cloud dashboard,
> to call the HelpDesk and ask them why their confidential application
> won't come up.
> 
> After the user confirms to the HelpDesk that their computer is plugged
> in, the problem will get fixed.  Either the broken hardware will be
> identified and idled out or the mighty sword of vengeance will be
> summoned down on whoever has all of the other cores on the socket
> pegged.
> 
> Final thoughts:
> 
> - RDSEED is probably a poor thing to be using.
> 
> - There may be a reasonable argument that RDSEED shouldn't have been
>   exposed above ring 0, but that ship has sailed.  Brownie points
>   moving forward for an RDsomething that is ring 0 and has guaranteed
>   access to some amount of functionally reasonable entropy.
> 
> - Intel and AMD are already doing a lot of 'special' stuff with their
>   COCO hardware in order to defy the long standing adage of: 'You
>   can't have security without physical security'.  Access to per core thermal
>   noise, as I suggested, is probably a big lift but clever engineers can
>   probably cook up some type of fairness doctrine for randomness in
>   TDX or SEV_SNP, given the particular importance of instruction based
>   randomness in COCO.
> 
> - Perfection is the enemy of good.
> 
> > Jason
> 
> Have a good day.
> 
> As always,
> Dr. Greg
> 
> The Quixote Project - Flailing at the Travails of Cybersecurity
>               https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Thu, Feb 01, 2024 at 07:26:15AM +0000, Reshetova, Elena wrote:

Good morning to everyone.

> > On Wed, Jan 31, 2024 at 02:06:13PM +0100, Jason A. Donenfeld wrote:
> > 
> > Hi again to everyone, beautiful day here in North Dakota.
> > 
> > > On Wed, Jan 31, 2024 at 9:17???AM Reshetova, Elena
> > > <elena.reshetova@intel.com> wrote:
> > > > This matches both my understanding (I do have cryptography background
> > > > and understanding how cryptographic RNGs work)
> > > > and official public docs that Intel published on this matter.
> > > > Given that the physical entropy source is limited anyhow, and by giving
> > > > enough pressure on the whole construction you should be able to
> > > > make RDRAND fail because if the intermediate AES-CBC MAC extractor/
> > > > conditioner is not getting its min entropy input rate, it wont
> > > > produce a proper seed for AES CTR DRBG.
> > > > Of course exact details/numbers can wary between different generations of
> > > > Intel DRNG implementation, and the platforms where it is running on,
> > > > so be careful to sticking to concrete numbers.
> > 
> > > Alright, so RDRAND is not reliable. The question for us now is: do
> > > we want RDRAND unreliability to translate to another form of
> > > unreliability elsewhere, e.g. DoS/infiniteloop/latency/WARN_ON()? Or
> > > would it be better to declare the hardware simply broken and ask
> > > Intel to fix it? (I don't know the answer to that question.)
> > 
> > I think it would demonstrate a lack of appropriate engineering
> > diligence on the part of our community to declare RDRAND 'busted' at
> > this point.
> > 
> > While it appeares to be trivially easy to force RDSEED into depletion,
> > there does not seem to be a suggestion, at least in the open
> > literature, that this directly or easily translates into stalling
> > output from RDRAND in any type of relevant adversarial fashion.
> > 
> > If this were the case, given what CVE's seem to be worth on a resume,
> > someone would have rented a cloud machine and come up with a POC
> > against RDRAND in a multi-tenant environment and then promptly put up
> > a web-site called 'Random Starve' or something equally ominous.
> > 
> > This is no doubt secondary to the 1022x amplication factor inherent in
> > the 'Bull Mountain' architecture.
> > 
> > I'm a bit surprised that no one from the Intel side of this
> > conversation didn't pitch this over the wall as soon as this
> > conversation came up, but I would suggest that everyone concerned
> > about this issue give the following a thorough read:
> > 
> > https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-
> > random-number-generator-drng-software-implementation-guide.html
> > 
> > Relevant highlights:
> > 
> > - As I suggested in my earlier e-mail, random number generation is a
> >   socket based resource, hence an adversarial domain limited to only
> >   the cores on a common socket.
> > 
> > - There is a maximum randomness throughput rate of 800 MB/s over all
> >   cores sharing common random number infrastructure.  Single thread
> >   throughput rates of 70-200 MB/s are demonstratable.
> > 
> > - A failure of RDRAND over 10 re-tries is 'astronomically' small, with
> >   no definition of astronomical provided, one would assume really
> >   small, given they are using the word astronomical.

> As I said, I want to investigate this properly before stating
> anything.  In a CoCo VM we cannot guarantee that a victim guest is
> able to execute this 10 re-try loop (there is also a tightness
> requirement listed in official guide that is not further specified)
> without interruption since all guest scheduling is under the host
> control. Again, this is the angle that was not present before and I
> want to make sure we are protected against this case.

I suspect that all of this may be the source of interesting
discussions inside of Intel, see my closing question below.

If nothing else, we will wait with baited breath for a definition of
astronomical, if of course, the definition of that value is
unprivileged and you would be free to forward it along... :-)

> > > > That said, I have taken an AR to follow up internally on what can be done
> > > > to improve our situation with RDRAND/RDSEED.
> > 
> > I think I can save you some time Elena.
> > 
> > > Specifying this is an interesting question. What exactly might our
> > > requirements be for a "non-broken" RDRAND? It seems like we have two
> > > basic ones:
> > >
> > > - One VMX (or host) context can't DoS another one.
> > > - Ring 3 can't DoS ring 0.
> > >
> > > I don't know whether that'd be implemented with context-tied rate
> > > limiting or more state or what. But I think, short of just making
> > > RDRAND never fail, that's basically what's needed.
> > 
> > I think we probably have that, for all intents and purposes, given
> > that we embrace the following methodogy:
> > 
> > - Use RDRAND exclusively.
> > 
> > - Be willing to take 10 swings at the plate.
> > 
> > - Given the somewhat demanding requirements for TDX/COCO, fail and
> >   either deadlock or panic after 10 swings since that would seem to
> >   suggest the hardware is broken, ie. RMA time.

> Again, my worry here that a CoCo guest is not in control of its own
> scheduling and this might make an impact on the above statement,
> i.e. it might theoretical be possible to cause this without
> physically broken HW.

So all of this leaves open a very significant question that would seem
to be worthy of further enlightenment from inside the bowels of
Intel engineering.

Our discussion has now led us to a point where there appears to be a
legitimate concern that the hypervisor has such significant control
over a confidential VM that the integrity of a simple re-try loop is
an open question.

Let us posit for argument, that confidential computing resolves down
to the implementation of a trusted computing platform that in turn
resolves to a requirement for competent and robust cryptography for
initial and ongoing attestation, let alone confidentiality in the face
of possible side-channel and timing attacks.

I'm sure there would be a great deal of interest in any information
that can be provided that this scenario is possible, given the level
of control that is being suggested that a hypervisor would enjoy over
an ostensibly confidential and trusted guest.

> Best Regards,
> Elena.

Have a good day.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Wed, Jan 31, 2024 at 08:16:56AM +0000, Reshetova, Elena wrote:

Good evening, I hope the week has started well for everyone.

> > On Tue, Jan 30, 2024 at 06:49:15PM +0100, Jason A. Donenfeld wrote:
> > > On Tue, Jan 30, 2024 at 6:32???PM Dave Hansen <dave.hansen@intel.com> wrote:
> > > >
> > > > On 1/30/24 05:45, Reshetova, Elena wrote:
> > > > >> You're the Intel employee so you can find out about this with much
> > > > >> more assurance than me, but I understand the sentence above to be _way
> > > > >> more_ true for RDRAND than for RDSEED. If your informed opinion is,
> > > > >> "RDRAND failing can only be due to totally broken hardware"
> > > > > No, this is not the case per Intel SDM. I think we can live under a simple
> > > > > assumption that both of these instructions can fail not just due to broken
> > > > > HW, but also due to enough pressure put into the whole DRBG construction
> > > > > that supplies random numbers via RDRAND/RDSEED.
> > > >
> > > > I don't think the SDM is the right thing to look at for guidance here.
> > > >
> > > > Despite the SDM allowing it, we (software) need RDRAND/RDSEED failures
> > > > to be exceedingly rare by design.  If they're not, we're going to get
> > > > our trusty torches and pitchforks and go after the folks who built the
> > > > broken hardware.
> > > >
> > > > Repeat after me:
> > > >
> > > >         Regular RDRAND/RDSEED failures only occur on broken hardware
> > > >
> > > > If it's nice hardware that's gone bad, then we WARN() and try to make
> > > > the best of it.  If it turns out that WARN() was because of a broken
> > > > hardware _design_ then we go sharpen the pitchforks.
> > > >
> > > > Anybody disagree?
> > >
> > > Yes, I disagree. I made a trivial test that shows RDSEED breaks easily
> > > in a busy loop. So at the very least, your statement holds true only
> > > for RDRAND.
> > >
> > > But, anyway, if the statement "RDRAND failures only occur on broken
> > > hardware" is true, then a WARN() in the failure path there presents no
> > > DoS potential of any kind, and so that's a straightforward conclusion
> > > to this discussion. However, that really hinges on  "RDRAND failures
> > > only occur on broken hardware" being a true statement.
> > 
> > There's a useful comment here from an Intel engineer
> > 
> > https://web.archive.org/web/20190219074642/https://software.intel.com/en-
> > us/blogs/2012/11/17/the-difference-between-rdrand-and-rdseed
> > 
> >   "RDRAND is, indeed, faster than RDSEED because it comes
> >    from a hardware-based pseudorandom number generator.
> >    One seed value (effectively, the output from one RDSEED
> >    command) can provide up to 511 128-bit random values
> >    before forcing a reseed"
> > 
> > We know we can exhaust RDSEED directly pretty trivially. Making your
> > test program run in parallel across 20 cpus, I got a mere 3% success
> > rate from RDSEED.
> > 
> > If RDRAND is reseeding every 511 values, RDRAND output would have
> > to be consumed significantly faster than RDSEED in order that the
> > reseed will happen frequently enough to exhaust the seeds.
> > 
> > This looks pretty hard, but maybe with a large enough CPU count
> > this will be possible in extreme load ?
> > 
> > So I'm not convinced we can blindly wave away RDRAND failures as
> > guaranteed to mean broken hardware.

> This matches both my understanding (I do have cryptography
> background and understanding how cryptographic RNGs work) and
> official public docs that Intel published on this matter.  Given
> that the physical entropy source is limited anyhow, and by giving
> enough pressure on the whole construction you should be able to make
> RDRAND fail because if the intermediate AES-CBC MAC extractor/
> conditioner is not getting its min entropy input rate, it wont
> produce a proper seed for AES CTR DRBG.  Of course exact
> details/numbers can wary between different generations of Intel DRNG
> implementation, and the platforms where it is running on, so be
> careful to sticking to concrete numbers.

In the spirit of that philosophy we proffer the response below.

> That said, I have taken an AR to follow up internally on what can be
> done to improve our situation with RDRAND/RDSEED. But I would still
> like to finish the discussion on what people think should be done in
> the meanwhile keeping in mind that the problem is not intel
> specific, despite us intel people bringing it for public discussion
> first. The old saying is still here: "Please don't shoot the
> messenger" )) We are actually trying to be open about these things
> and create a public discussion.

Actually, I now believe there is clear evidence that the problem is
indeed Intel specific.  In light of our testing, it will be
interesting to see what your 'AR' returns with respect to an official
response from Intel engineering on this issue.

One of the very bright young engineers collaborating on Quixote, who
has been following this conversation, took it upon himself to do some
very methodical engineering analysis on this issue.  I'm the messenger
but this is very much his work product.

Executive summary is as follows:

- No RDRAND depletion failures were observable with either the Intel
  or AMD hardware that was load tested.

- RDSEED depletion is an Intel specific issue, AMD's RDSEED
  implementation could not be provoked into failure.

- AMD's RDRAND/RDSEED implementation is significantly slower than
  Intel's.

Here are the engineer's lab notes verbatim:

---------------------------------------------------------------------------
I tested both the single-threaded and OMP-multithreaded versions of
the RDSEED/RDRAND depletion loop on each of the machines below.

AMD: 2X AMD EPYC 7713 (Milan) 64-Core Processor @ 2.0 GHz, 128
physical cores total

Intel: 2X Intel Xeon Gold 6140 (Skylake) 18-Core Processor @ 2.3 GHz,
36 physical cores total

Single-threaded results:

Test case: 1,000,000 iterations each for RDRAND and RDSEED, n=100
tests, single-threaded.

AMD: 100% success rate for both RDRAND and RDSEED for all tests,
runtime 0.909-1.055s (min-max).

Intel: 100% success rate for RDRAND for all tests, 20.01-20.12%
(min-max) success rate for RSEED, runtime 0.256-0.281s (min-max)

OMP multithreaded results:

Test case: 1,000,000 iterations per thread, for both RDRAND and
RDSEED, n=100 tests, OMP multithreaded with OMP_NUM_THREADS=<total
physical cores> (i.e. 128 for AMD and 36 for Intel)

AMD: 100% success rate for both RDRAND and RDSEED for all tests,
runtime 47.229-47.603s (min-max).

Intel: 100% success rate for RDRAND for all tests, 1.77-5.62%
(min-max) success rate for RSEED, runtime 0.562-0.595s (min-max)

CONCLUSION

RDSEED failure was reproducibly induced on the Intel Skylake platform,
for both single- and multithreaded tests, whereas RDSEED failure could
not be induced on the AMD platform for either test. RDRAND did not
fail on either platform for either test.

AMD execution time was roughly 4x slower than Intel (1s vs 0.25s) for
the single-threaded test, and almost 100x slower than Intel (47s vs
0.5s) for the multithreaded test. The difference in clock rates (2.0
GHz for AMD vs 2.3 GHz for Intel) is not sufficient to explain these
runtime differences. So it seems likely that AMD is gating the rate at
which a new RDSEED value can be requested.
---------------------------------------------------------------------------

Speaking now with my voice:

Unless additional information shows up, despite our collective
handwringing, as long as the RDRAND instruction is used as the
cryptographic primitive, there appears to be little likelihood of a
DOS randomness attack against a TDX based CoCo virtual machine.

While it is highly unlikely we will ever get an 'official' readout on
this issue, I suspect there is a high probability that Intel
engineering favored performance with their RDSEED/RDRAND
implementation.

AMD 'appears', and without engineering feedback from AMD I would
emphasize the notion of 'appears', to have embraced the principal of
taking steps to eliminate the possibility of a socket based adversary
attack against their RNG infrastructure.

> Elena.

Hopefully the above is useful for everyone interested in this issue.

Once again, a thank you to our version of 'Sancho' for his legwork on
this, who has also read Cervantes at length... :-)

Have a good remainder of the week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Daniel P. Berrangé]

On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> 
> Actually, I now believe there is clear evidence that the problem is
> indeed Intel specific.  In light of our testing, it will be
> interesting to see what your 'AR' returns with respect to an official
> response from Intel engineering on this issue.
> 
> One of the very bright young engineers collaborating on Quixote, who
> has been following this conversation, took it upon himself to do some
> very methodical engineering analysis on this issue.  I'm the messenger
> but this is very much his work product.
> 
> Executive summary is as follows:
> 
> - No RDRAND depletion failures were observable with either the Intel
>   or AMD hardware that was load tested.
> 
> - RDSEED depletion is an Intel specific issue, AMD's RDSEED
>   implementation could not be provoked into failure.

My colleague ran a multithread parallel stress test program on
his 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate
in RDSEED.

> - AMD's RDRAND/RDSEED implementation is significantly slower than
>   Intel's.

Yes, we also noticed the AMD impl is horribly slow compared to Intel,
had to cut test iterations x100

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:

Good morning to everyone.

> On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> > 
> > Actually, I now believe there is clear evidence that the problem is
> > indeed Intel specific.  In light of our testing, it will be
> > interesting to see what your 'AR' returns with respect to an official
> > response from Intel engineering on this issue.
> > 
> > One of the very bright young engineers collaborating on Quixote, who
> > has been following this conversation, took it upon himself to do some
> > very methodical engineering analysis on this issue.  I'm the messenger
> > but this is very much his work product.
> > 
> > Executive summary is as follows:
> > 
> > - No RDRAND depletion failures were observable with either the Intel
> >   or AMD hardware that was load tested.
> > 
> > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
> >   implementation could not be provoked into failure.

> My colleague ran a multithread parallel stress test program on his
> 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
> RDSEED.

Interesting datapoint, thanks for forwarding it along, so the issue
shows up on at least some AMD platforms as well.

On the 18 core/socket Intel Skylake platform, the parallelized
depletion test forces RDSEED success rates down to around 2%.  It
would appear that your tests suggest that the AMD platform fairs
better than the Intel platform.

So this is turning into even more of a morass, given that RDSEED
depletion on AMD may be a function of the micro-architecture the
platform is based on.  The other variable is that our AMD test
platform had a substantially higher core count per socket, one would
assume that would result in higher depletion rates, if the operative
theory of socket common RNG infrastructure is valid.

Unless AMD engineering understands the problem and has taken some type
of action on higher core count systems to address the issue.

Of course, the other variable may be how the parallelized stress test
is conducted.  If you would like to share your implementation source
we could give it a twirl on the systems we have access to.

The continuing operative question is whether or not any of this ever
leads to an RDRAND failure.

We've conducted some additional tests on the Intel platform where
RDSEED depletion was driven low as possible, ~1-2% success rates,
while RDRAND depletion tests were being run simultaneously.  No RDRAND
failures have been noted.

So the operative question remains, why worry about this if RDRAND is
used as the randomness primitive.

We haven't seen anything out of Intel yet on this, maybe AMD has a
quantifying definition for 'astronomical' when it comes to RDRAND
failures.

The silence appears to be deafening out of the respective engineering
camps... :-)

> > - AMD's RDRAND/RDSEED implementation is significantly slower than
> >   Intel's.

> Yes, we also noticed the AMD impl is horribly slow compared to
> Intel, had to cut test iterations x100.

The operative question is the impact of 'slow', in the absence of
artifical stress tests.

It would seem that a major question is what are or were the
engineering thought processes on the throughput of the hardware
randomness instructions.

Intel documents the following randomness throughput rates:

RDSEED: 3 Gbit/second
RDRAND: 6.4 Gbit/second

If there is the possibility of over-harvesting randomness, why not
design the implementations to be clamped at some per core value such
as a megabit/second.  In the case of the documented RDSEED generation
rates, that would allow the servicing of 3222 cores, if my math at
0530 in the morning is correct.

Would a core need more than 128 kilobytes of randomness, ie. one
second of output, to effectively seed a random number generator?

A cynical conclusion would suggest engineering acquiesing to marketing
demands... :-)

> With regards,
> Daniel

Have a good day.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Daniel P. Berrangé]

On Tue, Feb 06, 2024 at 06:04:45AM -0600, Dr. Greg wrote:
> On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:
> 
> Good morning to everyone.
> 
> > On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> > > 
> > > Actually, I now believe there is clear evidence that the problem is
> > > indeed Intel specific.  In light of our testing, it will be
> > > interesting to see what your 'AR' returns with respect to an official
> > > response from Intel engineering on this issue.
> > > 
> > > One of the very bright young engineers collaborating on Quixote, who
> > > has been following this conversation, took it upon himself to do some
> > > very methodical engineering analysis on this issue.  I'm the messenger
> > > but this is very much his work product.
> > > 
> > > Executive summary is as follows:
> > > 
> > > - No RDRAND depletion failures were observable with either the Intel
> > >   or AMD hardware that was load tested.
> > > 
> > > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
> > >   implementation could not be provoked into failure.
> 
> > My colleague ran a multithread parallel stress test program on his
> > 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
> > RDSEED.
> 
> Interesting datapoint, thanks for forwarding it along, so the issue
> shows up on at least some AMD platforms as well.
> 
> On the 18 core/socket Intel Skylake platform, the parallelized
> depletion test forces RDSEED success rates down to around 2%.  It
> would appear that your tests suggest that the AMD platform fairs
> better than the Intel platform.

Yes, given the speed of the AMD RDRAND/RDSEED ops, compared to my
Intel test platforms, their DRBG looks better able to keep up with
the demand for bits.

> Of course, the other variable may be how the parallelized stress test
> is conducted.  If you would like to share your implementation source
> we could give it a twirl on the systems we have access to.

It is just Jason's earlier test program, but moved into one thread
for each core....

$ cat cpurngstress.c
#include <stdio.h>
#include <immintrin.h>
#include <pthread.h>
#include <unistd.h>

/*
 * Gives about 25 seconds walllock time on my Alderlake CPU
 *
 * Probably want to reduce this x10, or possibly even x100
 * on AMD due to much slower ops.
 */
#define MAX_ITER 10000000

#define MAX_CPUS 4096

void *doit(void *f) {
    unsigned long long rand;
    unsigned int i, success_rand = 0, success_seed = 0;

    for (i = 0; i < MAX_ITER; ++i) {
        success_seed += !!_rdseed64_step(&rand);
    }
    for (i = 0; i < MAX_ITER; ++i) {
        success_rand += !!_rdrand64_step(&rand);
    }

    fprintf(stderr,
	    "RDRAND: %.2f%%, RDSEED: %.2f%%\n",
	    success_rand * 100.0 / MAX_ITER,
	    success_seed * 100.0 / MAX_ITER);

    return NULL;
}


int main(int argc, char *argv[])
{
    pthread_t th[MAX_CPUS];
    int nproc = sysconf(_SC_NPROCESSORS_ONLN);
    if (nproc > MAX_CPUS) {
      nproc = MAX_CPUS;
    }
    fprintf(stderr, "Stressing RDRAND/RDSEED across %d CPUs\n", nproc);

    for (int i = 0 ; i < nproc;i ++) {
      pthread_create(&th[i], NULL, doit,NULL);
    }

    for (int i = 0 ; i < nproc;i ++) {
      pthread_join(th[i], NULL);
    }

    return 0;
}

$ gcc -march=native -o cpurngstress cpurngstress.c


> If there is the possibility of over-harvesting randomness, why not
> design the implementations to be clamped at some per core value such
> as a megabit/second.  In the case of the documented RDSEED generation
> rates, that would allow the servicing of 3222 cores, if my math at
> 0530 in the morning is correct.
> 
> Would a core need more than 128 kilobytes of randomness, ie. one
> second of output, to effectively seed a random number generator?
> 
> A cynical conclusion would suggest engineering acquiesing to marketing
> demands... :-)

My assumption is that it was simply easier to not implement a
rate limiting feature at the CPU level and punt the starvation
problem to software :-)

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Tue, Feb 06, 2024 at 01:00:03PM +0000, Daniel P. Berrang?? wrote:

Good morning.

> On Tue, Feb 06, 2024 at 06:04:45AM -0600, Dr. Greg wrote:
> > On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:
> > 
> > Good morning to everyone.
> > 
> > > On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> > > > 
> > > > Actually, I now believe there is clear evidence that the problem is
> > > > indeed Intel specific.  In light of our testing, it will be
> > > > interesting to see what your 'AR' returns with respect to an official
> > > > response from Intel engineering on this issue.
> > > > 
> > > > One of the very bright young engineers collaborating on Quixote, who
> > > > has been following this conversation, took it upon himself to do some
> > > > very methodical engineering analysis on this issue.  I'm the messenger
> > > > but this is very much his work product.
> > > > 
> > > > Executive summary is as follows:
> > > > 
> > > > - No RDRAND depletion failures were observable with either the Intel
> > > >   or AMD hardware that was load tested.
> > > > 
> > > > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
> > > >   implementation could not be provoked into failure.
> > 
> > > My colleague ran a multithread parallel stress test program on his
> > > 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
> > > RDSEED.
> > 
> > Interesting datapoint, thanks for forwarding it along, so the issue
> > shows up on at least some AMD platforms as well.
> > 
> > On the 18 core/socket Intel Skylake platform, the parallelized
> > depletion test forces RDSEED success rates down to around 2%.  It
> > would appear that your tests suggest that the AMD platform fairs
> > better than the Intel platform.

> Yes, given the speed of the AMD RDRAND/RDSEED ops, compared to my
> Intel test platforms, their DRBG looks better able to keep up with
> the demand for bits.

We now believe the observed resiliency of AMD's RNG infrastructure
comes down to the fact that the completion times of their RNG
instructions are significantly slower than Intel's.

SkyLake and KabyLake instruction completion times are documented at
463 clock cycles, regardless of operand size.

AMD Ryzen documents variable completion times based on operand size.
16 and 32 bit transfers complete in 1200 clock cycles with 64 bit
requests completing in 2500 clock cycles.

Given that Jason's test program was issueing 64-bit RNG requests, the
AMD platforms are going to be approximately 5.4 times slower than
Intel platforms, provided the results are corrected for CPU clock
rates.

AMD's entropy source is execution jitter time over a bank of inverter
based ring oscillors, presumably sampled by a constant clock rate
sampler.  Slower instruction retirement times consumes less of the
constant rate entropy production.

Intel uses thermal/quantum noise across a diode junction retrieved by
a self-clocked sampler.  Faster instruction retirement translates into
increased bandwidth demands on the sampler.

> > Of course, the other variable may be how the parallelized stress test
> > is conducted.  If you would like to share your implementation source
> > we could give it a twirl on the systems we have access to.
> 
> It is just Jason's earlier test program, but moved into one thread
> for each core....
> 
> $ cat cpurngstress.c
> #include <stdio.h>
> #include <immintrin.h>
> #include <pthread.h>
> #include <unistd.h>
> 
> /*
>  * Gives about 25 seconds walllock time on my Alderlake CPU
>  *
>  * Probably want to reduce this x10, or possibly even x100
>  * on AMD due to much slower ops.
>  */
> #define MAX_ITER 10000000
> 
> #define MAX_CPUS 4096
> 
> void *doit(void *f) {
>     unsigned long long rand;
>     unsigned int i, success_rand = 0, success_seed = 0;
> 
>     for (i = 0; i < MAX_ITER; ++i) {
>         success_seed += !!_rdseed64_step(&rand);
>     }
>     for (i = 0; i < MAX_ITER; ++i) {
>         success_rand += !!_rdrand64_step(&rand);
>     }
> 
>     fprintf(stderr,
> 	    "RDRAND: %.2f%%, RDSEED: %.2f%%\n",
> 	    success_rand * 100.0 / MAX_ITER,
> 	    success_seed * 100.0 / MAX_ITER);
> 
>     return NULL;
> }
> 
> 
> int main(int argc, char *argv[])
> {
>     pthread_t th[MAX_CPUS];
>     int nproc = sysconf(_SC_NPROCESSORS_ONLN);
>     if (nproc > MAX_CPUS) {
>       nproc = MAX_CPUS;
>     }
>     fprintf(stderr, "Stressing RDRAND/RDSEED across %d CPUs\n", nproc);
> 
>     for (int i = 0 ; i < nproc;i ++) {
>       pthread_create(&th[i], NULL, doit,NULL);
>     }
> 
>     for (int i = 0 ; i < nproc;i ++) {
>       pthread_join(th[i], NULL);
>     }
> 
>     return 0;
> }
> 
> $ gcc -march=native -o cpurngstress cpurngstress.c

Thanks for forwarding your test code along, we've added it to our
tests for comparison.

> > If there is the possibility of over-harvesting randomness, why not
> > design the implementations to be clamped at some per core value such
> > as a megabit/second.  In the case of the documented RDSEED generation
> > rates, that would allow the servicing of 3222 cores, if my math at
> > 0530 in the morning is correct.
> > 
> > Would a core need more than 128 kilobytes of randomness, ie. one
> > second of output, to effectively seed a random number generator?
> > 
> > A cynical conclusion would suggest engineering acquiesing to marketing
> > demands... :-)

> My assumption is that it was simply easier to not implement a rate
> limiting feature at the CPU level and punt the starvation problem to
> software :-)

Could be, it does seem unlikely that random number generation speed
would be seen as fertile ground for marketing types.

Punting to software is certainly rationale, perhaps problematic in a
CoCo environment depending on the definition of 'astronomical'.  See
my response to Borislav who was kind enough to respond to all of this.

> With regards,
> Daniel

Have a good day.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Daniel P. Berrangé]

On Tue, Feb 06, 2024 at 06:04:45AM -0600, Dr. Greg wrote:
> On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:
> 
> Good morning to everyone.
> 
> > On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> > > 
> > > Actually, I now believe there is clear evidence that the problem is
> > > indeed Intel specific.  In light of our testing, it will be
> > > interesting to see what your 'AR' returns with respect to an official
> > > response from Intel engineering on this issue.
> > > 
> > > One of the very bright young engineers collaborating on Quixote, who
> > > has been following this conversation, took it upon himself to do some
> > > very methodical engineering analysis on this issue.  I'm the messenger
> > > but this is very much his work product.
> > > 
> > > Executive summary is as follows:
> > > 
> > > - No RDRAND depletion failures were observable with either the Intel
> > >   or AMD hardware that was load tested.
> > > 
> > > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
> > >   implementation could not be provoked into failure.
> 
> > My colleague ran a multithread parallel stress test program on his
> > 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
> > RDSEED.
> 
> Interesting datapoint, thanks for forwarding it along, so the issue
> shows up on at least some AMD platforms as well.

I got access to a couple more AMD machines. An EPYC 24core/2HT
(Zen-1 uarch) and an EPYC 2socket/16core/2HT (Zen-3 uarch).

Both of these show 100% success with RDSEED. So there's clearly
some variance across AMD SKUs. So perhaps this is an EPYC vs Ryzen
distinction, with the server focused EPYCs able to sustain RDSEED.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Borislav Petkov]

On Tue, Feb 06, 2024 at 06:04:45AM -0600, Dr. Greg wrote:
> The silence appears to be deafening out of the respective engineering
> camps... :-)

I usually wait for those threads to "relax" themselves first. :)

So, what do you wanna know?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Tue, Feb 06, 2024 at 04:35:29PM +0100, Borislav Petkov wrote:

Good morning, or perhaps afternoon, thanks for taking the time to
reply.

> On Tue, Feb 06, 2024 at 06:04:45AM -0600, Dr. Greg wrote:
> > The silence appears to be deafening out of the respective engineering
> > camps... :-)

> I usually wait for those threads to "relax" themselves first. :)

Indeed, my standard practice is to wait 24 hours before replying to
any public e-mail, hence the delay in my response.

> So, what do you wanna know?

I guess a useful starting point would be if AMD would like to offer
any type of quantification for 'astronomically small' when it comes to
the probability of failure over 10 RDRAND attempts... :-)

Secondly, given our test findings and those of RedHat, would it be
safe to assume that EPYC has engineering that prevents RDSEED failures
that Ryzen does not?

Given HPA's response in this thread, I do appreciate that all of this
may be shrouded in trade secrets and other issues.  With an
acknowledgement to that fact, let me see if I can extend the
discussion in a generic manner that may prove useful to the community
without being 'abusive'.

Both AMD and Intel designs start with a hardware based entropy source.
Intel samples thermal/quantum junction noise, AMD samples execution
jitter over a bank of inverter based oscillators.  An assumption of
constant clocked sampling implies a maximum randomness bandwidth
limit.

None of this implies that randomness is a finite resource, it will
always become available, with the caveat that a core may have to stand
in line, cup in hand, waiting for a dollop.

So this leaves the fundamental question of what does an RDRAND or
RDSEED failure return actually imply?

Silicon is a expensive resource, which would imply a queue depth
limitation for access to the socket common RNG infastructure.  If the
queue is full when an instruction issues, it would be a logical
response to signal an instruction failure quickly and let software try
again.

An alternate theory would be a requirement for constant instruction
time completion.  In that case a 'buffer' of cycles would be included
in the RNG instruction cycle allocation count.  If the instruction
would need to 'sleep', waiting for randomness, beyond this cycle
buffer, a failure would be returned.

Absent broken hardware, astronomical then becomes the probability of a
core being unlucky enough to run into these or alternate
implementation scenarios 10 times in a row.  Particularly given the
recommendation to sleep between attempts, which implies getting
scheduled onto different cores for the attempts.

Any enlightenment along these lines would seem to be useful in
facilitating an understanding of the issues at hand.

Given the time and engineering invested in the engineering behind both
TDX and SEV-SNP, it would seem unlikely that really smart engineers at
both Intel and AMD didn't anticipate this issue and its proper
resolution for CoCo environments.

> Regards/Gruss,
>     Boris.
> 
> https://people.kernel.org/tglx/notes-about-netiquette

All the best from the Upper Midwest.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Borislav Petkov]

On Thu, Feb 08, 2024 at 05:44:44AM -0600, Dr. Greg wrote:
> I guess a useful starting point would be if AMD would like to offer
> any type of quantification for 'astronomically small' when it comes to
> the probability of failure over 10 RDRAND attempts... :-)

Right, let's establish the common ground first: please have a look at
this, albeit a bit outdated whitepaper:

https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/white-papers/amd-random-number-generator.pdf

in case you haven't seen it yet.

Now, considering that this is a finite resource, you can imagine that
there can be scenarios where that source can be depleted.

And newer Zen generations perform significantly better. So much so that
on Zen3 and later 10 retries should never observe a failure unless it
is bad hardware. Also, I agree with hpa's note that any and all retries
should be time based.

> Secondly, given our test findings and those of RedHat, would it be
> safe to assume that EPYC has engineering that prevents RDSEED failures
> that Ryzen does not?

Well, roughly speaking, client is a less beefier and less performant
version of server. You can extrapolate that to the topic at hand.

But at least on AMD, any potential DoSing of RDRAND on client doesn't
matter for CoCo because client doesn't enable SEV*.

> Both AMD and Intel designs start with a hardware based entropy source.
> Intel samples thermal/quantum junction noise, AMD samples execution
> jitter over a bank of inverter based oscillators.

See above paper for the AMD side.

> An assumption of constant clocked sampling implies a maximum
> randomness bandwidth limit.

You said it.

> None of this implies that randomness is a finite resource

Huh? This contradicts with what you just said in the above sentence.

Or maybe I'm reading this wrong...

> So this leaves the fundamental question of what does an RDRAND or
> RDSEED failure return actually imply?

Simple: if no random data is ready at the time the insn executes, it
says "invalid". Because the generator is a finite resource as you said
above, if the software tries to pull random data faster than it can
generate, this is the main case for CF=0.

> Silicon is a expensive resource, which would imply a queue depth
> limitation for access to the socket common RNG infastructure.  If the
> queue is full when an instruction issues, it would be a logical
> response to signal an instruction failure quickly and let software try
> again.

That's actually in the APM documenting RDRAND:

"If the returned value is invalid, software must execute the instruction
again."

> Given the time and engineering invested in the engineering behind both
> TDX and SEV-SNP, it would seem unlikely that really smart engineers at
> both Intel and AMD didn't anticipate this issue and its proper
> resolution for CoCo environments.

You can probably imagine that no one can do a fully secure system in one
single attempt but rather needs to do an iterative process.

And I don't know how much you've followed those technologies but they
*are* the perfect example for such an iterative improvement process.

I hope this answers at least some of your questions.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Jason A. Donenfeld]

Hey Boris,

While you're here, I was wondering if you could comment on one thing related:

On Fri, Feb 9, 2024 at 6:31 PM Borislav Petkov <bp@alien8.de> wrote:
> Now, considering that this is a finite resource, you can imagine that
> there can be scenarios where that source can be depleted.

Yea, this makes sense.

[As an aside, I would like to note that a different construction of
RDRAND could keep outputting good random numbers for a reeeeeallly
long time without needing to reseed, or without penalty if RDSEED is
depleted, and so could be made to actually never fail. But given the
design goals of RDRAND, this kind of crypto is highly likely to never
be implemented, so I'm not even moving to suggest that AMD/Intel just
'fix' the crypto design goals of the instruction. It's not gonna
happen for lots of reasons.]

So assuming that RDSEED and hence RDRAND can never be made to never
fail, the options are:

1. Finite resource that refills faster than whatever instruction
issuance latency, so it's never observably empty. (Seems unlikely)
2. More secure sharing of the finite resource.

It's this second option I wanted to ask you about. I wrote down what I
thought "secure sharing" meant here [1]:

> - One VMX (or host) context can't DoS another one.
> - Ring 3 can't DoS ring 0.

It's a bit of a scheduling/queueing thing, where different security
contexts shouldn't be able to starve others out of the finite resource
indefinitely.

What I'm wondering is if that kind of fairness is even possible to
achieve in the hardware or the microcode. I don't really know how that
all works under the covers and what sorts of "policies" and such are
feasible to implement. In suggesting it, I feel like a bit of a
presumptuous kernel developer talking to hardware people, not fully
appreciating their domain and its challenges. For, if this were just a
C program, I know exactly what I'd do, but we're talking about a CPU
here.

Is it actually possible to make RDRAND usage "fair" between different
security contexts? Or am I totally delusional and this is not how the
hardware works or can ever work?

Jason

[1] https://lore.kernel.org/all/CAHmME9ps6W5snQrYeNVMFgfhMKFKciky=-UxxGFbAx_RrxSHoA@mail.gmail.com/

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dave Hansen]

On 2/9/24 11:49, Jason A. Donenfeld wrote:
> [As an aside, I would like to note that a different construction of
> RDRAND could keep outputting good random numbers for a reeeeeallly
> long time without needing to reseed, or without penalty if RDSEED is
> depleted, and so could be made to actually never fail. But given the
> design goals of RDRAND, this kind of crypto is highly likely to never
> be implemented, so I'm not even moving to suggest that AMD/Intel just
> 'fix' the crypto design goals of the instruction. It's not gonna
> happen for lots of reasons.]

Intel's RDRAND reseeding behavior is spelled out here:

> https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html

In the "Guaranteeing DBRG Reseeding" section.

> It's a bit of a scheduling/queueing thing, where different security
> contexts shouldn't be able to starve others out of the finite resource
> indefinitely.
> 
> What I'm wondering is if that kind of fairness is even possible to
> achieve in the hardware or the microcode.
...

Even ignoring different security contexts, Intel's whitepaper claims
that no starvation happens with RDRAND:

> If multiple threads are invoking RDRAND simultaneously, total RDRAND
> throughput (across all threads) scales approximately linearly with
> the number of threads until no more hardware threads remain, the bus
> limits of the processor are reached, or the DRNG interface is fully
> saturated. Past this point, the maximum throughput is divided equally
> among the active threads. No threads get starved.

800 MB/sec of total RDRAND throughput across all threads, guaranteed
reseeding, and no starvation sounds pretty good to me.

Does that need improving?

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Borislav Petkov]

On Fri, Feb 09, 2024 at 08:49:40PM +0100, Jason A. Donenfeld wrote:
> While you're here,

I was here the whole time, lurking in the shadows. :)

> Is it actually possible to make RDRAND usage "fair" between different
> security contexts? Or am I totally delusional and this is not how the
> hardware works or can ever work?

Yeah, I know exactly what you mean and I won't go into details for
obvious reasons. Two things:

* Starting with Zen3, provided properly configured hw RDRAND will never
fail. It is also fair when feeding the different contexts.

* My hardware engineers tell me that this is tough to do for RDSEED

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[H. Peter Anvin]

On February 6, 2024 4:04:45 AM PST, "Dr. Greg" <greg@enjellic.com> wrote:
>On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:
>
>Good morning to everyone.
>
>> On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
>> > 
>> > Actually, I now believe there is clear evidence that the problem is
>> > indeed Intel specific.  In light of our testing, it will be
>> > interesting to see what your 'AR' returns with respect to an official
>> > response from Intel engineering on this issue.
>> > 
>> > One of the very bright young engineers collaborating on Quixote, who
>> > has been following this conversation, took it upon himself to do some
>> > very methodical engineering analysis on this issue.  I'm the messenger
>> > but this is very much his work product.
>> > 
>> > Executive summary is as follows:
>> > 
>> > - No RDRAND depletion failures were observable with either the Intel
>> >   or AMD hardware that was load tested.
>> > 
>> > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
>> >   implementation could not be provoked into failure.
>
>> My colleague ran a multithread parallel stress test program on his
>> 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
>> RDSEED.
>
>Interesting datapoint, thanks for forwarding it along, so the issue
>shows up on at least some AMD platforms as well.
>
>On the 18 core/socket Intel Skylake platform, the parallelized
>depletion test forces RDSEED success rates down to around 2%.  It
>would appear that your tests suggest that the AMD platform fairs
>better than the Intel platform.
>
>So this is turning into even more of a morass, given that RDSEED
>depletion on AMD may be a function of the micro-architecture the
>platform is based on.  The other variable is that our AMD test
>platform had a substantially higher core count per socket, one would
>assume that would result in higher depletion rates, if the operative
>theory of socket common RNG infrastructure is valid.
>
>Unless AMD engineering understands the problem and has taken some type
>of action on higher core count systems to address the issue.
>
>Of course, the other variable may be how the parallelized stress test
>is conducted.  If you would like to share your implementation source
>we could give it a twirl on the systems we have access to.
>
>The continuing operative question is whether or not any of this ever
>leads to an RDRAND failure.
>
>We've conducted some additional tests on the Intel platform where
>RDSEED depletion was driven low as possible, ~1-2% success rates,
>while RDRAND depletion tests were being run simultaneously.  No RDRAND
>failures have been noted.
>
>So the operative question remains, why worry about this if RDRAND is
>used as the randomness primitive.
>
>We haven't seen anything out of Intel yet on this, maybe AMD has a
>quantifying definition for 'astronomical' when it comes to RDRAND
>failures.
>
>The silence appears to be deafening out of the respective engineering
>camps... :-)
>
>> > - AMD's RDRAND/RDSEED implementation is significantly slower than
>> >   Intel's.
>
>> Yes, we also noticed the AMD impl is horribly slow compared to
>> Intel, had to cut test iterations x100.
>
>The operative question is the impact of 'slow', in the absence of
>artifical stress tests.
>
>It would seem that a major question is what are or were the
>engineering thought processes on the throughput of the hardware
>randomness instructions.
>
>Intel documents the following randomness throughput rates:
>
>RDSEED: 3 Gbit/second
>RDRAND: 6.4 Gbit/second
>
>If there is the possibility of over-harvesting randomness, why not
>design the implementations to be clamped at some per core value such
>as a megabit/second.  In the case of the documented RDSEED generation
>rates, that would allow the servicing of 3222 cores, if my math at
>0530 in the morning is correct.
>
>Would a core need more than 128 kilobytes of randomness, ie. one
>second of output, to effectively seed a random number generator?
>
>A cynical conclusion would suggest engineering acquiesing to marketing
>demands... :-)
>
>> With regards,
>> Daniel
>
>Have a good day.
>
>As always,
>Dr. Greg
>
>The Quixote Project - Flailing at the Travails of Cybersecurity
>              https://github.com/Quixote-Project

You do realize, right, that the "deafening silence" is due to the need for research and discussions on our part, and presumably AMD's.

In addition, quite frankly, your rather abusive language isn't exactly encouraging people to speak publicly based on immediately available and therefore inherently incomplete and/or dated information, meaning that we have had to take even what discussions we might have been able to have in public without IP concerns behind the scenes.

Yes, we work for Intel. No, we don't know every detail about every Intel chip ever created off the top of my head, nor do we necessarily know the exact person that is *currently* in charge of the architecture of a particular unit, nor is it necessarily true that even *that* person knows all the exact properties of the behavior of their unit when integrated into a particular SoC, as units are modular by design.

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Dr. Greg]

On Tue, Feb 06, 2024 at 10:49:59AM -0800, H. Peter Anvin wrote:

Good morning HPA, I hope your week is going well, thanks for taking
the time to extend comments.

> On February 6, 2024 4:04:45 AM PST, "Dr. Greg" <greg@enjellic.com> wrote:
> >On Tue, Feb 06, 2024 at 08:04:57AM +0000, Daniel P. Berrang?? wrote:
> >
> >Good morning to everyone.
> >
> >> On Mon, Feb 05, 2024 at 07:12:47PM -0600, Dr. Greg wrote:
> >> > 
> >> > Actually, I now believe there is clear evidence that the problem is
> >> > indeed Intel specific.  In light of our testing, it will be
> >> > interesting to see what your 'AR' returns with respect to an official
> >> > response from Intel engineering on this issue.
> >> > 
> >> > One of the very bright young engineers collaborating on Quixote, who
> >> > has been following this conversation, took it upon himself to do some
> >> > very methodical engineering analysis on this issue.  I'm the messenger
> >> > but this is very much his work product.
> >> > 
> >> > Executive summary is as follows:
> >> > 
> >> > - No RDRAND depletion failures were observable with either the Intel
> >> >   or AMD hardware that was load tested.
> >> > 
> >> > - RDSEED depletion is an Intel specific issue, AMD's RDSEED
> >> >   implementation could not be provoked into failure.
> >
> >> My colleague ran a multithread parallel stress test program on his
> >> 16core/2HT AMD Ryzen (Zen4 uarch) and saw a 80% failure rate in
> >> RDSEED.
> >
> >Interesting datapoint, thanks for forwarding it along, so the issue
> >shows up on at least some AMD platforms as well.
> >
> >On the 18 core/socket Intel Skylake platform, the parallelized
> >depletion test forces RDSEED success rates down to around 2%.  It
> >would appear that your tests suggest that the AMD platform fairs
> >better than the Intel platform.
> >
> >So this is turning into even more of a morass, given that RDSEED
> >depletion on AMD may be a function of the micro-architecture the
> >platform is based on.  The other variable is that our AMD test
> >platform had a substantially higher core count per socket, one would
> >assume that would result in higher depletion rates, if the operative
> >theory of socket common RNG infrastructure is valid.
> >
> >Unless AMD engineering understands the problem and has taken some type
> >of action on higher core count systems to address the issue.
> >
> >Of course, the other variable may be how the parallelized stress test
> >is conducted.  If you would like to share your implementation source
> >we could give it a twirl on the systems we have access to.
> >
> >The continuing operative question is whether or not any of this ever
> >leads to an RDRAND failure.
> >
> >We've conducted some additional tests on the Intel platform where
> >RDSEED depletion was driven low as possible, ~1-2% success rates,
> >while RDRAND depletion tests were being run simultaneously.  No RDRAND
> >failures have been noted.
> >
> >So the operative question remains, why worry about this if RDRAND is
> >used as the randomness primitive.
> >
> >We haven't seen anything out of Intel yet on this, maybe AMD has a
> >quantifying definition for 'astronomical' when it comes to RDRAND
> >failures.
> >
> >The silence appears to be deafening out of the respective engineering
> >camps... :-)
> >
> >> > - AMD's RDRAND/RDSEED implementation is significantly slower than
> >> >   Intel's.
> >
> >> Yes, we also noticed the AMD impl is horribly slow compared to
> >> Intel, had to cut test iterations x100.
> >
> >The operative question is the impact of 'slow', in the absence of
> >artifical stress tests.
> >
> >It would seem that a major question is what are or were the
> >engineering thought processes on the throughput of the hardware
> >randomness instructions.
> >
> >Intel documents the following randomness throughput rates:
> >
> >RDSEED: 3 Gbit/second
> >RDRAND: 6.4 Gbit/second
> >
> >If there is the possibility of over-harvesting randomness, why not
> >design the implementations to be clamped at some per core value such
> >as a megabit/second.  In the case of the documented RDSEED generation
> >rates, that would allow the servicing of 3222 cores, if my math at
> >0530 in the morning is correct.
> >
> >Would a core need more than 128 kilobytes of randomness, ie. one
> >second of output, to effectively seed a random number generator?
> >
> >A cynical conclusion would suggest engineering acquiesing to marketing
> >demands... :-)
> >
> >> With regards,
> >> Daniel
> >
> >Have a good day.
> >
> >As always,
> >Dr. Greg
> >
> >The Quixote Project - Flailing at the Travails of Cybersecurity
> >              https://github.com/Quixote-Project

> You do realize, right, that the "deafening silence" is due to the
> need for research and discussions on our part, and presumably AMD's.

That would certainly be anticipated if not embraced, while those
discussions ensue, let me explain where we are coming from on this
issue.

I have a long time friend and valued personal consigliere who is a
tremendous attorney and legal scholar.  She has long advised me that
two basic concepts are instilled in law school; how to make sense out
of legal writing and don't ask any questions you don't know the answer
to.

CoCo is an engineering endeavor to defy the long held, and difficult
to deny, premise in information technology that if you don't have
physical security of a platform you don't have security.

We value ourselves as a good engineering team with considerable
interest and experience in all of this.  Any suggestion that there may
be some type of, even subtle, concern over the behavior of fundamental
hardware security primitives causes us to start asking questions and
testing things.

In this case the testing, quickly and easily, caused even more
questions to emerge.

> In addition, quite frankly, your rather abusive language isn't
> exactly encouraging people to speak publicly based on immediately
> available and therefore inherently incomplete and/or dated
> information, meaning that we have had to take even what discussions we
> might have been able to have in public without IP concerns behind the
> scenes.

Abusive?

I will freely don the moniker of being a practioner and purveyor of
rapier cynicism and wit, however, everyone who knows me beyond e-mail
would tell you that abusive would be the last definition they would
use in describing my intent and character.

I had the opportunity to sit next to Jim Gordon at dinner, who at the
time ran the Platform Security Division for Intel, at the SGX
Development Outreach Meeting that was held in Tel Aviv.  That was
after a pretty direct, but productive, technical exchange with the SGX
hardware engineers from Haifa.

I told him that I didn't mean to put the engineers on the spot but we
had been asked to voice our concerns as SGX infrastructure developers.
He told me the purpose of the meeting was for Intel to get tough and
demanding questions on issues of concern to developers so Intel could
deliver better and more relevant technology.

Just for the record and to close the abusive issue.  A review of this
thread will show that I never threw out accusations that hardware was
busted, backdoored nor did I advocate that the solution was to find
a better hardware vendor.

You fix engineering problems with engineering facts, hence our
interest in seeing how the question that got asked, perhaps
inadvertently, gets answered, so appropriate engineering changes can
be made in security dependent products.

> Yes, we work for Intel. No, we don't know every detail about every
> Intel chip ever created off the top of my head, nor do we
> necessarily know the exact person that is *currently* in charge of
> the architecture of a particular unit, nor is it necessarily true
> that even *that* person knows all the exact properties of the
> behavior of their unit when integrated into a particular SoC, as
> units are modular by design.

Interesting.

From the outside looking in, as engineers, this raises the obvious
question if the 'bus factor' for Bull Mountain has been exceeded.

Let me toss out, hopefully as a positive contribution, a 'Gedanken
Thought Dilemma' that the Intel team can take into their internal
deliberations.

One of the other products of this thread was the suggestion that a
CoCo hypervisor/hardware contribution could exert sufficient timing or
scheduling control so as to defeat the SDM's 10 try RDRAND
recommendation and induce a denial-of-service condition.

If that is indeed a possibility, given the long history of timing
based observation attacks on confidentiality, what guidance can be
offered to consumers of the relevant technologies that CoCo is indeed
a valid concept.  Particularly given the fact that the hardware that
consumers are trusting is physically in the hands of highly skilled
personnel, who have both the skills and phsyical control of the
hardware needed, to mount such an attack?

This is obviously a somewhat larger question than if RDRAND depletion
can be practically induced, so no need to rush the deliberations on
our behalf.

We will stand by in a quiet and decidedly non-abusive and
non-threatening posture, waiting to see what reflections that Borislav
might have on all of this... :-)

Have a good weekend.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 2/2] x86/random: Issue a warning if RDRAND or RDSEED fails

[Kuppuswamy Sathyanarayanan]

On 1/30/24 12:30 AM, Kirill A. Shutemov wrote:
> RDRAND and RDSEED instructions rarely fail. Ten retries should be
> sufficient to account for occasional failures.
>
> If the instruction fails more than ten times, it is likely that the
> hardware is broken or someone is attempting to exceed the rate at which
> the random number generator hardware can provide random numbers.
>
> Issue a warning if ten retries were not enough.

Did you come across a case where it fails? Wondering why add this
warning now?

>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  arch/x86/include/asm/archrandom.h | 11 +++++++++++
>  1 file changed, 11 insertions(+)
>
> diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
> index 918c5880de9e..fc8d837fb3b9 100644
> --- a/arch/x86/include/asm/archrandom.h
> +++ b/arch/x86/include/asm/archrandom.h
> @@ -13,6 +13,12 @@
>  #include <asm/processor.h>
>  #include <asm/cpufeature.h>
>  
> +#ifdef KASLR_COMPRESSED_BOOT
> +#define rd_warn(msg) warn(msg)

Why not use warn_once in both cases?

> +#else
> +#define rd_warn(msg) WARN_ONCE(1, msg)
> +#endif
> +
>  #define RDRAND_RETRY_LOOPS	10
>  
>  /* Unconditional execution of RDRAND and RDSEED */
> @@ -28,6 +34,9 @@ static inline bool __must_check rdrand_long(unsigned long *v)
>  		if (ok)
>  			return true;
>  	} while (--retry);
> +
> +	rd_warn("RDRAND failed\n");
> +
>  	return false;
>  }
>  
> @@ -45,6 +54,8 @@ static inline bool __must_check rdseed_long(unsigned long *v)
>  			return true;
>  	} while (--retry);
>  
> +	rd_warn("RDSEED failed\n");
> +
>  	return false;
>  }
>  

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Kirill,

I've been following the other discussion closely thinking about the
matter, but I suppose I'll jump in here directly on this patch, if
this is the approach the discussion is congealing around.

A comment below:

On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
<kirill.shutemov@linux.intel.com> wrote:
>  static inline bool __must_check rdseed_long(unsigned long *v)
>  {
> +       unsigned int retry = RDRAND_RETRY_LOOPS;
>         bool ok;
> -       asm volatile("rdseed %[out]"
> -                    CC_SET(c)
> -                    : CC_OUT(c) (ok), [out] "=r" (*v));
> -       return ok;
> +
> +       do {
> +               asm volatile("rdseed %[out]"
> +                            CC_SET(c)
> +                            : CC_OUT(c) (ok), [out] "=r" (*v));
> +
> +               if (ok)
> +                       return true;
> +       } while (--retry);
> +
> +       return false;
>  }

So, my understanding of RDRAND vs RDSEED -- deliberately leaving out
any cryptographic discussion here -- is roughly that RDRAND will
expand the seed material for longer, while RDSEED will mostly always
try to sample more bits from the environment. AES is fast, while
sampling is slow, so RDRAND gives better performance and is less
likely to fail, whereas RDSEED always has to wait on the hardware to
collect some bits, so is more likely to fail.

For that reason, most of the usage of RDRAND and RDSEED inside of
random.c is something to the tune of `if (!rdseed(out)) rdrand(out);`,
first trying RDSEED but falling back to RDRAND if it's busy. That
still seems to me like a reasonable approach, which this patch would
partly undermine (in concert with the next patch, which I'll comment
on in a follow up email there).

So maybe this patch #1 (of 2) can be dropped?

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 01:29:10PM +0100, Jason A. Donenfeld wrote:
> Hi Kirill,
> 
> I've been following the other discussion closely thinking about the
> matter, but I suppose I'll jump in here directly on this patch, if
> this is the approach the discussion is congealing around.
> 
> A comment below:
> 
> On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
> <kirill.shutemov@linux.intel.com> wrote:
> >  static inline bool __must_check rdseed_long(unsigned long *v)
> >  {
> > +       unsigned int retry = RDRAND_RETRY_LOOPS;
> >         bool ok;
> > -       asm volatile("rdseed %[out]"
> > -                    CC_SET(c)
> > -                    : CC_OUT(c) (ok), [out] "=r" (*v));
> > -       return ok;
> > +
> > +       do {
> > +               asm volatile("rdseed %[out]"
> > +                            CC_SET(c)
> > +                            : CC_OUT(c) (ok), [out] "=r" (*v));
> > +
> > +               if (ok)
> > +                       return true;
> > +       } while (--retry);
> > +
> > +       return false;
> >  }
> 
> So, my understanding of RDRAND vs RDSEED -- deliberately leaving out
> any cryptographic discussion here -- is roughly that RDRAND will
> expand the seed material for longer, while RDSEED will mostly always
> try to sample more bits from the environment. AES is fast, while
> sampling is slow, so RDRAND gives better performance and is less
> likely to fail, whereas RDSEED always has to wait on the hardware to
> collect some bits, so is more likely to fail.
> 
> For that reason, most of the usage of RDRAND and RDSEED inside of
> random.c is something to the tune of `if (!rdseed(out)) rdrand(out);`,
> first trying RDSEED but falling back to RDRAND if it's busy. That
> still seems to me like a reasonable approach, which this patch would
> partly undermine (in concert with the next patch, which I'll comment
> on in a follow up email there).
> 
> So maybe this patch #1 (of 2) can be dropped?

Unless there's a difference between ring 0 and ring 3, this simple test
is telling:

  #include <stdio.h>
  #include <immintrin.h>
  
  int main(int argc, char *argv[])
  {
    unsigned long long rand;
    unsigned int i, success_rand = 0, success_seed = 0;
    enum { TOTAL = 1000000 };
  
    for (i = 0; i < TOTAL; ++i)
    	success_rand += !!_rdrand64_step(&rand);
    for (i = 0; i < TOTAL; ++i)
    	success_seed += !!_rdseed64_step(&rand);
    
    printf("RDRAND: %.2f%%, RDSEED: %.2f%%\n", success_rand * 100.0 / TOTAL, success_seed * 100.0 / TOTAL);
    return 0;
  }

Result on my i7-11850H:

  RDRAND: 100.00%, RDSEED: 29.26%

And this doesn't even test multicore stuff.

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

 
> Hi Kirill,
> 
> I've been following the other discussion closely thinking about the
> matter, but I suppose I'll jump in here directly on this patch, if
> this is the approach the discussion is congealing around.
> 
> A comment below:
> 
> On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
> <kirill.shutemov@linux.intel.com> wrote:
> >  static inline bool __must_check rdseed_long(unsigned long *v)
> >  {
> > +       unsigned int retry = RDRAND_RETRY_LOOPS;
> >         bool ok;
> > -       asm volatile("rdseed %[out]"
> > -                    CC_SET(c)
> > -                    : CC_OUT(c) (ok), [out] "=r" (*v));
> > -       return ok;
> > +
> > +       do {
> > +               asm volatile("rdseed %[out]"
> > +                            CC_SET(c)
> > +                            : CC_OUT(c) (ok), [out] "=r" (*v));
> > +
> > +               if (ok)
> > +                       return true;
> > +       } while (--retry);
> > +
> > +       return false;
> >  }
> 
> So, my understanding of RDRAND vs RDSEED -- deliberately leaving out
> any cryptographic discussion here -- is roughly that RDRAND will
> expand the seed material for longer, while RDSEED will mostly always
> try to sample more bits from the environment. AES is fast, while
> sampling is slow, so RDRAND gives better performance and is less
> likely to fail, whereas RDSEED always has to wait on the hardware to
> collect some bits, so is more likely to fail.

The internals of Intel DRBG behind RDRAND/RDSEED has been publicly
documented, so the structure is no secret. Please see [1] for overall
structure and other aspects. So, yes, your overall understanding is correct
(there are many more details though). 

[1] https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html


> 
> For that reason, most of the usage of RDRAND and RDSEED inside of
> random.c is something to the tune of `if (!rdseed(out)) rdrand(out);`,
> first trying RDSEED but falling back to RDRAND if it's busy. That
> still seems to me like a reasonable approach, which this patch would
> partly undermine (in concert with the next patch, which I'll comment
> on in a follow up email there).

I agree that for the purpose of extracting entropy for Linux RNG falling
back to RDRAND (current behavior) is perfectly ok, so I think you are doing
the right thing. However, in principle it is not always the case, there are
situations when a fallback to RDRAND should not be used, but it is also
true that the user of this interface should know/understand this situation. 

> 
> So maybe this patch #1 (of 2) can be dropped?

Before we start debating this patchset, what is your opinion on the original
problem we raised for CoCo VMs when both RDRAND/RDSEED are made to
fail deliberately? 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 2:10 PM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
> The internals of Intel DRBG behind RDRAND/RDSEED has been publicly
> documented, so the structure is no secret. Please see [1] for overall
> structure and other aspects. So, yes, your overall understanding is correct
> (there are many more details though).

Indeed, have read it.

> > So maybe this patch #1 (of 2) can be dropped?
>
> Before we start debating this patchset, what is your opinion on the original
> problem we raised for CoCo VMs when both RDRAND/RDSEED are made to
> fail deliberately?

My general feeling is that this seems like a hardware problem.

If you have a VM, the hypervisor should provide a seed. But with CoCo,
you can't trust the host to do that. But can't the host do anything to
the VM that it wants, like fiddle with its memory? No, there are
special new hardware features to encrypt and protect ram to prevent
this. So if you've found yourself in a situation where you absolutely
cannot trust the host, AND the hardware already has working guest
protections from the host, then it would seem you also need a hardware
solution to handle seeding. And you're claiming that RDRAND/RDSEED is
the *only* hardware solution available for it.

Is that an accurate summary? If it is, then the actual problem is that
the hardware provided to solve this problem doesn't actually solve it
that well, so we're caught deciding between guest-guest DoS (some
other guest on the system uses all RDRAND resources) and cryptographic
failure because of a malicious host creating a deterministic
environment.

But I have two questions:

1) Is this CoCo VM stuff even real? Is protecting guests from hosts
actually possible in the end? Is anybody doing this? I assume they
are, so maybe ignore this question, but I would like to register my
gut feeling that on the Intel platform this seems like an endless
whack-a-mole problem like SGX.

2) Can a malicious host *actually* create a fully deterministic
environment? One that'll produce the same timing for the jitter
entropy creation, and all the other timers and interrupts and things?
I imagine the attestation part of CoCo means these VMs need to run on
real Intel silicon and so it can't be single stepped in TCG or
something, right? So is this problem actually a real one? And to what
degree? Any good experimental research on this?

Either way, if you're convinced RDRAND is the *only* way here, adding
a `WARN_ON(is_in_early_boot)` to the RDRAND (but not RDSEED) failure
path seems a fairly lightweight bandaid. I just wonder if the hardware
people could come up with something more reliable that we wouldn't
have to agonize over in the kernel.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Daniel P. Berrangé]

On Tue, Jan 30, 2024 at 03:06:14PM +0100, Jason A. Donenfeld wrote:
> Is that an accurate summary? If it is, then the actual problem is that
> the hardware provided to solve this problem doesn't actually solve it
> that well, so we're caught deciding between guest-guest DoS (some
> other guest on the system uses all RDRAND resources) and cryptographic
> failure because of a malicious host creating a deterministic
> environment.

In a CoCo VM environment, a guest DoS is not a unique threat
scenario, as it is unrelated to confidentiality. Ensuring
fair subdivision of resources between competeing guests is
just a general VM threat. There are many easy ways a host
admin can stop a guest making computational progress. Simply
not scheduling the guest vCPU threads is one. CoCo doesn't
try to solve this problem.

Preserving confidentiality is the primary aim of CoCo.

IOW, if the guest boot is stalled because the kernel is spinning
waiting on RDRAND to return data, that's fine. If the kernel
panics after "n" RDRAND failures in a row that's fine too. They
are both just yet another DoS scenario. 

If the kernel ignores the RDRAND failure and lets it boot with
degraded RNG state there were susceptible to attacks, that would
not be OK for CoCo. 

> But I have two questions:
> 
> 1) Is this CoCo VM stuff even real? Is protecting guests from hosts
> actually possible in the end? Is anybody doing this? I assume they
> are, so maybe ignore this question, but I would like to register my
> gut feeling that on the Intel platform this seems like an endless
> whack-a-mole problem like SGX.

It is real, but it is also not perfect. I expect it /will/ be an
endless whack-a-mole problem though.

None the less, it is a significant layer of defence, as compared
to traditional VMs where the guest RAM is nothing more than a
'cat' command away from host admin exposure.

> 2) Can a malicious host *actually* create a fully deterministic
> environment? One that'll produce the same timing for the jitter
> entropy creation, and all the other timers and interrupts and things?
> I imagine the attestation part of CoCo means these VMs need to run on
> real Intel silicon and so it can't be single stepped in TCG or
> something, right? So is this problem actually a real one? And to what
> degree? Any good experimental research on this?
> 
> Either way, if you're convinced RDRAND is the *only* way here, adding
> a `WARN_ON(is_in_early_boot)` to the RDRAND (but not RDSEED) failure
> path seems a fairly lightweight bandaid. I just wonder if the hardware
> people could come up with something more reliable that we wouldn't
> have to agonize over in the kernel.

If RDRAND failure is more of a theoretical problem than a practical
real world problem, I'd be inclined to just let the kernel loop on
RDRAND failure until it suceeds, with a WARN after 'n' iterations to
aid diagnosis of the stall in the unlikely even it did hit.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Tue, Jan 30, 2024 at 02:43:19PM +0000, Daniel P. Berrangé wrote:
> On Tue, Jan 30, 2024 at 03:06:14PM +0100, Jason A. Donenfeld wrote:
> > Is that an accurate summary? If it is, then the actual problem is that
> > the hardware provided to solve this problem doesn't actually solve it
> > that well, so we're caught deciding between guest-guest DoS (some
> > other guest on the system uses all RDRAND resources) and cryptographic
> > failure because of a malicious host creating a deterministic
> > environment.
> 
> In a CoCo VM environment, a guest DoS is not a unique threat
> scenario, as it is unrelated to confidentiality. Ensuring
> fair subdivision of resources between competeing guests is
> just a general VM threat. There are many easy ways a host
> admin can stop a guest making computational progress. Simply
> not scheduling the guest vCPU threads is one. CoCo doesn't
> try to solve this problem.
> 
> Preserving confidentiality is the primary aim of CoCo.
> 
> IOW, if the guest boot is stalled because the kernel is spinning
> waiting on RDRAND to return data, that's fine. If the kernel
> panics after "n" RDRAND failures in a row that's fine too. They
> are both just yet another DoS scenario. 
> 
> If the kernel ignores the RDRAND failure and lets it boot with
> degraded RNG state there were susceptible to attacks, that would
> not be OK for CoCo. 

Yea, that's why I said "we're caught deciding..." One case is a DoS that
would affect all VMs, so while one guest preventing new guests from
booting seems like not a CoCo problem, yes, it is still a problem.

At least in theory. And in practice this is easy with RDSEED too. In
practice, could you actually indefinably starve RDRAND between guests?
Is this pretty easy to do with a little tinkering, or is this a
practically impossible DoS vector? I don't actually know.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Elena,

On Tue, Jan 30, 2024 at 3:06 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> 2) Can a malicious host *actually* create a fully deterministic
> environment? One that'll produce the same timing for the jitter
> entropy creation, and all the other timers and interrupts and things?
> I imagine the attestation part of CoCo means these VMs need to run on
> real Intel silicon and so it can't be single stepped in TCG or
> something, right? So is this problem actually a real one? And to what
> degree? Any good experimental research on this?

I'd like to re-up this question. It seems like assessing the reality
of the concern would be worthwhile.

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> Elena,
> 
> On Tue, Jan 30, 2024 at 3:06 PM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > 2) Can a malicious host *actually* create a fully deterministic
> > environment? One that'll produce the same timing for the jitter
> > entropy creation, and all the other timers and interrupts and things?
> 
> I'd like to re-up this question. It seems like assessing the reality
> of the concern would be worthwhile.

Yes, sorry, I am just behind answering this thread and it is getting late here. 
This is exactly what I would like to have an open discussion about
with inputs from everyone. 
We have to remember that it is not only about host 'producing' 
a fully deterministic environment, but also about host being able to 
*observe* the entropy input. So the more precise question to ask is
how much can a host observe? My personal understanding is that host can
observe all guest interrupts and their timings, including APIC timer interrupts
(and IPIs), so what is actually left for the guest as unobservable entropy
input? 
And let's also please remember that this is by no means Intel-specific,
we have other confidential computing vendors, so we need a common
agreement on what is the superset of attacker powers that we can
assume.

> > I imagine the attestation part of CoCo means these VMs need to run on
> > real Intel silicon and so it can't be single stepped in TCG or
> > something, right? 

Yes, there is an attestation of a confidential VM and some protections in place
that helps against single-stepping attacks. But I am not sure how this is relevant
for this, could you please clarify? 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Elena,

On Tue, Jan 30, 2024 at 8:06 PM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
> Yes, sorry, I am just behind answering this thread and it is getting late here.
> This is exactly what I would like to have an open discussion about
> with inputs from everyone.
> We have to remember that it is not only about host 'producing'
> a fully deterministic environment, but also about host being able to
> *observe* the entropy input. So the more precise question to ask is
> how much can a host observe?

Right, observation is just as relevant.

> My personal understanding is that host can
> observe all guest interrupts and their timings, including APIC timer interrupts
> (and IPIs), so what is actually left for the guest as unobservable entropy
> input?

Check out try_to_generate_entropy() and random_get_entropy(), for
example. How observable is RDTSC? Other HPTs?

> > > I imagine the attestation part of CoCo means these VMs need to run on
> > > real Intel silicon and so it can't be single stepped in TCG or
> > > something, right?
>
> Yes, there is an attestation of a confidential VM and some protections in place
> that helps against single-stepping attacks. But I am not sure how this is relevant
> for this, could you please clarify?

I was just thinking that if this didn't require genuine Intel hardware
with prebaked keys in it that you could emulate a CPU and all its
peripherals and ram with defined latencies and such, and run the VM in
a very straightforwardly deterministic environment, because nothing
would be real. But if this does have to hit metal somewhere, then
there's some possibility you at least interact with some hard-to-model
physical hardware.

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> Hi Elena,
> 
> On Tue, Jan 30, 2024 at 8:06 PM Reshetova, Elena
> <elena.reshetova@intel.com> wrote:
> > Yes, sorry, I am just behind answering this thread and it is getting late here.
> > This is exactly what I would like to have an open discussion about
> > with inputs from everyone.
> > We have to remember that it is not only about host 'producing'
> > a fully deterministic environment, but also about host being able to
> > *observe* the entropy input. So the more precise question to ask is
> > how much can a host observe?
> 
> Right, observation is just as relevant.
> 
> > My personal understanding is that host can
> > observe all guest interrupts and their timings, including APIC timer interrupts
> > (and IPIs), so what is actually left for the guest as unobservable entropy
> > input?
> 
> Check out try_to_generate_entropy() and random_get_entropy(), for
> example. How observable is RDTSC? Other HPTs?

Ok, here imo it gets arch-specific and so please treat my answers only 
with Intel TDX arch in mind. I do know that for example AMD behavior
for TSC is different, albeit I am not sure of details. Other archs might also
have different behavior.

For Intel TDX, when a guest executes RDTSC, it gets a virtual TSC value that
is calculated deterministically based on a bunch of inputs that are either
platform HW specific or VMM/host configured. The physical TSC value is taken 
into account also in calculations.  The guest itself is not able to
use usual controls (such as IA32_TSC_ADJUST and such). For details (albeit not
exact calculations) please see [1]. If you are interested in exact calculations, 
the public source code of TDX module is a better reference [2], check
calculate_virt_tsc() or just grep with "tsc" it would show you both comments
explaining what is happening and calculations.
So given this, I would personally consider the virtual guest TSC value
observable by host/VMM. 

 [1] TDX module spec, section 11.13 Time Stamp Counter (TSC)
https://cdrdv2.intel.com/v1/dl/getContent/733575 
[2] TDX module source code:
https://www.intel.com/content/www/us/en/download/738875/782152/intel-trust-domain-extension-intel-tdx-module.html

For the high resolution timers, host controls guest apic timers and interrupts fully. 
So, it has the power to see and even affect when a certain interrupt happens
or doesnt happen in the guest. It can delay guest timers at its will on pretty
extensive time periods. This seems powerful enough for me. 
Things like HPET are also fully under host control. 

> > > > I imagine the attestation part of CoCo means these VMs need to run on
> > > > real Intel silicon and so it can't be single stepped in TCG or
> > > > something, right?
> >
> > Yes, there is an attestation of a confidential VM and some protections in place
> > that helps against single-stepping attacks. But I am not sure how this is relevant
> > for this, could you please clarify?
> 
> I was just thinking that if this didn't require genuine Intel hardware
> with prebaked keys in it that you could emulate a CPU and all its
> peripherals and ram with defined latencies and such, and run the VM in
> a very straightforwardly deterministic environment, because nothing
> would be real. But if this does have to hit metal somewhere, then
> there's some possibility you at least interact with some hard-to-model
> physical hardware.

Yes, in practice there will be physical hw underneath, but the problem imo is
that the host is in between and still very powerful when it comes to interrupts and
timers at the moment. So, I want to make sure people understand the potential
implications overall, and in this case the potential implications on such a
critical security component as Linux RNG. 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Wed, Jan 31, 2024 at 8:56 AM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
> So given this, I would personally consider the virtual guest TSC value
> observable by host/VMM.
> [2] TDX module source code:
> https://www.intel.com/content/www/us/en/download/738875/782152/intel-trust-domain-extension-intel-tdx-module.html

Thanks for the explanation and link. Indeed if this is all mediated by
the host, we're in bad shape.

> For the high resolution timers, host controls guest apic timers and interrupts fully.
> So, it has the power to see and even affect when a certain interrupt happens
> or doesnt happen in the guest. It can delay guest timers at its will on pretty
> extensive time periods. This seems powerful enough for me.
> Things like HPET are also fully under host control.

And I suppose RDPMC is similar?

And it's not like the guest can just take an excessive amount of TSC
samples and randomly select which ones it uses because chickens and
eggs...

The situation you paint is that all of our entropy inputs -- timers,
rdrand, etc -- are either host controllable, host observable, or host
(and guest sibling) DoS'able, so if you don't trust the host, there
are no good inputs. That's not a great position to be in, and I wonder
if something can be done on the hardware side to remedy it, as this
seems like a major shortcoming in TDX. So far, all of the proposed
mitigations introduce some other DoS.

> Yes, in practice there will be physical hw underneath, but the problem imo is
> that the host is in between and still very powerful when it comes to interrupts and
> timers at the moment.

Sure sounds like it.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

What about simply treating boot-time initialization of the /dev/random
state as special.  That is, on x86, if the hardware promises that
RDSEED or RDRAND is available, we use them to initialization our RNG
state at boot.  On bare metal, there can't be anyone else trying to
exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
aren't working available --- panic, since the hardware is clearly
busted.

On a guest OS, if confidential compute is enabled, and if RDSEED and
RDRAND don't work after N retries, and we know CC is enabled, panic,
since the kernel can't provide the promised security gaurantees, and
the CC developers and users are cordially invited to sharpen their
pitchforks and to send their tender regards to the Intel RNG
engineers.

For non-confidential compute guests, the question is what is the
appropriate reaction if another VM, possibly belonging to a different
user/customer, is carrying out a RDRAND DOS attack.  I'd argue that in
these cases, if the guest VM is using virtio-random, then the host's
/dev/random should be able to cover for cases of Intel RNG exhaustion,
and allowing other customer to be able to prevent other user's VM's
from being able to boot is the the greater evil, so we shouldn't treat
boot-time RDRAND/RDSEED failures as panic-worthy.

						- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Wed, Jan 31, 2024 at 09:07:56AM -0500, Theodore Ts'o wrote:
> What about simply treating boot-time initialization of the /dev/random
> state as special.  That is, on x86, if the hardware promises that
> RDSEED or RDRAND is available, we use them to initialization our RNG
> state at boot.  On bare metal, there can't be anyone else trying to
> exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
> aren't working available --- panic, since the hardware is clearly
> busted.

This is the first thing I suggested here: https://lore.kernel.org/all/CAHmME9qsfOdOEHHw_MOBmt6YAtncbbqP9LPK2dRjuOp1CrHzRA@mail.gmail.com/

But Elena found this dissatisfying because we still can't guarantee new
material later.

> On a guest OS, if confidential compute is enabled, and if RDSEED and
> RDRAND don't work after N retries, and we know CC is enabled, panic,
> since the kernel can't provide the promised security gaurantees, and
> the CC developers and users are cordially invited to sharpen their
> pitchforks and to send their tender regards to the Intel RNG
> engineers.

Yea, maybe bubbling the RDRAND DoS up to another DoS in the CoCo case is
a good tradeoff that will produce the right pitchforkers without
breaking anything real.

> For non-confidential compute guests, the question is what is the
> appropriate reaction if another VM, possibly belonging to a different
> user/customer, is carrying out a RDRAND DOS attack.  I'd argue that in
> these cases, if the guest VM is using virtio-random, then the host's
> /dev/random should be able to cover for cases of Intel RNG exhaustion,
> and allowing other customer to be able to prevent other user's VM's
> from being able to boot is the the greater evil, so we shouldn't treat
> boot-time RDRAND/RDSEED failures as panic-worthy.

The non-CoCo case is fine, because guests can trust hosts, so things are
as they have been forever.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Wed, Jan 31, 2024 at 03:45:06PM +0100, Jason A. Donenfeld wrote:
> On Wed, Jan 31, 2024 at 09:07:56AM -0500, Theodore Ts'o wrote:
> > What about simply treating boot-time initialization of the /dev/random
> > state as special.  That is, on x86, if the hardware promises that
> > RDSEED or RDRAND is available, we use them to initialization our RNG
> > state at boot.  On bare metal, there can't be anyone else trying to
> > exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
> > aren't working available --- panic, since the hardware is clearly
> > busted.
> 
> This is the first thing I suggested here: https://lore.kernel.org/all/CAHmME9qsfOdOEHHw_MOBmt6YAtncbbqP9LPK2dRjuOp1CrHzRA@mail.gmail.com/
> 
> But Elena found this dissatisfying because we still can't guarantee new
> material later.
> 
> > On a guest OS, if confidential compute is enabled, and if RDSEED and
> > RDRAND don't work after N retries, and we know CC is enabled, panic,
> > since the kernel can't provide the promised security gaurantees, and
> > the CC developers and users are cordially invited to sharpen their
> > pitchforks and to send their tender regards to the Intel RNG
> > engineers.
> 
> Yea, maybe bubbling the RDRAND DoS up to another DoS in the CoCo case is
> a good tradeoff that will produce the right pitchforkers without
> breaking anything real.

One problem, though, is userspace can DoS the kernel's use of RDRAND.
So probably infinitely retrying in CoCo environments is better than
panicing/warning, since ostensibly a kthread will eventually succeed.

Maybe, though, the Intel platform just simply isn't ready for CoCo, and
marketing got a little bit ahead of the tech?

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Wed, Jan 31, 2024 at 03:45:06PM +0100, Jason A. Donenfeld wrote:
> On Wed, Jan 31, 2024 at 09:07:56AM -0500, Theodore Ts'o wrote:
> > What about simply treating boot-time initialization of the /dev/random
> > state as special.  That is, on x86, if the hardware promises that
> > RDSEED or RDRAND is available, we use them to initialization our RNG
> > state at boot.  On bare metal, there can't be anyone else trying to
> > exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
> > aren't working available --- panic, since the hardware is clearly
> > busted.
> 
> This is the first thing I suggested here: https://lore.kernel.org/all/CAHmME9qsfOdOEHHw_MOBmt6YAtncbbqP9LPK2dRjuOp1CrHzRA@mail.gmail.com/
> 
> But Elena found this dissatisfying because we still can't guarantee new
> material later.

Right, but this is good enough that modulo in-kernel RNG state
compromise, or the ability to attack the underlying cryptographic
primitives (in which case we have much bigger vulnerabilities than
this largely theoretical one), even if we don't have new material
later, the in-kernel RNG for the CC VM should be sufficiently
trustworthy for government work.

> Yea, maybe bubbling the RDRAND DoS up to another DoS in the CoCo case is
> a good tradeoff that will produce the right pitchforkers without
> breaking anything real.

<Evil Grin>

					- Ted

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> On Wed, Jan 31, 2024 at 03:45:06PM +0100, Jason A. Donenfeld wrote:
> > On Wed, Jan 31, 2024 at 09:07:56AM -0500, Theodore Ts'o wrote:
> > > What about simply treating boot-time initialization of the /dev/random
> > > state as special.  That is, on x86, if the hardware promises that
> > > RDSEED or RDRAND is available, we use them to initialization our RNG
> > > state at boot.  On bare metal, there can't be anyone else trying to
> > > exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
> > > aren't working available --- panic, since the hardware is clearly
> > > busted.
> >
> > This is the first thing I suggested here:
> https://lore.kernel.org/all/CAHmME9qsfOdOEHHw_MOBmt6YAtncbbqP9LPK2dRjuO
> p1CrHzRA@mail.gmail.com/
> >
> > But Elena found this dissatisfying because we still can't guarantee new
> > material later.
> 
> Right, but this is good enough that modulo in-kernel RNG state
> compromise, or the ability to attack the underlying cryptographic
> primitives (in which case we have much bigger vulnerabilities than
> this largely theoretical one), even if we don't have new material
> later, the in-kernel RNG for the CC VM should be sufficiently
> trustworthy for government work.

I agree, this is probably the best we can do at the moment. 
I did want to point out the runtime need of fresh entropy also, but
as we discussed in this thread we might not be able to get it 
without introducing a DoS path for the userspace. 
In this case, it is the best to only loose the forward prediction property
vs. the whole Linux RNG. 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Wed, Jan 31, 2024 at 6:37 PM Reshetova, Elena
<elena.reshetova@intel.com> wrote:
>
>
>
> > On Wed, Jan 31, 2024 at 03:45:06PM +0100, Jason A. Donenfeld wrote:
> > > On Wed, Jan 31, 2024 at 09:07:56AM -0500, Theodore Ts'o wrote:
> > > > What about simply treating boot-time initialization of the /dev/random
> > > > state as special.  That is, on x86, if the hardware promises that
> > > > RDSEED or RDRAND is available, we use them to initialization our RNG
> > > > state at boot.  On bare metal, there can't be anyone else trying to
> > > > exhaust the on-chip RNG's entropy supply, so if RDSEED or RDRAND
> > > > aren't working available --- panic, since the hardware is clearly
> > > > busted.
> > >
> > > This is the first thing I suggested here:
> > https://lore.kernel.org/all/CAHmME9qsfOdOEHHw_MOBmt6YAtncbbqP9LPK2dRjuO
> > p1CrHzRA@mail.gmail.com/
> > >
> > > But Elena found this dissatisfying because we still can't guarantee new
> > > material later.
> >
> > Right, but this is good enough that modulo in-kernel RNG state
> > compromise, or the ability to attack the underlying cryptographic
> > primitives (in which case we have much bigger vulnerabilities than
> > this largely theoretical one), even if we don't have new material
> > later, the in-kernel RNG for the CC VM should be sufficiently
> > trustworthy for government work.
>
> I agree, this is probably the best we can do at the moment.
> I did want to point out the runtime need of fresh entropy also, but
> as we discussed in this thread we might not be able to get it
> without introducing a DoS path for the userspace.
> In this case, it is the best to only loose the forward prediction property
> vs. the whole Linux RNG.

So if this is what we're congealing around, I guess we can:

0) Leave RDSEED alone and focus on RDRAND.
1) Add `WARN_ON_ONCE(in_early_boot);` to the failure path of RDRAND
(and simply hope this doesn't get exploited for guest-guest boot DoS).
2) Loop forever in RDRAND on CoCo VMs, post-boot, with the comments
and variable naming making it clear that this is a hardware bug
workaround, not a "feature" added for "extra security".
3) Complain loudly to Intel and get them to fix the hardware.

Though, a large part of me would really like to skip that step (2),
first because it's a pretty gross bandaid that adds lots of
complexity, and second because it'll make (3) less poignant.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Wed, Jan 31, 2024 at 07:01:01PM +0100, Jason A. Donenfeld wrote:
> So if this is what we're congealing around, I guess we can:
> 
> 0) Leave RDSEED alone and focus on RDRAND.
> 1) Add `WARN_ON_ONCE(in_early_boot);` to the failure path of RDRAND
> (and simply hope this doesn't get exploited for guest-guest boot DoS).
> 2) Loop forever in RDRAND on CoCo VMs, post-boot, with the comments
> and variable naming making it clear that this is a hardware bug
> workaround, not a "feature" added for "extra security".
> 3) Complain loudly to Intel and get them to fix the hardware.
> 
> Though, a large part of me would really like to skip that step (2),
> first because it's a pretty gross bandaid that adds lots of
> complexity, and second because it'll make (3) less poignant

If we need to loop more than, say, 10 seconds in a CoCo VM, I'd just
panic with a repeated RDRAND failure message.  This makes the point of
(3) that much pointed, and it's better than having a CoCo VM
mysteriously hang in the face of a DOS attack.

I'll note that it should be relatively easy for Intel to make sure
that if there is an undue draw on RDRAND, to at that point enforce
"fair share" mode where each of the N cores get at most 1/N of the
available entropy.  So if you have single core CoCo VM on a 256 core
machine trying to boot, and the evil attacker has purchased 255 cores
worth of VM's, all of which are busy-looping on RDRAND, while the CoCo
VM is booting, if it is looping on RDRAND, it should be getting
1/256th of the availabe RDRAND output, and since it is only trying to
grab enough randomness to seed the /dev/random CRNG, if it can't get
enough randomness in 10 seconds --- well, Intel's customers should be
finding another vendor's CPU that can do a better job.

			     	      	 - Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Ted, Elena, Dave,

On Thu, Feb 1, 2024 at 5:57 AM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Wed, Jan 31, 2024 at 07:01:01PM +0100, Jason A. Donenfeld wrote:
> > So if this is what we're congealing around, I guess we can:
> >
> > 0) Leave RDSEED alone and focus on RDRAND.
> > 1) Add `WARN_ON_ONCE(in_early_boot);` to the failure path of RDRAND
> > (and simply hope this doesn't get exploited for guest-guest boot DoS).
> > 2) Loop forever in RDRAND on CoCo VMs, post-boot, with the comments
> > and variable naming making it clear that this is a hardware bug
> > workaround, not a "feature" added for "extra security".
> > 3) Complain loudly to Intel and get them to fix the hardware.
> >
> > Though, a large part of me would really like to skip that step (2),
> > first because it's a pretty gross bandaid that adds lots of
> > complexity, and second because it'll make (3) less poignant
>
> If we need to loop more than, say, 10 seconds in a CoCo VM, I'd just
> panic with a repeated RDRAND failure message.  This makes the point of
> (3) that much pointed, and it's better than having a CoCo VM
> mysteriously hang in the face of a DOS attack.

Yea, true. Problem is that in theory, userspace can DoS the kernel's
use of RDRAND. Of course in practice, a userspace process preempting a
kthread for >10 seconds is probably a larger problem.

Anyway, I want to lay out the various potential solutions discussed.
As they all have some drawback, it's worth enumerating them.

==

Solution A) WARN_ON_ONCE(is_early_boot)/BUG_ON(is_early_boot) in the
RDRAND failure path (> 10 retries).

The biggest advantage here is that this is super simple and isn't
CoCo-specific. The premise is that if RDRAND fails 10 times in a row
before userspace has started, it's most definitely a hardware problem.
Systems-wise, the drawback is that, in a VM, it alternatively might be
a guest-guest DoS attack on RDRAND, or in the CoCo case, a host-guest
DoS attack (which is presumably easier because the host controls
scheduling). In the CoCo case, not booting is better than losing
confidentiality. In the non-CoCo case, that seems like theoretically a
DoS we might not want. RNG-wise, the drawback is that this doesn't
help deal with secure reseeding later in time, which is a RNG property
that we otherwise enjoy.

Solution B) BUG_ON(is_early_boot && is_coco_system) in the RDRAND
failure path (> 10 retries).

This is slightly less simple than A, because we have to plumb
CoCo-detection through to the RDRAND helper. [Side note: I feel
ridiculous typing 'CoCo'.] Systems-wise, I don't see drawbacks.
RNG-wise, the drawback is that this doesn't help deal with secure
reseeding later in time, which is a RNG property that we otherwise
enjoy.

Solution C) WARN_ONCE()/BUG() in the RDRAND failure path (> 10 retries).

The advantage here is also simplicity, and the fact that it "ensures"
we'll be able to securely reseed later on. Systems-wise, the drawback
is that userspace can in theory DoS the kernel's RDRAND and cause a
crash.

Solution D) BUG_ON(is_coco_system) in the RDRAND failure path (> 10 retries).

This is slightly less simple than A, because we have to plumb
CoCo-detection through to the RDRAND helper, but it "ensures" we'll be
able to securely reseed later on. Systems-wise, the drawback is that
userspace can in theory DoS the kernel's RDRAND and cause a crash.

Solution E) BUG() in a new time-based RDRAND failure path on CoCo
systems (> 10 seconds).

This adds a lot of complexity, and we'd need some alternative code
path for CoCo with an infinite loop that breaks on a jiffies
comparison. But it at least makes it harder for userspace to DoS the
kernel's use of RDRAND, because it seems hard for a user thread to
preempt a kthread for that long, though maybe somebody has some nasty
scheduler tricks here that would break that hope.

Solution F) Loop forever in RDRAND on CoCo systems.

This makes debugging harder because of lockups (though I suppose we
could WARN after some amount of time), but at least it's somewhat
"sound".

==

I am currently leaning toward (B) as being the lightest touch that has
the least potential to break anything. (F) is also tempting because it
doesn't have the RNG-drawback. The others seem complex or incomplete
or otherwise annoying somehow.

There is also "Solution G" -- do nothing and raise a fuss and let
security researchers go to town and hope Intel gets their act
together. Given that the CoCo thing seems kind of
imaginary/aspirational anyway at this point, I'm very attracted by
this. I don't mean to say that I intend to mount a large argument that
we *should* do nothing, but it's just sort of sitting there in the
back of my mind as an appealing possibility.

Also, I wanted to enumerate currently open questions:

==

Question i) Just how deterministic can these CoCo VMs be? Elena
pointed to some TDX code regarding RDTSC that seemed fairly damning,
but I also wonder what gotchas a motivated researcher might run into
and how those could help us (or not).

Question ii) Just how DoS-able is RDRAND? From host to guest, where
the host controls scheduling, that seems easier, but how much so, and
what's the granularity of these operations, and could retries still
help, or not at all? What about from guest to guest, where the
scheduling is out of control; in that case is there a value of N for
which N retries makes it actually impossible to DoS? What about from
userspace to kernelspace; good value of N?

Question iii) How likely is Intel to actually fix this in a
satisfactory way (see "specifying this is an interesting question" in
[1])? And if they would, what would the timeline even be?

==

Anyway, that's about where I'm at. I figure I'll wait to see if the
internal inquiry within Intel yields anything interesting, and then
maybe we can move forward with solutions (B) or (F) or (G) or a
different Roald Dahl novel instead.

Jason

[1] https://lore.kernel.org/all/CAHmME9ps6W5snQrYeNVMFgfhMKFKciky=-UxxGFbAx_RrxSHoA@mail.gmail.com/

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dave Hansen]

On 2/1/24 10:09, Jason A. Donenfeld wrote:
> Question ii) Just how DoS-able is RDRAND? From host to guest, where
> the host controls scheduling, that seems easier, but how much so, and
> what's the granularity of these operations, and could retries still
> help, or not at all? What about from guest to guest, where the
> scheduling is out of control; in that case is there a value of N for
> which N retries makes it actually impossible to DoS? What about from
> userspace to kernelspace; good value of N?

So far, in practice, I haven't seen a single failure of RDRAND.  It's
been limited to RDSEED.  In a perfect world, I'd change the architecture
docs to say, "RDRAND only fails when the hardware breaks" and leave
RDSEED defined to be the one that fails easily.

Dealing with a fragile RDSEED seems like a much easier problem than
dealing with a fragile RDRAND since RDSEED is used _much_ more sparingly
in the kernel today.

But I'm not sure if the hardware implementations fit into this perfect
world I've conjured up.  We're going to wrangle up the folks at Intel
who can hopefully tell me if I'm totally deluded.

Has anyone seen RDRAND failures in practice?  Or just RDSEED?

> Question iii) How likely is Intel to actually fix this in a
> satisfactory way (see "specifying this is an interesting question" in
> [1])? And if they would, what would the timeline even be?

If the fix is pure documentation, it's on the order of months.  I'm
holding out hope that some kind of anti-DoS claims like you mentioned:

> Specifying this is an interesting question. What exactly might our
> requirements be for a "non-broken" RDRAND? It seems like we have two
> basic ones:
> 
> - One VMX (or host) context can't DoS another one.
> - Ring 3 can't DoS ring 0.

are still possible on existing hardware, at least for RDRAND.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[H. Peter Anvin]

On February 1, 2024 10:46:06 AM PST, Dave Hansen <dave.hansen@intel.com> wrote:
>On 2/1/24 10:09, Jason A. Donenfeld wrote:
>> Question ii) Just how DoS-able is RDRAND? From host to guest, where
>> the host controls scheduling, that seems easier, but how much so, and
>> what's the granularity of these operations, and could retries still
>> help, or not at all? What about from guest to guest, where the
>> scheduling is out of control; in that case is there a value of N for
>> which N retries makes it actually impossible to DoS? What about from
>> userspace to kernelspace; good value of N?
>
>So far, in practice, I haven't seen a single failure of RDRAND.  It's
>been limited to RDSEED.  In a perfect world, I'd change the architecture
>docs to say, "RDRAND only fails when the hardware breaks" and leave
>RDSEED defined to be the one that fails easily.
>
>Dealing with a fragile RDSEED seems like a much easier problem than
>dealing with a fragile RDRAND since RDSEED is used _much_ more sparingly
>in the kernel today.
>
>But I'm not sure if the hardware implementations fit into this perfect
>world I've conjured up.  We're going to wrangle up the folks at Intel
>who can hopefully tell me if I'm totally deluded.
>
>Has anyone seen RDRAND failures in practice?  Or just RDSEED?
>
>> Question iii) How likely is Intel to actually fix this in a
>> satisfactory way (see "specifying this is an interesting question" in
>> [1])? And if they would, what would the timeline even be?
>
>If the fix is pure documentation, it's on the order of months.  I'm
>holding out hope that some kind of anti-DoS claims like you mentioned:
>
>> Specifying this is an interesting question. What exactly might our
>> requirements be for a "non-broken" RDRAND? It seems like we have two
>> basic ones:
>> 
>> - One VMX (or host) context can't DoS another one.
>> - Ring 3 can't DoS ring 0.
>
>are still possible on existing hardware, at least for RDRAND.

The real question is: what do we actually need?

During startup, we could afford a *lot* of looping to collect enough entropy before giving up. After that, even if RDSEED fails 99% of the time, it will still produce far more entropy than a typical external randomness source. We don't want to loop that long, obviously (*), but instead try periodically and let the entropy accumulate.

(*) We *could* of course choose to aggressively loop in task context if there task would otherwise block on /dev/random.

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> Hi Ted, Elena, Dave,
> 
> On Thu, Feb 1, 2024 at 5:57 AM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > On Wed, Jan 31, 2024 at 07:01:01PM +0100, Jason A. Donenfeld wrote:
> > > So if this is what we're congealing around, I guess we can:
> > >
> > > 0) Leave RDSEED alone and focus on RDRAND.
> > > 1) Add `WARN_ON_ONCE(in_early_boot);` to the failure path of RDRAND
> > > (and simply hope this doesn't get exploited for guest-guest boot DoS).
> > > 2) Loop forever in RDRAND on CoCo VMs, post-boot, with the comments
> > > and variable naming making it clear that this is a hardware bug
> > > workaround, not a "feature" added for "extra security".
> > > 3) Complain loudly to Intel and get them to fix the hardware.
> > >
> > > Though, a large part of me would really like to skip that step (2),
> > > first because it's a pretty gross bandaid that adds lots of
> > > complexity, and second because it'll make (3) less poignant
> >
> > If we need to loop more than, say, 10 seconds in a CoCo VM, I'd just
> > panic with a repeated RDRAND failure message.  This makes the point of
> > (3) that much pointed, and it's better than having a CoCo VM
> > mysteriously hang in the face of a DOS attack.
> 
> Yea, true. Problem is that in theory, userspace can DoS the kernel's
> use of RDRAND. Of course in practice, a userspace process preempting a
> kthread for >10 seconds is probably a larger problem.
> 
> Anyway, I want to lay out the various potential solutions discussed.
> As they all have some drawback, it's worth enumerating them.
> 
> ==
> 
> Solution A) WARN_ON_ONCE(is_early_boot)/BUG_ON(is_early_boot) in the
> RDRAND failure path (> 10 retries).
> 
> The biggest advantage here is that this is super simple and isn't
> CoCo-specific. The premise is that if RDRAND fails 10 times in a row
> before userspace has started, it's most definitely a hardware problem.
> Systems-wise, the drawback is that, in a VM, it alternatively might be
> a guest-guest DoS attack on RDRAND, or in the CoCo case, a host-guest
> DoS attack (which is presumably easier because the host controls
> scheduling). In the CoCo case, not booting is better than losing
> confidentiality. In the non-CoCo case, that seems like theoretically a
> DoS we might not want. RNG-wise, the drawback is that this doesn't
> help deal with secure reseeding later in time, which is a RNG property
> that we otherwise enjoy.
> 
> Solution B) BUG_ON(is_early_boot && is_coco_system) in the RDRAND
> failure path (> 10 retries).
> 
> This is slightly less simple than A, because we have to plumb
> CoCo-detection through to the RDRAND helper. [Side note: I feel
> ridiculous typing 'CoCo'.] Systems-wise, I don't see drawbacks.
> RNG-wise, the drawback is that this doesn't help deal with secure
> reseeding later in time, which is a RNG property that we otherwise
> enjoy.
> 
> Solution C) WARN_ONCE()/BUG() in the RDRAND failure path (> 10 retries).
> 
> The advantage here is also simplicity, and the fact that it "ensures"
> we'll be able to securely reseed later on. Systems-wise, the drawback
> is that userspace can in theory DoS the kernel's RDRAND and cause a
> crash.
> 
> Solution D) BUG_ON(is_coco_system) in the RDRAND failure path (> 10 retries).
> 
> This is slightly less simple than A, because we have to plumb
> CoCo-detection through to the RDRAND helper, but it "ensures" we'll be
> able to securely reseed later on. Systems-wise, the drawback is that
> userspace can in theory DoS the kernel's RDRAND and cause a crash.
> 
> Solution E) BUG() in a new time-based RDRAND failure path on CoCo
> systems (> 10 seconds).
> 
> This adds a lot of complexity, and we'd need some alternative code
> path for CoCo with an infinite loop that breaks on a jiffies
> comparison. But it at least makes it harder for userspace to DoS the
> kernel's use of RDRAND, because it seems hard for a user thread to
> preempt a kthread for that long, though maybe somebody has some nasty
> scheduler tricks here that would break that hope.
> 
> Solution F) Loop forever in RDRAND on CoCo systems.
> 
> This makes debugging harder because of lockups (though I suppose we
> could WARN after some amount of time), but at least it's somewhat
> "sound".
> 
> ==

This is a great summary of options, thank you Jason!
My proposal would be to wait on result of our internal investigation 
before proceeding to choose the approach. 

> 
> I am currently leaning toward (B) as being the lightest touch that has
> the least potential to break anything. (F) is also tempting because it
> doesn't have the RNG-drawback. The others seem complex or incomplete
> or otherwise annoying somehow.
> 
> There is also "Solution G" -- do nothing and raise a fuss and let
> security researchers go to town and hope Intel gets their act
> together. Given that the CoCo thing seems kind of
> imaginary/aspirational anyway at this point, I'm very attracted by
> this. I don't mean to say that I intend to mount a large argument that
> we *should* do nothing, but it's just sort of sitting there in the
> back of my mind as an appealing possibility.
> 
> Also, I wanted to enumerate currently open questions:
> 
> ==
> 
> Question i) Just how deterministic can these CoCo VMs be? Elena
> pointed to some TDX code regarding RDTSC that seemed fairly damning,
> but I also wonder what gotchas a motivated researcher might run into
> and how those could help us (or not).

This would be great imo to have a discussion on. I don’t think the internal
design or implementation of TDX module is complicated to scare
anyone off. So I think it would be a question on how practical would
be for VMM to make such an attack on guest kernel? A lot of times such
things are about precision, reliability and an ability to filter out the noise.  
So questions like how precisely *in practice* can VMM measure guest's
virtual TSC and other parameters that are used as entropy inputs?

But overall both in crypto and security, we don’t like to be too near the
security bounds, because we always assume our understanding might be
incomplete, so putting a reasonable and clear countermeasure is usually the
better approach. 

> 
> Question ii) Just how DoS-able is RDRAND? From host to guest, where
> the host controls scheduling, that seems easier, but how much so, and
> what's the granularity of these operations, and could retries still
> help, or not at all? What about from guest to guest, where the
> scheduling is out of control; in that case is there a value of N for
> which N retries makes it actually impossible to DoS? What about from
> userspace to kernelspace; good value of N?

All valid questions that I am also trying to understand the answers. 

Best Regards,
Elena.

> 
> Question iii) How likely is Intel to actually fix this in a
> satisfactory way (see "specifying this is an interesting question" in
> [1])? And if they would, what would the timeline even be?
> 
> ==
> 
> Anyway, that's about where I'm at. I figure I'll wait to see if the
> internal inquiry within Intel yields anything interesting, and then
> maybe we can move forward with solutions (B) or (F) or (G) or a
> different Roald Dahl novel instead.
> 
> Jason
> 
> [1] https://lore.kernel.org/all/CAHmME9ps6W5snQrYeNVMFgfhMKFKciky=-
> UxxGFbAx_RrxSHoA@mail.gmail.com/

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Fri, Feb 02, 2024 at 07:25:42AM +0000, Reshetova, Elena wrote:
> This is a great summary of options, thank you Jason!
> My proposal would be to wait on result of our internal investigation 
> before proceeding to choose the approach. 

I'm happy for the option "Do nothing for now", but if we do want to do
something in the absence of more detailed information, I'd suggest
doing something simple for first, on the theory that it doesn't make
things worse, and we can always do something more complicated if it
turns out to be needed.

In that vein, my suggestion is:

> > Solution B) BUG_ON(is_early_boot && is_coco_system) in the RDRAND
> > failure path (> 10 retries).
> > 
> > This is slightly less simple than A, because we have to plumb
> > CoCo-detection through to the RDRAND helper. [Side note: I feel
> > ridiculous typing 'CoCo'.] Systems-wise, I don't see drawbacks.
> > RNG-wise, the drawback is that this doesn't help deal with secure
> > reseeding later in time, which is a RNG property that we otherwise
> > enjoy.

If there isn't a global variable we can test to see if Confidential
Compute is enabled, I suspect we should just add one.  I would assume
that /dev/random isn't the only place where we might need to do
whether Confidential Compute is enabled.

So I don't think plumbing CC into the /dev/random code, and since we
are only doing this in early boot, I wouldn't put it in the RDRAND
helper, but rather in the caller of the RDRAND helper that gets used
in the early boot path.

(Side note, internally, at least in my part of my company, we use CC
as the acronym of convenience.  And any comments that I make are my
own opinion, and do not reflect the positions or desires of my
employer...)

> > Question iii) How likely is Intel to actually fix this in a
> > satisfactory way (see "specifying this is an interesting question" in
> > [1])? And if they would, what would the timeline even be?

Here are at least two obvious ways that Intel could fix or mitigate
this issue:

(1) Add more hardware RNG IP's for chips with a huge number of cores.
This is the *obvious* way to address the problem with hundreds of CPU
cores, although it's only something that can be done on newer chips.

(2) Have a per-core throttle where a core is not allowed to issue
RDRAND or RDSEED instructions more than N times per millisecond (or
some other unit of time).  So long as N is larger than the maximum
number of SSL connections that a front-end server can actually
terminate, it's not going to impact legitimate workloads.  This can be
approximated by the number of public key operations per unit time that
a single CPU core achieve.  And if RDRAND isn't sufficient to support
that today, then see solution (1), or CPU customers should switch to
some other CPU vendor that can...  

(3) Provide some kind of perf counter so the host can see which cores
are issuing a huge number of RDRAND/RDSEED instructions, and which
cores have been suffering from entropy exhaustion RDRAND/RDSEED
failures.  This would allow the operator of the host to detect which
VM's might be carrying out DOS attacks, so that the operator can kill
those VM's, and disable the customer account that was launching these
abusive VM's.


Hopefully mitigation #2 in particular (and maybe mitigation #3) is
something that Intel could implement as a firmware update; I'd love
comments from Intel if that is the case.

I'll also note that the threat model where customer A is trying to
start a CC VM, and customer B has purchased VM's that use all of the
other cores on a server, is primarily the sort of thing that a public
cloud vendor would need to worry about.  And *if* this become a real
issue, where some researcher demonstrates that there is a problem, the
cloud provider will be hugely incentivized to apply host-side
mitigations, and to lean on the guest OS providers to apply guest-side
mitigations.

So if this is only a DOS which applies for CC VM's, and it turns out
that solution (B) is not sufficient, we can do something more
complicated, such as having the guest retry the RDRAND instruction for
ten seconds.  And if some hypothetical "RandExhaust" attack is being
written about by the New York Times, I suspect it won't be that hard
to get Red Hat to apply mitigations to the RHEL kernel.  :-)

So I don't think it really is *that* big of a deal; if it turns out to
be an issue, we will be able to deal with it.

      	     	     	     	     - Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Ted, Kirill,

On Fri, Feb 02, 2024 at 10:39:27AM -0500, Theodore Ts'o wrote:
> On Fri, Feb 02, 2024 at 07:25:42AM +0000, Reshetova, Elena wrote:
> > This is a great summary of options, thank you Jason!
> > My proposal would be to wait on result of our internal investigation 
> > before proceeding to choose the approach. 
> 
> I'm happy for the option "Do nothing for now", but if we do want to do
> something in the absence of more detailed information, I'd suggest
> doing something simple for first, on the theory that it doesn't make
> things worse, and we can always do something more complicated if it
> turns out to be needed.
> 
> In that vein, my suggestion is:
> 
> > > Solution B) BUG_ON(is_early_boot && is_coco_system) in the RDRAND
> > > failure path (> 10 retries).
> > > 
> > > This is slightly less simple than A, because we have to plumb
> > > CoCo-detection through to the RDRAND helper. [Side note: I feel
> > > ridiculous typing 'CoCo'.] Systems-wise, I don't see drawbacks.
> > > RNG-wise, the drawback is that this doesn't help deal with secure
> > > reseeding later in time, which is a RNG property that we otherwise
> > > enjoy.
> 
> If there isn't a global variable we can test to see if Confidential
> Compute is enabled, I suspect we should just add one.  I would assume
> that /dev/random isn't the only place where we might need to do
> whether Confidential Compute is enabled.
> 
> So I don't think plumbing CC into the /dev/random code, and since we
> are only doing this in early boot, I wouldn't put it in the RDRAND
> helper, but rather in the caller of the RDRAND helper that gets used
> in the early boot path.

Yea, actually, I had a pretty similar idea for something like that
that's very non-invasive, where none of this even touches the RDRAND
core code, much less random.c. Specifically, we consider "adding some
extra RDRAND to the pool" like any other driver that wants to add some
of its own seeds to the pool, with add_device_randomness(), a call that
lives in various driver code, doesn't influence any entropy readiness
aspects of random.c, and can safely be sprinkled in any device or
platform driver.

Specifically what I'm thinking about is something like:

void coco_main_boottime_init_function_somewhere_deep_in_arch_code(void)
{
  // [...]
  // bring up primary CoCo nuts
  // [...]

  /* CoCo requires an explicit RDRAND seed, because the host can make the
   * rest of the system deterministic.
   */
  unsigned long seed[32 / sizeof(long)];
  size_t i, longs;
  for (i = 0; i < ARRAY_SIZE(seed); i += longs) {
    longs = arch_get_random_longs(&seed[i], ARRAY_SIZE(seed) - i);
    /* If RDRAND is being DoS'd, panic, because we can't ensure
     * confidentiality.
     */
    BUG_ON(!longs);
  }
  add_device_randomness(seed, sizeof(seed));
  memzero_explicit(seed, sizeof(seed));

  // [...]
  // do other CoCo things
  // [...]
}

I would have no objection to the CoCo people adding something like this
and would give it my Ack, but more importantly, my Ack for that doesn't
even matter, because add_device_randomness() is pretty innocuous.

So Kirill, if nobody else here objects to that approach, and you want to
implement it in some super minimal way like that, that would be fine
with me. Or maybe we want to wait for that internal inquiry at Intel to
return some answers first. But either way, this might be an easy
approach that doesn't add too much complexity.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Kirill,

On Sat, Feb 3, 2024 at 11:12 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> Yea, actually, I had a pretty similar idea for something like that
> that's very non-invasive, where none of this even touches the RDRAND
> core code, much less random.c. Specifically, we consider "adding some
> extra RDRAND to the pool" like any other driver that wants to add some
> of its own seeds to the pool, with add_device_randomness(), a call that
> lives in various driver code, doesn't influence any entropy readiness
> aspects of random.c, and can safely be sprinkled in any device or
> platform driver.
>
> Specifically what I'm thinking about is something like:
>
> void coco_main_boottime_init_function_somewhere_deep_in_arch_code(void)
> {
>   // [...]
>   // bring up primary CoCo nuts
>   // [...]
>
>   /* CoCo requires an explicit RDRAND seed, because the host can make the
>    * rest of the system deterministic.
>    */
>   unsigned long seed[32 / sizeof(long)];
>   size_t i, longs;
>   for (i = 0; i < ARRAY_SIZE(seed); i += longs) {
>     longs = arch_get_random_longs(&seed[i], ARRAY_SIZE(seed) - i);
>     /* If RDRAND is being DoS'd, panic, because we can't ensure
>      * confidentiality.
>      */
>     BUG_ON(!longs);
>   }
>   add_device_randomness(seed, sizeof(seed));
>   memzero_explicit(seed, sizeof(seed));
>
>   // [...]
>   // do other CoCo things
>   // [...]
> }
>
> I would have no objection to the CoCo people adding something like this
> and would give it my Ack, but more importantly, my Ack for that doesn't
> even matter, because add_device_randomness() is pretty innocuous.
>
> So Kirill, if nobody else here objects to that approach, and you want to
> implement it in some super minimal way like that, that would be fine
> with me. Or maybe we want to wait for that internal inquiry at Intel to
> return some answers first. But either way, this might be an easy
> approach that doesn't add too much complexity.

I went ahead and implemented this just to have something concrete out there:
https://lore.kernel.org/all/20240209164946.4164052-1-Jason@zx2c4.com/

I probably screwed up some x86 platform conventions/details, but
that's the general idea I had in mind.

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> Hi Kirill,
> 
> On Sat, Feb 3, 2024 at 11:12 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
> > Yea, actually, I had a pretty similar idea for something like that
> > that's very non-invasive, where none of this even touches the RDRAND
> > core code, much less random.c. Specifically, we consider "adding some
> > extra RDRAND to the pool" like any other driver that wants to add some
> > of its own seeds to the pool, with add_device_randomness(), a call that
> > lives in various driver code, doesn't influence any entropy readiness
> > aspects of random.c, and can safely be sprinkled in any device or
> > platform driver.
> >
> > Specifically what I'm thinking about is something like:
> >
> > void coco_main_boottime_init_function_somewhere_deep_in_arch_code(void)
> > {
> >   // [...]
> >   // bring up primary CoCo nuts
> >   // [...]
> >
> >   /* CoCo requires an explicit RDRAND seed, because the host can make the
> >    * rest of the system deterministic.
> >    */
> >   unsigned long seed[32 / sizeof(long)];
> >   size_t i, longs;
> >   for (i = 0; i < ARRAY_SIZE(seed); i += longs) {
> >     longs = arch_get_random_longs(&seed[i], ARRAY_SIZE(seed) - i);
> >     /* If RDRAND is being DoS'd, panic, because we can't ensure
> >      * confidentiality.
> >      */
> >     BUG_ON(!longs);
> >   }
> >   add_device_randomness(seed, sizeof(seed));
> >   memzero_explicit(seed, sizeof(seed));
> >
> >   // [...]
> >   // do other CoCo things
> >   // [...]
> > }
> >
> > I would have no objection to the CoCo people adding something like this
> > and would give it my Ack, but more importantly, my Ack for that doesn't
> > even matter, because add_device_randomness() is pretty innocuous.
> >
> > So Kirill, if nobody else here objects to that approach, and you want to
> > implement it in some super minimal way like that, that would be fine
> > with me. Or maybe we want to wait for that internal inquiry at Intel to
> > return some answers first. But either way, this might be an easy
> > approach that doesn't add too much complexity.
> 
> I went ahead and implemented this just to have something concrete out there:
> https://lore.kernel.org/all/20240209164946.4164052-1-Jason@zx2c4.com/
> 
> I probably screwed up some x86 platform conventions/details, but
> that's the general idea I had in mind.
> 

Thank you Jason!
I want to bring another potential idea here for a discussion, which Peter Anvin
proposed in our internal discussions, and I like it conceptually better
than any options we discussed so far since it is much more generic. 

What if we instead of doing some special treatment on rdrand/seed, we
try to fix the underneath problem of Linux RNG not supporting CoCo threat
model. Linux RNG has almost set in stone definition of what sources contribute
entropy and what don’t (with some additional flexibility with flags like trust_cpu).
This works well for the current fixed threat model, but doesn’t work for
CoCo because some sources are suddenly not trusted anymore to contribute
entropy. However, some are still trusted and that is not just rdrand/rdseed,
but we would also trust add_hwgenerator_randomness (given that we use
TEE IO device here or have a way to get this input securely). So, even in
theoretical scenario that both rdrand/rdseed is broken (let's say HW failure),
a Linux RNG can actually boot securely in the guest if we have enough
entropy from add_hwgenerator_randomness.

So the change would be around adding the notion of conditional entropy
counting (we will always take input as we do now because it wont hurt),
which would automatically give us a correct behavior in _credit_init_bits()
for initial seeding of crng. Also we need to have a generic way to stop the
boot if the entropy is not increasing (for any reasons) and prevent booting
with insecurely seeded crng. 

I do understand that this is going to be much bigger change than anything we
are discussing so far, but conceptually it sounds right to be able to have a say
what sources of entropy one trusts in runtime (probably applicable beyond
CoCo in the future also) and what is the action when we cannot collect the
entropy from these sources. 

What does everyone think? 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Mon, Feb 12, 2024 at 08:25:33AM +0000, Reshetova, Elena wrote:
> What if we instead of doing some special treatment on rdrand/seed, we
> try to fix the underneath problem of Linux RNG not supporting CoCo threat
> model. Linux RNG has almost set in stone definition of what sources contribute
> entropy and what don’t (with some additional flexibility with flags like trust_cpu).
> This works well for the current fixed threat model, but doesn’t work for
> CoCo because some sources are suddenly not trusted anymore to contribute
> entropy. However, some are still trusted and that is not just rdrand/rdseed,
> but we would also trust add_hwgenerator_randomness (given that we use
> TEE IO device here or have a way to get this input securely). So, even in
> theoretical scenario that both rdrand/rdseed is broken (let's say HW failure),
> a Linux RNG can actually boot securely in the guest if we have enough
> entropy from add_hwgenerator_randomness.

So the problem with this is that there is now way we can authenticate
the hardware RNG.  For example, the hypervisor could claim that there
is a ChaosKey USB key attached, and at the moment, unlike all other
hardware random number generators, the Linux kernel is configured to
blindly trust the ChaosKey because it was designed by Keith Packard
and Bdale Garbee, and "It Must Be Good".  But the only way that we
know that it is a ChaosKey is by its USB major and minor id numbers
--- and a malicious hypervisor could fake up such a device.

And of course, that's not unique to the hypervisor --- someone could
create a hardware USB key that claimed to be a ChaosKey, but which
generated a fixed sequence, say 3,1,4,1,5,9,2,6,... and it would pass
most RNG quality checkers, since it's obviously not a repeated
sequence of digits, so the mandated FIPS required check would give it
a thumbs up.  And it doesn't have to be a faked ChaosKey device; a
hypervisor could claim that there is a virtual TPM with its hardware
random number generator, but it's also gimmicked to always give the
same fixed sequence, and there's no way the guest OS could know
otherwise.

Hence, for the unique requirements of Confidential Compute, I'm afraid
it's RDRAND/RSEED or bust....

						- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dan Williams]

Theodore Ts'o wrote:
> On Mon, Feb 12, 2024 at 08:25:33AM +0000, Reshetova, Elena wrote:
> > What if we instead of doing some special treatment on rdrand/seed, we
> > try to fix the underneath problem of Linux RNG not supporting CoCo threat
> > model. Linux RNG has almost set in stone definition of what sources contribute
> > entropy and what don’t (with some additional flexibility with flags like trust_cpu).
> > This works well for the current fixed threat model, but doesn’t work for
> > CoCo because some sources are suddenly not trusted anymore to contribute
> > entropy. However, some are still trusted and that is not just rdrand/rdseed,
> > but we would also trust add_hwgenerator_randomness (given that we use
> > TEE IO device here or have a way to get this input securely). So, even in
> > theoretical scenario that both rdrand/rdseed is broken (let's say HW failure),
> > a Linux RNG can actually boot securely in the guest if we have enough
> > entropy from add_hwgenerator_randomness.
> 
> So the problem with this is that there is now way we can authenticate
> the hardware RNG.

Sure there is, that is what, for example, PCI TDISP (TEE Device
Interface Security Protocol) is about. Set aside the difficulty of doing
the PCI TDISP flow early in boot, and validating the device certficate
and measurements based on golden values without talking to a remote
verifier etc..., but if such a device has been accepted and its driver
calls hwrng_register() it should be added as an entropy source.

Now maybe there is something fatal in that "etc", and RDRAND needs to
work for early entropy, but if a PCI device passes guest acceptance
there should be no additional concerns for it to be considered a CC
approved RNG.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Mon, Feb 12, 2024 at 11:28:31PM -0800, Dan Williams wrote:
> Sure there is, that is what, for example, PCI TDISP (TEE Device
> Interface Security Protocol) is about. Set aside the difficulty of doing
> the PCI TDISP flow early in boot, and validating the device certficate
> and measurements based on golden values without talking to a remote
> verifier etc..., but if such a device has been accepted and its driver
> calls hwrng_register() it should be added as an entropy source.

How real is TDISP?  What hardware exists today and how much of this
support is ready to land in the kernel?  Looking at the news articles,
it appears to me like bleeding edge technology, and what an unkind
person might call "vaporware"?  Is that an unfair characterization?

There have plenty of things that have squirted out of standards
bodies, like for example, "objected base storage", which has turned
out to be a complete commercial failure and was never actually
deployed in any real numbers, other than sample hardare being provided
to academic researchers.  How can we be sure that PCI TDISP won't end
up going down that route?

In any case, if we are going to go down this path, we will need to
have some kind of policy engine hwrng_register() reject
non-authenticated hardware if Confidential Compute is enabled (and
possibly in other cases).

				- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dan Williams]

Theodore Ts'o wrote:
> On Mon, Feb 12, 2024 at 11:28:31PM -0800, Dan Williams wrote:
> > Sure there is, that is what, for example, PCI TDISP (TEE Device
> > Interface Security Protocol) is about. Set aside the difficulty of doing
> > the PCI TDISP flow early in boot, and validating the device certficate
> > and measurements based on golden values without talking to a remote
> > verifier etc..., but if such a device has been accepted and its driver
> > calls hwrng_register() it should be added as an entropy source.
> 
> How real is TDISP?  What hardware exists today and how much of this
> support is ready to land in the kernel?  Looking at the news articles,
> it appears to me like bleeding edge technology, and what an unkind
> person might call "vaporware"?  Is that an unfair characterization?

Indeed it is. Typically when you have x86, riscv, arm, and s390 folks
all show up at a Linux Plumbers session [1] to talk about their approach
to handling a new platform paradigm, that is a decent indication that
the technology is more real than not. Point taken that it is not here
today, but it is also not multiple hardware generations away as the
Plumbers participation indicated.

> There have plenty of things that have squirted out of standards
> bodies, like for example, "objected base storage", which has turned
> out to be a complete commercial failure and was never actually
> deployed in any real numbers, other than sample hardare being provided
> to academic researchers.  How can we be sure that PCI TDISP won't end
> up going down that route?

Of course, that is always a risk. History is littered with obsolesence,
some of it before seeing any commercial uptake, some after.

> In any case, if we are going to go down this path, we will need to
> have some kind of policy engine hwrng_register() reject
> non-authenticated hardware if Confidential Compute is enabled (and
> possibly in other cases).

Sounds reasonable, that recognition is all I wanted from mentioning PCI
TDISP.

[1]: https://lpc.events/event/17/contributions/1633/

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Tue, Feb 13, 2024 at 04:53:06PM -0800, Dan Williams wrote:
> 
> Indeed it is. Typically when you have x86, riscv, arm, and s390 folks
> all show up at a Linux Plumbers session [1] to talk about their approach
> to handling a new platform paradigm, that is a decent indication that
> the technology is more real than not. Point taken that it is not here
> today, but it is also not multiple hardware generations away as the
> Plumbers participation indicated.

My big concerns with TDISP which make me believe it may not be a
silver bullet is that (a) it's hyper-complex (although to be fair
Confidential Compute isn't exactly simple, and (b) it's one thing to
digitally sign software so you know that it comes from a trusted
source; but it's a **lot** harder to prove that hardware hasn't been
tampered with --- a digital siganture can't tell you much about
whether or not the hardware is in an as-built state coming from the
factory --- this requires things like wrapping the device with
resistive wire in multiple directions with a whetstone bridge to
detect if the wire has gotten cut or shorted, then dunking the whole
thing in epoxy, so that any attempt to tamper with the hardware will
result it self-destructing (via a thermite charge or equivalent :-)

Remember, the whole conceit of Confidential Compute is that you don't
trust the cloud provider --- but if that entity controls the PCI cards
installed in their servers, and and that entity has the ability to
*modify* the PCI cards in the server, all of the digital signatures
and fancy-schmancy TDISP complexity isn't necessarily going to save
you.

The final concern is that it may take quite a while before these
devices become real, and then for cloud providers like Amazon, Azure,
to actually deploy them.  And in the meantime, Confidential Compute
VM's are already something which are available for customers to
purchase *today*.  So we need some kind of solution right now, and
preferably, something which is simple enough that it is likely to be
back-portable to RHEL.

(And I fear that even if TDISP hardware existed today, it is so
complicated that it may be a heavy lift to get it backported into
enterprise distro kernels.)

Ultimately, if CPU's can actually have an architectgural RNG ala
RDRAND/RDSEED that actually can do the right thing in the face of
entropy draining attacks, that seems to be a **much** simpler
solution.  And even if it requires waiting for the next generation of
CPU's, this might be faster than waiting for the TDISP ecosystem
mature.

					- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dan Williams]

Theodore Ts'o wrote:
> On Tue, Feb 13, 2024 at 04:53:06PM -0800, Dan Williams wrote:
> > 
> > Indeed it is. Typically when you have x86, riscv, arm, and s390 folks
> > all show up at a Linux Plumbers session [1] to talk about their approach
> > to handling a new platform paradigm, that is a decent indication that
> > the technology is more real than not. Point taken that it is not here
> > today, but it is also not multiple hardware generations away as the
> > Plumbers participation indicated.
> 
> My big concerns with TDISP which make me believe it may not be a
> silver bullet is that (a) it's hyper-complex (although to be fair
> Confidential Compute isn't exactly simple, and (b) it's one thing to
> digitally sign software so you know that it comes from a trusted
> source; but it's a **lot** harder to prove that hardware hasn't been
> tampered with --- a digital siganture can't tell you much about
> whether or not the hardware is in an as-built state coming from the
> factory --- this requires things like wrapping the device with
> resistive wire in multiple directions with a whetstone bridge to
> detect if the wire has gotten cut or shorted, then dunking the whole
> thing in epoxy, so that any attempt to tamper with the hardware will
> result it self-destructing (via a thermite charge or equivalent :-)
> 
> Remember, the whole conceit of Confidential Compute is that you don't
> trust the cloud provider --- but if that entity controls the PCI cards
> installed in their servers, and and that entity has the ability to
> *modify* the PCI cards in the server, all of the digital signatures
> and fancy-schmancy TDISP complexity isn't necessarily going to save
> you.
>
> The final concern is that it may take quite a while before these
> devices become real, and then for cloud providers like Amazon, Azure,
> to actually deploy them.  And in the meantime, Confidential Compute
> VM's are already something which are available for customers to
> purchase *today*.  So we need some kind of solution right now, and
> preferably, something which is simple enough that it is likely to be
> back-portable to RHEL.
> 
> (And I fear that even if TDISP hardware existed today, it is so
> complicated that it may be a heavy lift to get it backported into
> enterprise distro kernels.)

No lies detected.

Something is broken if you need to rely on TDISP to get a reliable
random number in a guest. All it can enforce is that the VMM is not
emulating a HWRNG. Also, VMM denial of service is outside of the TDISP
threat model, so if VMM can steal all the entropy, or DoS RDSEED, you
are back at square one. The only reason for jumping in on this tangent
was to counterpoint the implication that the RNG core must always hard
code a dependency on CPU HWRNG for confidential computing.

However, yes, given the timelines for TDISP Linux could hard code that
choice in the near term for expediency and leave it to the TDISP folks
to unwind it later.

> Ultimately, if CPU's can actually have an architectgural RNG ala
> RDRAND/RDSEED that actually can do the right thing in the face of
> entropy draining attacks, that seems to be a **much** simpler
> solution.  And even if it requires waiting for the next generation of
> CPU's, this might be faster than waiting for the TDISP ecosystem
> mature.

Yes, please. I am happy if TDISP flies below the hype cycle so that its
implications can be considered carefullly. At the same time I will keep
an eye out for discussions like this where guest attestation of hardware
provenance is raised.

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

 
> Ultimately, if CPU's can actually have an architectgural RNG ala
> RDRAND/RDSEED that actually can do the right thing in the face of
> entropy draining attacks, that seems to be a **much** simpler
> solution.  

I don’t think anyone would object that the rdrand approach we are
discussing here is simpler. My point (and also Peter original idea) was
that if we want to do it correctly and generically (and *not* just about 
confidential computing), we ought to provide a way for users to define
what entropy sources for Linux RNG they are willing to trust or not.
This should not be a policy decision that kernel hardcodes (we try hard
to avoid policies in kernel), but left for users to decide/configure based
on their preferences, trust notions, fears of backdooring, whatelse.
This of course has the flip part that some users will get it wrong, but
reasonable secure defaults can be provided also. 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Nikolay Borisov]

On 14.02.24 г. 6:32 ч., Theodore Ts'o wrote:
> On Tue, Feb 13, 2024 at 04:53:06PM -0800, Dan Williams wrote:
>>
>> Indeed it is. Typically when you have x86, riscv, arm, and s390 folks
>> all show up at a Linux Plumbers session [1] to talk about their approach
>> to handling a new platform paradigm, that is a decent indication that
>> the technology is more real than not. Point taken that it is not here
>> today, but it is also not multiple hardware generations away as the
>> Plumbers participation indicated.
> 
> My big concerns with TDISP which make me believe it may not be a
> silver bullet is that (a) it's hyper-complex (although to be fair
> Confidential Compute isn't exactly simple, and (b) it's one thing to
> digitally sign software so you know that it comes from a trusted
> source; but it's a **lot** harder to prove that hardware hasn't been
> tampered with --- a digital siganture can't tell you much about
> whether or not the hardware is in an as-built state coming from the
> factory --- this requires things like wrapping the device with
> resistive wire in multiple directions with a whetstone bridge to
> detect if the wire has gotten cut or shorted, then dunking the whole
> thing in epoxy, so that any attempt to tamper with the hardware will
> result it self-destructing (via a thermite charge or equivalent :-)

This really reminds me of the engineering that goes into the omnipresent 
POS terminals ate every store, since they store certificates from the 
card (Visa/Master) operators. So I wonder if at somepoint we'll have a 
pos-like device (by merit of its engineering) in every server....

> 
> Remember, the whole conceit of Confidential Compute is that you don't
> trust the cloud provider --- but if that entity controls the PCI cards
> installed in their servers, and and that entity has the ability to
> *modify* the PCI cards in the server, all of the digital signatures
> and fancy-schmancy TDISP complexity isn't necessarily going to save
> you.

Can't the same argument go for the CPU, though it's a lot more 
"integrated" into the silicong substrate, yet we somehow believe CoCo 
ascertains that a vm is running on trusted hardware? But ultimately the 
CPU is still a part that comes from the untrusted CSP.


<snip>

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dr. Greg]

On Wed, Feb 14, 2024 at 10:34:48AM +0200, Nikolay Borisov wrote:

Hi, I hope the week is going well for everyone.

> On 14.02.24 ??. 6:32 ??., Theodore Ts'o wrote:
> >On Tue, Feb 13, 2024 at 04:53:06PM -0800, Dan Williams wrote:
> >>
> >>Indeed it is. Typically when you have x86, riscv, arm, and s390 folks
> >>all show up at a Linux Plumbers session [1] to talk about their approach
> >>to handling a new platform paradigm, that is a decent indication that
> >>the technology is more real than not. Point taken that it is not here
> >>today, but it is also not multiple hardware generations away as the
> >>Plumbers participation indicated.
> >
> >My big concerns with TDISP which make me believe it may not be a
> >silver bullet is that (a) it's hyper-complex (although to be fair
> >Confidential Compute isn't exactly simple, and (b) it's one thing to
> >digitally sign software so you know that it comes from a trusted
> >source; but it's a **lot** harder to prove that hardware hasn't been
> >tampered with --- a digital siganture can't tell you much about
> >whether or not the hardware is in an as-built state coming from the
> >factory --- this requires things like wrapping the device with
> >resistive wire in multiple directions with a whetstone bridge to
> >detect if the wire has gotten cut or shorted, then dunking the whole
> >thing in epoxy, so that any attempt to tamper with the hardware will
> >result it self-destructing (via a thermite charge or equivalent :-)

> This really reminds me of the engineering that goes into the
> omnipresent POS terminals ate every store, since they store
> certificates from the card (Visa/Master) operators. So I wonder if
> at somepoint we'll have a pos-like device (by merit of its
> engineering) in every server....

It already exists.  CoCo, at least the Intel implementation, is
dependent on what amounts to this concept.

> >Remember, the whole conceit of Confidential Compute is that you don't
> >trust the cloud provider --- but if that entity controls the PCI cards
> >installed in their servers, and and that entity has the ability to
> >*modify* the PCI cards in the server, all of the digital signatures
> >and fancy-schmancy TDISP complexity isn't necessarily going to save
> >you.

> Can't the same argument go for the CPU, though it's a lot more
> "integrated" into the silicong substrate, yet we somehow believe
> CoCo ascertains that a vm is running on trusted hardware? But
> ultimately the CPU is still a part that comes from the untrusted
> CSP.

The attestation model for TDX is largely built on top of SGX.

The Intel predicate with respect to SGX/TDX is that you have to trust
the CPU silicon implementation, if you can't entertain that level of
trust, it is game over for security.

To support that security model, Intel provides infrastructure that
proves that the software is running on a 'Genuine Intel' CPU.

Roughly, a root key is burned into the silicon that is used as the
basis for additional derived keys.  The key access and derivation
processes can only occur when the process is running software with a
known signature in a protected region of memory (enclave).

The model is to fill a structure with data that defines the
hardware/software state.  A keyed checksum is run over the structure
that allows a relying party to verify that the data structure contents
could have only been generated on a valid Intel CPU.

This process verifies that the CPU is from a known vendor, which is of
course only the initial starting point for verifying that something
like a VM is running in a known and trusted state.  But, if you can't
start with that predicate you have nothing to build on.

The actual implementation nowadays is a bit more complex, given that
all of this has to happen on multi-socket systems which involve more
than one CPU, but the concept is the same.

Have a good day.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity
              https://github.com/Quixote-Project

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Mon, Feb 12, 2024 at 08:25:33AM +0000, Reshetova, Elena wrote:
> So the change would be around adding the notion of conditional entropy
> counting (we will always take input as we do now because it wont hurt),
> which would automatically give us a correct behavior in _credit_init_bits()
> for initial seeding of crng.

I basically have zero interest in this kind of highly complex addition,
and I think that'll lead us back toward how the RNG was in the past.
"Entropy counting" is mostly an illusion, at least in terms of doing so
from measurement. We've got some heuristics to mitigate "premature
first" but these things will mostly only ever be heuristic. If a
platform like CoCo knows nothing else will work, then a
platform-specific choice like the one in this patch is sufficient to
do the trick. And in general, this seems like a weird thing to design
around: if the CPU is actually just totally broken and defective, maybe
CoCo shouldn't continue executing anyway? So I'm pretty loathe to go in
this direction of highly complex policy frameworks and such.

Anyway, based on your last email (and my reply to it), it seems like
we're mostly in the clear anyway, and we can rely on RDRAND failure ==>
hardware failure.

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

 
> This is a great summary of options, thank you Jason!
> My proposal would be to wait on result of our internal investigation
> before proceeding to choose the approach.

Hi everyone, 

I am finally able to share the result of my AR and here is the statement
about rdrand/rdseed on Intel platforms:

"The RdRand in a non-defective device is designed to be faster than the bus,
so when a core accesses the output from the DRNG, it will always get a
random number.
As a result, it is hard to envision a scenario where the RdRand, on a fully
functional device, will underflow.
The carry flag after RdRand signals an underflow so in the case of a defective chip,
this will prevent the code thinking it has a random number when it does not.

RdSeed however is limited by the speed of the noise source. So it is not faster
than the bus and there may be an underflow signaled by the carry flag. 
When reading for multiple values, the total throughput of RdSeed random
numbers varies over different products due to variation in the silicon processes,
operating voltage and speed vs power tradeoffs.
The throughput is shared between the cores"

In addition there is a plan to publish a whitepaper and add clarifications to
Intel official documentation on this topic, but this would obviously take longer. 

Best Regards,
Elena.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Elena,

On Wed, Feb 14, 2024 at 4:18 PM Reshetova, Elena <elena.reshetova@intel.com> wrote:
> "The RdRand in a non-defective device is designed to be faster than the bus,
> so when a core accesses the output from the DRNG, it will always get a
> random number.
> As a result, it is hard to envision a scenario where the RdRand, on a fully
> functional device, will underflow.
> The carry flag after RdRand signals an underflow so in the case of a defective chip,
> this will prevent the code thinking it has a random number when it does not.

That's really great news, especially combined with a very similar
statement from Borislav about AMD chips:

On Fri, Feb 9, 2024 at 10:45 PM Borislav Petkov <bp@alien8.de> wrote:
> Yeah, I know exactly what you mean and I won't go into details for
> obvious reasons. Two things:
>
> * Starting with Zen3, provided properly configured hw RDRAND will never
> fail. It is also fair when feeding the different contexts.

I assume that this faster-than-the-bus-ness also takes into account the
various accesses required to even switch contexts when scheduling VMs,
so your proposed host-guest scheduling attack can't really happen
either. Correct?

One clarifying question in all of this: what is the point of the "try 10
times" advice? Is the "faster than the bus" statement actually "faster
than the bus if you try 10 times"? Or is the "10 times" advice just old
and not relevant.

In other words, is the following a reasonable patch?

diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
index 02bae8e0758b..2d5bf5aa9774 100644
--- a/arch/x86/include/asm/archrandom.h
+++ b/arch/x86/include/asm/archrandom.h
@@ -13,22 +13,16 @@
 #include <asm/processor.h>
 #include <asm/cpufeature.h>
 
-#define RDRAND_RETRY_LOOPS	10
-
 /* Unconditional execution of RDRAND and RDSEED */
 
 static inline bool __must_check rdrand_long(unsigned long *v)
 {
 	bool ok;
-	unsigned int retry = RDRAND_RETRY_LOOPS;
-	do {
-		asm volatile("rdrand %[out]"
-			     CC_SET(c)
-			     : CC_OUT(c) (ok), [out] "=r" (*v));
-		if (ok)
-			return true;
-	} while (--retry);
-	return false;
+	asm volatile("rdrand %[out]"
+		     CC_SET(c)
+		     : CC_OUT(c) (ok), [out] "=r" (*v));
+	WARN_ON(!ok);
+	return ok;
 }
 
 static inline bool __must_check rdseed_long(unsigned long *v)

(As for the RDSEED clarification, that also matches Borislav's reply, is
what we expected and knew experimentally, and doesn't really have any
bearing on Linux's RNG or this discussion, since RDRAND is all we need
anyway.)

Regards,
Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

 Hi Elena,
> 
> On Wed, Feb 14, 2024 at 4:18 PM Reshetova, Elena <elena.reshetova@intel.com>
> wrote:
> > "The RdRand in a non-defective device is designed to be faster than the bus,
> > so when a core accesses the output from the DRNG, it will always get a
> > random number.
> > As a result, it is hard to envision a scenario where the RdRand, on a fully
> > functional device, will underflow.
> > The carry flag after RdRand signals an underflow so in the case of a defective chip,
> > this will prevent the code thinking it has a random number when it does not.
> 
> That's really great news, especially combined with a very similar
> statement from Borislav about AMD chips:
> 
> On Fri, Feb 9, 2024 at 10:45 PM Borislav Petkov <bp@alien8.de> wrote:
> > Yeah, I know exactly what you mean and I won't go into details for
> > obvious reasons. Two things:
> >
> > * Starting with Zen3, provided properly configured hw RDRAND will never
> > fail. It is also fair when feeding the different contexts.
> 
> I assume that this faster-than-the-bus-ness also takes into account the
> various accesses required to even switch contexts when scheduling VMs,
> so your proposed host-guest scheduling attack can't really happen
> either. Correct?

Yes, this attack wont be possible for rdrand, so we are good. 

> 
> One clarifying question in all of this: what is the point of the "try 10
> times" advice? Is the "faster than the bus" statement actually "faster
> than the bus if you try 10 times"? Or is the "10 times" advice just old
> and not relevant.

The whitepaper should clarify this more in the future, but in short 
10 times retry is not relevant based on the above statement. 
"when core accesses the output from the DRNG, it will always get a
random number" - there are no statements of re-try here. 

> 
> In other words, is the following a reasonable patch?
> 
> diff --git a/arch/x86/include/asm/archrandom.h
> b/arch/x86/include/asm/archrandom.h
> index 02bae8e0758b..2d5bf5aa9774 100644
> --- a/arch/x86/include/asm/archrandom.h
> +++ b/arch/x86/include/asm/archrandom.h
> @@ -13,22 +13,16 @@
>  #include <asm/processor.h>
>  #include <asm/cpufeature.h>
> 
> -#define RDRAND_RETRY_LOOPS	10
> -
>  /* Unconditional execution of RDRAND and RDSEED */
> 
>  static inline bool __must_check rdrand_long(unsigned long *v)
>  {
>  	bool ok;
> -	unsigned int retry = RDRAND_RETRY_LOOPS;
> -	do {
> -		asm volatile("rdrand %[out]"
> -			     CC_SET(c)
> -			     : CC_OUT(c) (ok), [out] "=r" (*v));
> -		if (ok)
> -			return true;
> -	} while (--retry);
> -	return false;
> +	asm volatile("rdrand %[out]"
> +		     CC_SET(c)
> +		     : CC_OUT(c) (ok), [out] "=r" (*v));
> +	WARN_ON(!ok);
> +	return ok;
>  }

Do you intend this as a generic rdrand change or also a fix for CoCo
case problem? I personally don’t like WARN_ON from security
pov, but I know I am in minority with this. 

> 
>  static inline bool __must_check rdseed_long(unsigned long *v)
> 
> (As for the RDSEED clarification, that also matches Borislav's reply, is
> what we expected and knew experimentally, and doesn't really have any
> bearing on Linux's RNG or this discussion, since RDRAND is all we need
> anyway.)

Agree. Just wanted to have it also included for the overall picture. 

> 
> Regards,
> Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Elena,

On Wed, Feb 14, 2024 at 05:59:48PM +0000, Reshetova, Elena wrote:
> > 
> > In other words, is the following a reasonable patch?
> > 
> > diff --git a/arch/x86/include/asm/archrandom.h
> > b/arch/x86/include/asm/archrandom.h
> > index 02bae8e0758b..2d5bf5aa9774 100644
> > --- a/arch/x86/include/asm/archrandom.h
> > +++ b/arch/x86/include/asm/archrandom.h
> > @@ -13,22 +13,16 @@
> >  #include <asm/processor.h>
> >  #include <asm/cpufeature.h>
> > 
> > -#define RDRAND_RETRY_LOOPS	10
> > -
> >  /* Unconditional execution of RDRAND and RDSEED */
> > 
> >  static inline bool __must_check rdrand_long(unsigned long *v)
> >  {
> >  	bool ok;
> > -	unsigned int retry = RDRAND_RETRY_LOOPS;
> > -	do {
> > -		asm volatile("rdrand %[out]"
> > -			     CC_SET(c)
> > -			     : CC_OUT(c) (ok), [out] "=r" (*v));
> > -		if (ok)
> > -			return true;
> > -	} while (--retry);
> > -	return false;
> > +	asm volatile("rdrand %[out]"
> > +		     CC_SET(c)
> > +		     : CC_OUT(c) (ok), [out] "=r" (*v));
> > +	WARN_ON(!ok);
> > +	return ok;
> >  }
> 
> Do you intend this as a generic rdrand change or also a fix for CoCo
> case problem?

I was thinking generic, since in all cases, RDRAND failing points to a
hardware bug in the CPU ITSELF (!), which is solid grounds for a WARN().

> I personally don’t like WARN_ON from security
> pov, but I know I am in minority with this. 

I share the same opinion as you, that WARN_ON() is a little weak and we
should BUG_ON() or panic() or whatever, but I also know that this ship
has really sailed long ago, that in lots of ways Linus is also right
that BUG() is bad and shouldn't be used for much, and this just isn't a
hill to die on. And the "panic_on_warn" flag exists and "security guides"
sometimes say to turn this on, etc, so I think WARN_ON() remains the
practical compromise that won't get everyone's feathers ruffelled up.


By the way, there is still one question lingering in the back of my
mind, but I don't know if answering it would divulge confidential
implementation details.

You said that RDRAND is faster than the bus, so failures won't be
observable, while RDSEED is not because it requires collecting entropy
from the ether which is slow. That makes intuitive sense on a certain
dumb simplistic level: AES is just an algorithm so is fast, while
entropy collection is a more physical thing so is slow. But if you read
the implementation details, RDRAND is supposed to reseed after 511
calls. So what's to stop you from exhausting RDSEED in one place, while
also getting RDRAND to the end of its 511 calls, and *then* having your
victim make the subsequent RDRAND call, which tries to reseed (or is in
progress of doing so), finds that RDSEED is out of batteries, and
underflows? What's the magic detail that makes this scenario not
possible?

Jason

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Reshetova, Elena]

> You said that RDRAND is faster than the bus, so failures won't be
> observable, while RDSEED is not because it requires collecting entropy
> from the ether which is slow. That makes intuitive sense on a certain
> dumb simplistic level: AES is just an algorithm so is fast, while
> entropy collection is a more physical thing so is slow. But if you read
> the implementation details, RDRAND is supposed to reseed after 511
> calls. So what's to stop you from exhausting RDSEED in one place, while
> also getting RDRAND to the end of its 511 calls, and *then* having your
> victim make the subsequent RDRAND call, which tries to reseed (or is in
> progress of doing so), finds that RDSEED is out of batteries, and
> underflows? What's the magic detail that makes this scenario not
> possible?

This was on my list of scenarios to double check whenever it is possible
or not, and the answer is that it is not possible (at least for Intel).
This scenario is also briefly described in the public doc [1]:

" Note that the conditioner does not send the same seed values to both the
 DRBG and the ENRNG. This pathway can be thought of as an alternating
switch, with one seed going to the DRGB and the next seed going to the ENRNG. 
*This construction ensures* that a software application can never obtain the
 value used to seed the DRBG, *nor can it launch a Denial of Service (DoS) 
attack against the DRBG through repeated executions of the RDSEED instruction.*"

The upcoming whitepaper hopefully should provide more details on this also.

[1] https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Thu, Feb 15, 2024 at 07:07:45AM +0000, Reshetova, Elena wrote:
> > You said that RDRAND is faster than the bus, so failures won't be
> > observable, while RDSEED is not because it requires collecting entropy
> > from the ether which is slow. That makes intuitive sense on a certain
> > dumb simplistic level: AES is just an algorithm so is fast, while
> > entropy collection is a more physical thing so is slow. But if you read
> > the implementation details, RDRAND is supposed to reseed after 511
> > calls. So what's to stop you from exhausting RDSEED in one place, while
> > also getting RDRAND to the end of its 511 calls, and *then* having your
> > victim make the subsequent RDRAND call, which tries to reseed (or is in
> > progress of doing so), finds that RDSEED is out of batteries, and
> > underflows? What's the magic detail that makes this scenario not
> > possible?
> 
> This was on my list of scenarios to double check whenever it is possible
> or not, and the answer is that it is not possible (at least for Intel).
> This scenario is also briefly described in the public doc [1]:
> 
> " Note that the conditioner does not send the same seed values to both the
>  DRBG and the ENRNG. This pathway can be thought of as an alternating
> switch, with one seed going to the DRGB and the next seed going to the ENRNG. 
> *This construction ensures* that a software application can never obtain the
>  value used to seed the DRBG, *nor can it launch a Denial of Service (DoS) 
> attack against the DRBG through repeated executions of the RDSEED instruction.*"

Interesting, and good to hear. So also implicit must be that the time
required by 511 calls to RDRAND exceeds the reseeding time, so that you
couldn't exhaust the seeds indirectly by flushing RDRAND.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Tom Lendacky]

On 2/14/24 11:21, Jason A. Donenfeld wrote:
> Hi Elena,
> 
> On Wed, Feb 14, 2024 at 4:18 PM Reshetova, Elena <elena.reshetova@intel.com> wrote:
>> "The RdRand in a non-defective device is designed to be faster than the bus,
>> so when a core accesses the output from the DRNG, it will always get a
>> random number.
>> As a result, it is hard to envision a scenario where the RdRand, on a fully
>> functional device, will underflow.
>> The carry flag after RdRand signals an underflow so in the case of a defective chip,
>> this will prevent the code thinking it has a random number when it does not.
> 
> That's really great news, especially combined with a very similar
> statement from Borislav about AMD chips:
> 
> On Fri, Feb 9, 2024 at 10:45 PM Borislav Petkov <bp@alien8.de> wrote:
>> Yeah, I know exactly what you mean and I won't go into details for
>> obvious reasons. Two things:
>>
>> * Starting with Zen3, provided properly configured hw RDRAND will never
>> fail. It is also fair when feeding the different contexts.
> 
> I assume that this faster-than-the-bus-ness also takes into account the
> various accesses required to even switch contexts when scheduling VMs,
> so your proposed host-guest scheduling attack can't really happen
> either. Correct?
> 
> One clarifying question in all of this: what is the point of the "try 10
> times" advice? Is the "faster than the bus" statement actually "faster
> than the bus if you try 10 times"? Or is the "10 times" advice just old
> and not relevant.
> 
> In other words, is the following a reasonable patch?
> 
> diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
> index 02bae8e0758b..2d5bf5aa9774 100644
> --- a/arch/x86/include/asm/archrandom.h
> +++ b/arch/x86/include/asm/archrandom.h
> @@ -13,22 +13,16 @@
>   #include <asm/processor.h>
>   #include <asm/cpufeature.h>
>   
> -#define RDRAND_RETRY_LOOPS	10
> -
>   /* Unconditional execution of RDRAND and RDSEED */
>   
>   static inline bool __must_check rdrand_long(unsigned long *v)
>   {
>   	bool ok;
> -	unsigned int retry = RDRAND_RETRY_LOOPS;
> -	do {
> -		asm volatile("rdrand %[out]"
> -			     CC_SET(c)
> -			     : CC_OUT(c) (ok), [out] "=r" (*v));
> -		if (ok)
> -			return true;
> -	} while (--retry);
> -	return false;
> +	asm volatile("rdrand %[out]"
> +		     CC_SET(c)
> +		     : CC_OUT(c) (ok), [out] "=r" (*v));
> +	WARN_ON(!ok);
> +	return ok;

Don't forget that Linux will run on older hardware as well, so the 10 
retries might be valid for that. Or do you intend this change purely for CVMs?

Thanks,
Tom

>   }
>   
>   static inline bool __must_check rdseed_long(unsigned long *v)
> 
> (As for the RDSEED clarification, that also matches Borislav's reply, is
> what we expected and knew experimentally, and doesn't really have any
> bearing on Linux's RNG or this discussion, since RDRAND is all we need
> anyway.)
> 
> Regards,
> Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

Hi Tom,

On Wed, Feb 14, 2024 at 8:46 PM Tom Lendacky <thomas.lendacky@amd.com> wrote:
> Don't forget that Linux will run on older hardware as well, so the 10
> retries might be valid for that. Or do you intend this change purely for CVMs?

Oh, grr, darnit. That is indeed a very important detail. I meant this
for generic code, so yea, if it's actually just Zen3+, then this won't
fly.

AMD people, Intel people: what are the fullest statements we can rely
on here? Do the following two statements work?

1) On newer chips, RDRAND never fails.
2) On older chips, RDRAND never fails if you try 10 times in a loop,
unless you consider host->guest attacks, which we're not, because CoCo
is only a thing on the newer chips.

If those hold true, then the course of action would be to just add a
WARN_ON(!ok) but keep the loop as-is.

(Anyway, I posted
https://lore.kernel.org/lkml/20240214195744.8332-1-Jason@zx2c4.com/
just before seeing this message.)

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Wed, Feb 14, 2024 at 09:04:34PM +0100, Jason A. Donenfeld wrote:
> AMD people, Intel people: what are the fullest statements we can rely
> on here? Do the following two statements work?
> 
> 1) On newer chips, RDRAND never fails.
> 2) On older chips, RDRAND never fails if you try 10 times in a loop,
> unless you consider host->guest attacks, which we're not, because CoCo
> is only a thing on the newer chips.
> 
> If those hold true, then the course of action would be to just add a
> WARN_ON(!ok) but keep the loop as-is.

I think we may only want to do the WARN_ON in early boot.  Otherwise,
on older chips, if a userspace process executes RDRAND is a tight
loop, it might cause the WARN_ON to trigger, which is considered
undesirable (and is certainly going to be something that could result
in a syzbot complaint).

					- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Jason A. Donenfeld]

On Wed, Feb 14, 2024 at 03:11:03PM -0500, Theodore Ts'o wrote:
> On Wed, Feb 14, 2024 at 09:04:34PM +0100, Jason A. Donenfeld wrote:
> > AMD people, Intel people: what are the fullest statements we can rely
> > on here? Do the following two statements work?
> > 
> > 1) On newer chips, RDRAND never fails.
> > 2) On older chips, RDRAND never fails if you try 10 times in a loop,
> > unless you consider host->guest attacks, which we're not, because CoCo
> > is only a thing on the newer chips.
> > 
> > If those hold true, then the course of action would be to just add a
> > WARN_ON(!ok) but keep the loop as-is.
> 
> I think we may only want to do the WARN_ON in early boot.  Otherwise,
> on older chips, if a userspace process executes RDRAND is a tight
> loop, it might cause the WARN_ON to trigger, which is considered
> undesirable (and is certainly going to be something that could result
> in a syzbot complaint).

Yea, seems reasonable. Or maybe we just don't bother adding any WARN
there and just address the CoCo thing with the patch 2/2. As it turns
out, on normal systems, the RNG is designed anyway to deal with a broken
or missing RDRAND. So maybe adding these heuristics to warn when the CPU
is broken isn't worth it? Or maybe that's an interesting thing to do?
Dunno, I'm indifferent about it I suppose. But I agree if it's added,
doing it at early boot only makes most sense.

Jason

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Dave Hansen]

On 2/14/24 09:21, Jason A. Donenfeld wrote:
> One clarifying question in all of this: what is the point of the "try 10
> times" advice? Is the "faster than the bus" statement actually "faster
> than the bus if you try 10 times"? Or is the "10 times" advice just old
> and not relevant.
> 
> In other words, is the following a reasonable patch?
> 
> diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
> index 02bae8e0758b..2d5bf5aa9774 100644
> --- a/arch/x86/include/asm/archrandom.h
> +++ b/arch/x86/include/asm/archrandom.h
> @@ -13,22 +13,16 @@
>  #include <asm/processor.h>
>  #include <asm/cpufeature.h>
>  
> -#define RDRAND_RETRY_LOOPS	10
> -
>  /* Unconditional execution of RDRAND and RDSEED */
>  
>  static inline bool __must_check rdrand_long(unsigned long *v)
>  {
>  	bool ok;
> -	unsigned int retry = RDRAND_RETRY_LOOPS;
> -	do {
> -		asm volatile("rdrand %[out]"
> -			     CC_SET(c)
> -			     : CC_OUT(c) (ok), [out] "=r" (*v));
> -		if (ok)
> -			return true;
> -	} while (--retry);
> -	return false;
> +	asm volatile("rdrand %[out]"
> +		     CC_SET(c)
> +		     : CC_OUT(c) (ok), [out] "=r" (*v));
> +	WARN_ON(!ok);
> +	return ok;
>  }

The key question here is if RDRAND can ever fail on perfectly good hardware.

I think it's theoretically possible for the entropy source health checks
to fail on perfectly good hardware for an arbitrarily long time.  But
the odds of this happening to the point of it affecting RDRAND are
rather small.

There's a reason that the guidance says: "the odds of ten failures in a
row are astronomically small" _instead_ of claiming the same about a
single RDRAND.

Given the scale that the kernel operates at, I think we should leave the
loop.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[James Bottomley]

On Thu, 2024-02-01 at 19:09 +0100, Jason A. Donenfeld wrote:
[...]
> Anyway, that's about where I'm at. I figure I'll wait to see if the
> internal inquiry within Intel yields anything interesting, and then
> maybe we can move forward with solutions (B) or (F) or (G) or a
> different Roald Dahl novel instead.

It's a lot to quote, so I cut it, but all of your solutions assume a
rdseed/rdrand failure equates to a system one but it really doesn't: in
most systems there are other entropy sources.  In confidential
computing it is an issue because we have no other trusted sources.  The
problem with picking on rdseed/rdrand is that there are bound to be
older CPUs somewhere that have rng generation bugs that this will
expose.    How about making the failure contingent on the entropy pool
not having any entropy when the first random number is requested?  That
way systems with more than one usable entropy source won't flag a bug,
but it will still flag up confidential computing systems where there's
a malicious entropy depleter.

James

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Fri, Feb 02, 2024 at 04:47:11PM +0100, James Bottomley wrote:
> 
> It's a lot to quote, so I cut it, but all of your solutions assume a
> rdseed/rdrand failure equates to a system one but it really doesn't: in
> most systems there are other entropy sources.  In confidential
> computing it is an issue because we have no other trusted sources.  The
> problem with picking on rdseed/rdrand is that there are bound to be
> older CPUs somewhere that have rng generation bugs that this will
> expose.

I'm not sure what you're concerned about.  As far as I know, all of
the CPU's have some variant of Confidential Compute have some kind of
RDRAND-like command.  And while we're using the term RDRAND, I'd
extend this to any CPU architecture-level RNG instruction which can
return failure if it is subject to exhaustion attacks.

> How about making the failure contingent on the entropy pool
> not having any entropy when the first random number is requested?

We have tried to avoid characterizing entropy sources as "valid" or
"invalid".  First of all, it's rarely quite so black-and-white.
Something which is vulnerable to someone who can spy on inter-packet
arrival times by having a hardware tap between the CPU and the network
switch, or a wireless radio right next to the device being attacked,
might not be easily carried out by someone who doesn't have local
physical access.

So we may be measuring various things that might or might not have
"entropy".  In the case of Confidential Compute, we have declared that
none of those other sources constitute "entropy".  But that's not a
decision that can be made by the computer, or at least until we've
tracked the AGI problem.  (At which point, we might have other
problems --- "I'm sorry, I'm afraid I can't do that.")

	     	  	     	      - Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[James Bottomley]

On Fri, 2024-02-02 at 11:05 -0500, Theodore Ts'o wrote:
> On Fri, Feb 02, 2024 at 04:47:11PM +0100, James Bottomley wrote:
> > 
> > It's a lot to quote, so I cut it, but all of your solutions assume
> > a rdseed/rdrand failure equates to a system one but it really
> > doesn't: in most systems there are other entropy sources.  In
> > confidential computing it is an issue because we have no other
> > trusted sources. The problem with picking on rdseed/rdrand is that
> > there are bound to be older CP s somewhere that have rng generation
> > bugs that this will
> > expose.
> 
> I'm not sure what you're concerned about.  As far as I know, all of
> the CPU's have some variant of Confidential Compute have some kind of
> RDRAND-like command.  And while we're using the term RDRAND, I'd
> extend this to any CPU architecture-level RNG instruction which can
> return failure if it is subject to exhaustion attacks.

My big concern is older cpus where rdrand/rdseed don't produce useful
entropy.  Exhaustion attacks are going to be largely against VMs not
physical systems, so I worry about physical systems with older CPUs
that might have rdrand issues which then trip our Confidential
Computing checks.


> > How about making the failure contingent on the entropy pool
> > not having any entropy when the first random number is requested?
> 
> We have tried to avoid characterizing entropy sources as "valid" or
> "invalid".  First of all, it's rarely quite so black-and-white.
> Something which is vulnerable to someone who can spy on inter-packet
> arrival times by having a hardware tap between the CPU and the
> network switch, or a wireless radio right next to the device being
> attacked, might not be easily carried out by someone who doesn't have
> local physical access.
> 
> So we may be measuring various things that might or might not have
> "entropy".  In the case of Confidential Compute, we have declared
> that none of those other sources constitute "entropy".  But that's
> not a decision that can be made by the computer, or at least until
> we've tracked the AGI problem.  (At which point, we might have other
> problems --- "I'm sorry, I'm afraid I can't do that.")

The signal for rdseed failing is fairly clear, so if the node has other
entropy sources, it should continue otherwise it should signal failure.
Figuring out how a confidential computing environment signals that
failure is TBD.

James

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Theodore Ts'o]

On Fri, Feb 02, 2024 at 10:28:01PM +0100, James Bottomley wrote:
> 
> My big concern is older cpus where rdrand/rdseed don't produce useful
> entropy.  Exhaustion attacks are going to be largely against VMs not
> physical systems, so I worry about physical systems with older CPUs
> that might have rdrand issues which then trip our Confidential
> Computing checks.

For (non-CC) VM's the answer is virtio-rng.  This solves the
exhaustion problem, since if you can't trust the host, the VM's
security is taost anyway (again, ignoring Confidential Compute).

> The signal for rdseed failing is fairly clear, so if the node has other
> entropy sources, it should continue otherwise it should signal failure.
> Figuring out how a confidential computing environment signals that
> failure is TBD.

That's a design decision, and I believe we've been converging on a
panic during early boot.  Post boot, if we've successfully succeeded
in initializing the guest kernel's RNG, we're secure so long as the
cryptographic primitives haven't been defeated --- and if we have,
such as if Quantuum Computing because practical, we've got bigger
problems anyway.

					- Ted

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[H. Peter Anvin]

On February 3, 2024 6:35:47 AM PST, Theodore Ts'o <tytso@mit.edu> wrote:
>On Fri, Feb 02, 2024 at 10:28:01PM +0100, James Bottomley wrote:
>> 
>> My big concern is older cpus where rdrand/rdseed don't produce useful
>> entropy.  Exhaustion attacks are going to be largely against VMs not
>> physical systems, so I worry about physical systems with older CPUs
>> that might have rdrand issues which then trip our Confidential
>> Computing checks.
>
>For (non-CC) VM's the answer is virtio-rng.  This solves the
>exhaustion problem, since if you can't trust the host, the VM's
>security is taost anyway (again, ignoring Confidential Compute).
>
>> The signal for rdseed failing is fairly clear, so if the node has other
>> entropy sources, it should continue otherwise it should signal failure.
>> Figuring out how a confidential computing environment signals that
>> failure is TBD.
>
>That's a design decision, and I believe we've been converging on a
>panic during early boot.  Post boot, if we've successfully succeeded
>in initializing the guest kernel's RNG, we're secure so long as the
>cryptographic primitives haven't been defeated --- and if we have,
>such as if Quantuum Computing because practical, we've got bigger
>problems anyway.
>
>					- Ted

I also want to emphasize that there is a huge difference between boot (initialization) time and runtime. Runtime harvesting has always been opportunistic in Linux, and so if RDSEED fails, try again later – unless perhaps a task is blocked on /dev/random in which case it might make sense to aggressively loop on the blocked core instead of just putting the process to sleep.

Initialization time is a different game entirely. Until we have accumulated about 256-512 bits of seed data, even the best PRNG can't really be considered "completely random." Thus a far more aggressive approach may be called for; furthermore, this is the time to look for total failure of the NRBG if after some number N attempts (where I believe N should be quite large, if we spend a full second in the very worst case that is probably better than declaring failure and optionally panic the system) we have not acquired enough entropy then warn and optionally panic the system.

By setting the limit in terms of time rather than iterations, this avoids the awkward issue of "the interface to the RDSEED unit is too fast and so it returns failure too often." I don't think anyone would argue that the right thing would be to slow down the response time of RDSEED for that reason, even though it would most likely radically reduce the failure rate (because the NRBG would have more time to produce entropy between queries at the maximum rate.)

Let's say, entirely hypothetically (as of right now I have absolutely *no* insider information of the RNG unit roadmap), that we were to implement a prefetch buffer in the core, such that a single or a handful of RD* instructions could execute in a handful of cycles, with the core itself issuing the request to the RNG unit when there is space in the queue. Such a prefetch buffer could rather obviously get *very* quickly exhausted because the poll rate could be dramatically increased, and having the core stall until there is data may or may not be a good solution (advantage: the CPU can go to a lower power state while waiting; disadvantage: opportunistic harvesting would prefer a "poll and fail fast" variation, *especially* if the CPU is going to fulfill the request autonomously anyway.)

RE: [PATCH 1/2] x86/random: Retry on RDSEED failure

[H. Peter Anvin]

On January 30, 2024 5:10:20 AM PST, "Reshetova, Elena" <elena.reshetova@intel.com> wrote:
> 
>> Hi Kirill,
>> 
>> I've been following the other discussion closely thinking about the
>> matter, but I suppose I'll jump in here directly on this patch, if
>> this is the approach the discussion is congealing around.
>> 
>> A comment below:
>> 
>> On Tue, Jan 30, 2024 at 9:30 AM Kirill A. Shutemov
>> <kirill.shutemov@linux.intel.com> wrote:
>> >  static inline bool __must_check rdseed_long(unsigned long *v)
>> >  {
>> > +       unsigned int retry = RDRAND_RETRY_LOOPS;
>> >         bool ok;
>> > -       asm volatile("rdseed %[out]"
>> > -                    CC_SET(c)
>> > -                    : CC_OUT(c) (ok), [out] "=r" (*v));
>> > -       return ok;
>> > +
>> > +       do {
>> > +               asm volatile("rdseed %[out]"
>> > +                            CC_SET(c)
>> > +                            : CC_OUT(c) (ok), [out] "=r" (*v));
>> > +
>> > +               if (ok)
>> > +                       return true;
>> > +       } while (--retry);
>> > +
>> > +       return false;
>> >  }
>> 
>> So, my understanding of RDRAND vs RDSEED -- deliberately leaving out
>> any cryptographic discussion here -- is roughly that RDRAND will
>> expand the seed material for longer, while RDSEED will mostly always
>> try to sample more bits from the environment. AES is fast, while
>> sampling is slow, so RDRAND gives better performance and is less
>> likely to fail, whereas RDSEED always has to wait on the hardware to
>> collect some bits, so is more likely to fail.
>
>The internals of Intel DRBG behind RDRAND/RDSEED has been publicly
>documented, so the structure is no secret. Please see [1] for overall
>structure and other aspects. So, yes, your overall understanding is correct
>(there are many more details though). 
>
>[1] https://www.intel.com/content/www/us/en/developer/articles/guide/intel-digital-random-number-generator-drng-software-implementation-guide.html
>
>
>> 
>> For that reason, most of the usage of RDRAND and RDSEED inside of
>> random.c is something to the tune of `if (!rdseed(out)) rdrand(out);`,
>> first trying RDSEED but falling back to RDRAND if it's busy. That
>> still seems to me like a reasonable approach, which this patch would
>> partly undermine (in concert with the next patch, which I'll comment
>> on in a follow up email there).
>
>I agree that for the purpose of extracting entropy for Linux RNG falling
>back to RDRAND (current behavior) is perfectly ok, so I think you are doing
>the right thing. However, in principle it is not always the case, there are
>situations when a fallback to RDRAND should not be used, but it is also
>true that the user of this interface should know/understand this situation. 
>
>> 
>> So maybe this patch #1 (of 2) can be dropped?
>
>Before we start debating this patchset, what is your opinion on the original
>problem we raised for CoCo VMs when both RDRAND/RDSEED are made to
>fail deliberately? 
>
>Best Regards,
>Elena.
>
>

I have a real concern with this. We already have the option to let the entropy pool fill before the boot can proceed. This would have the risk of massively increasing the interrupt latency for what will be retried anyway.

Re: [PATCH 1/2] x86/random: Retry on RDSEED failure

[Kuppuswamy Sathyanarayanan]

On 1/30/24 12:30 AM, Kirill A. Shutemov wrote:
> The function rdrand_long() retries 10 times before returning failure to
> the caller. On the other hand, rdseed_long() gives up on the first
> failure.
>
> According to the Intel SDM, both instructions should follow the same
> retry approach. This information can be found in the section titled
> "Random Number Generator Instructions".
>
> To align the behavior of rdseed_long() with rdrand_long(), it should be
> modified to retry 10 times before giving up.
>
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---

Change looks good to me.

Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>

Wondering whether this needs to go to stable trees?

>  arch/x86/include/asm/archrandom.h | 16 ++++++++++++----
>  1 file changed, 12 insertions(+), 4 deletions(-)
>
> diff --git a/arch/x86/include/asm/archrandom.h b/arch/x86/include/asm/archrandom.h
> index 02bae8e0758b..918c5880de9e 100644
> --- a/arch/x86/include/asm/archrandom.h
> +++ b/arch/x86/include/asm/archrandom.h
> @@ -33,11 +33,19 @@ static inline bool __must_check rdrand_long(unsigned long *v)
>  
>  static inline bool __must_check rdseed_long(unsigned long *v)
>  {
> +	unsigned int retry = RDRAND_RETRY_LOOPS;
>  	bool ok;
> -	asm volatile("rdseed %[out]"
> -		     CC_SET(c)
> -		     : CC_OUT(c) (ok), [out] "=r" (*v));
> -	return ok;
> +
> +	do {
> +		asm volatile("rdseed %[out]"
> +			     CC_SET(c)
> +			     : CC_OUT(c) (ok), [out] "=r" (*v));
> +
> +		if (ok)
> +			return true;
> +	} while (--retry);
> +
> +	return false;
>  }
>  
>  /*

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer