[Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 804 bytes --]

Hi,

Multiple Qubes users have reported issues with resuming from S3 on AMD
systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
is:

(XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)

If I read it right, this is:
  - OSXSAVE: 0 -> 1
  - HYPERVISOR: 1 -> 0

Commenting out the panic on a failed recheck_cpu_features() in power.c
makes the system work after resume, reportedly stable. But that doesn't
sounds like a good idea generally.

Is this difference a Xen fault (some missing MSR / other register
restore on resume)? Or BIOS vendor / AMD, that could be worked around in
Xen?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Jan Beulich]

On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
> Hi,
> 
> Multiple Qubes users have reported issues with resuming from S3 on AMD
> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
> is:
> 
> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
> 
> If I read it right, this is:
>   - OSXSAVE: 0 -> 1
>   - HYPERVISOR: 1 -> 0
> 
> Commenting out the panic on a failed recheck_cpu_features() in power.c
> makes the system work after resume, reportedly stable. But that doesn't
> sounds like a good idea generally.
> 
> Is this difference a Xen fault (some missing MSR / other register
> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
> Xen?

The transition of the HYPERVISOR bit is definitely a Xen issue,
with Andrew having sent a patch already (iirc). The OSXSAVE part
is a little more surprising, as I wouldn't think behavior there
should differ between Intel and AMD. I take your report though
to imply that you see the issue on AMD systems only? (You also
don't mention the Xen version, which may matter as there may
have been ordering changes of when the full original CR4 value
gets restored.)

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Andrew Cooper]

On 10/02/2020 08:55, Jan Beulich wrote:
> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
>> Hi,
>>
>> Multiple Qubes users have reported issues with resuming from S3 on AMD
>> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
>> is:
>>
>> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
>>
>> If I read it right, this is:
>>   - OSXSAVE: 0 -> 1
>>   - HYPERVISOR: 1 -> 0
>>
>> Commenting out the panic on a failed recheck_cpu_features() in power.c
>> makes the system work after resume, reportedly stable. But that doesn't
>> sounds like a good idea generally.
>>
>> Is this difference a Xen fault (some missing MSR / other register
>> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
>> Xen?
> The transition of the HYPERVISOR bit is definitely a Xen issue,
> with Andrew having sent a patch already (iirc).

https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com/

Code is correct.  Commit message needs rework, including in light of
this discovery.  (I may eventually split it into two patches.)

> The OSXSAVE part is a little more surprising,

Not to me.  The checks only care if feature bits have gone missing, not
if new ones have appeared.

mmu_cr4_features includes OSXSAVE (from much later on boot than features
get cached), so the s3 path observing the gain of OSXSAVE will have been
happening ever since the checks were introduced (even on Intel.)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 1934 bytes --]

On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
> On 10/02/2020 08:55, Jan Beulich wrote:
> > On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
> >> Hi,
> >>
> >> Multiple Qubes users have reported issues with resuming from S3 on AMD
> >> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
> >> is:
> >>
> >> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
> >>
> >> If I read it right, this is:
> >>   - OSXSAVE: 0 -> 1
> >>   - HYPERVISOR: 1 -> 0
> >>
> >> Commenting out the panic on a failed recheck_cpu_features() in power.c
> >> makes the system work after resume, reportedly stable. But that doesn't
> >> sounds like a good idea generally.
> >>
> >> Is this difference a Xen fault (some missing MSR / other register
> >> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
> >> Xen?
> > The transition of the HYPERVISOR bit is definitely a Xen issue,
> > with Andrew having sent a patch already (iirc).
> 
> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com/
> 
> Code is correct.  Commit message needs rework, including in light of
> this discovery.  (I may eventually split it into two patches.)

Claudia, do you want to test with this patch?

> > The OSXSAVE part is a little more surprising,
> 
> Not to me.  The checks only care if feature bits have gone missing, not
> if new ones have appeared.
> 
> mmu_cr4_features includes OSXSAVE (from much later on boot than features
> get cached), so the s3 path observing the gain of OSXSAVE will have been
> happening ever since the checks were introduced (even on Intel.)

Is "x86: store cr4 during suspend/resume" patch from Roger related to
this?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Andrew Cooper]

On 10/02/2020 12:14, Marek Marczykowski-Górecki wrote:
> On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
>> On 10/02/2020 08:55, Jan Beulich wrote:
>>> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
>>>> Hi,
>>>>
>>>> Multiple Qubes users have reported issues with resuming from S3 on AMD
>>>> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
>>>> is:
>>>>
>>>> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
>>>>
>>>> If I read it right, this is:
>>>>   - OSXSAVE: 0 -> 1
>>>>   - HYPERVISOR: 1 -> 0
>>>>
>>>> Commenting out the panic on a failed recheck_cpu_features() in power.c
>>>> makes the system work after resume, reportedly stable. But that doesn't
>>>> sounds like a good idea generally.
>>>>
>>>> Is this difference a Xen fault (some missing MSR / other register
>>>> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
>>>> Xen?
>>> The transition of the HYPERVISOR bit is definitely a Xen issue,
>>> with Andrew having sent a patch already (iirc).
>> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com/
>>
>> Code is correct.  Commit message needs rework, including in light of
>> this discovery.  (I may eventually split it into two patches.)
> Claudia, do you want to test with this patch?
>
>>> The OSXSAVE part is a little more surprising,
>> Not to me.  The checks only care if feature bits have gone missing, not
>> if new ones have appeared.
>>
>> mmu_cr4_features includes OSXSAVE (from much later on boot than features
>> get cached), so the s3 path observing the gain of OSXSAVE will have been
>> happening ever since the checks were introduced (even on Intel.)
> Is "x86: store cr4 during suspend/resume" patch from Roger related to
> this?

No.  It wouldn't have any effect on this issue, and hasn't/won't been
taken for the reasons I described out in the email chain.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Claudia]

February 10, 2020 12:14 PM, "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com> wrote:

> On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
> 
>> On 10/02/2020 08:55, Jan Beulich wrote:
>> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
>> Hi,
>> 
>> Multiple Qubes users have reported issues with resuming from S3 on AMD
>> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
>> is:
>> 
>> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
>> 
>> If I read it right, this is:
>> - OSXSAVE: 0 -> 1
>> - HYPERVISOR: 1 -> 0
>> 
>> Commenting out the panic on a failed recheck_cpu_features() in power.c
>> makes the system work after resume, reportedly stable. But that doesn't
>> sounds like a good idea generally.
>> 
>> Is this difference a Xen fault (some missing MSR / other register
>> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
>> Xen?
>> The transition of the HYPERVISOR bit is definitely a Xen issue,
>> with Andrew having sent a patch already (iirc).
>> 
>> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com
>> 
>> Code is correct.  Commit message needs rework, including in light of
>> this discovery.  (I may eventually split it into two patches.)
> 
> Claudia, do you want to test with this patch?

I'm getting hunk failed in domctl.c applying to R4.1 default repo (fc31, Xen 4.13). I'll see if I can fix it but bear with me, I'm new at this.

Marek: Would you by any chance be willing to merge this into a test branch on your repo, so the rest of us can pull it directly into qubes-builder? It'll take you a fraction of the time it'll take me, plus then zachm and awokd and anyone else can pull it also.

Jan Beulich: Yes, based on symptoms this appears to only affect AMD from what I've seen so far, although I'm not sure if any Intel users have tried the workaround patch. In my case, I experience the exact same symptoms on 4.8, 4.12, and 4.13, and patching 4.8 resulted in successful resume and confirmed the cause of the issue, but I haven't tried patching any other versions. 

Thank you everyone for your attention to this issue so far. This is all over my head but I'm happy to provide any additional information I can. I have the Ryzen 2500U.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 2364 bytes --]

On Tue, Feb 11, 2020 at 12:59:22PM +0000, Claudia wrote:
> February 10, 2020 12:14 PM, "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com> wrote:
> 
> > On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
> > 
> >> On 10/02/2020 08:55, Jan Beulich wrote:
> >> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
> >> Hi,
> >> 
> >> Multiple Qubes users have reported issues with resuming from S3 on AMD
> >> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
> >> is:
> >> 
> >> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
> >> 
> >> If I read it right, this is:
> >> - OSXSAVE: 0 -> 1
> >> - HYPERVISOR: 1 -> 0
> >> 
> >> Commenting out the panic on a failed recheck_cpu_features() in power.c
> >> makes the system work after resume, reportedly stable. But that doesn't
> >> sounds like a good idea generally.
> >> 
> >> Is this difference a Xen fault (some missing MSR / other register
> >> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
> >> Xen?
> >> The transition of the HYPERVISOR bit is definitely a Xen issue,
> >> with Andrew having sent a patch already (iirc).
> >> 
> >> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com
> >> 
> >> Code is correct.  Commit message needs rework, including in light of
> >> this discovery.  (I may eventually split it into two patches.)
> > 
> > Claudia, do you want to test with this patch?
> 
> I'm getting hunk failed in domctl.c applying to R4.1 default repo (fc31, Xen 4.13). I'll see if I can fix it but bear with me, I'm new at this.
> 
> Marek: Would you by any chance be willing to merge this into a test branch on your repo, so the rest of us can pull it directly into qubes-builder? It'll take you a fraction of the time it'll take me, plus then zachm and awokd and anyone else can pull it also.

Here is one for Xen 4.13:
https://github.com/QubesOS/qubes-vmm-xen/pull/71
builder.conf snippet for qubes-builder:

    BRANCH_vmm_xen=xen-4.13-amd-suspend
    GIT_URL_vmm_xen=https://github.com/marmarek/qubes-vmm-xen

This is already v2 patch from the other thread.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Claudia]

February 11, 2020 9:09 PM, "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com> wrote:

> On Tue, Feb 11, 2020 at 12:59:22PM +0000, Claudia wrote:
> 
>> February 10, 2020 12:14 PM, "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com> wrote:
>> 
>> On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
>> 
>> On 10/02/2020 08:55, Jan Beulich wrote:
>> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
>> Hi,
>> 
>> Multiple Qubes users have reported issues with resuming from S3 on AMD
>> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
>> is:
>> 
>> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
>> 
>> If I read it right, this is:
>> - OSXSAVE: 0 -> 1
>> - HYPERVISOR: 1 -> 0
>> 
>> Commenting out the panic on a failed recheck_cpu_features() in power.c
>> makes the system work after resume, reportedly stable. But that doesn't
>> sounds like a good idea generally.
>> 
>> Is this difference a Xen fault (some missing MSR / other register
>> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
>> Xen?
>> The transition of the HYPERVISOR bit is definitely a Xen issue,
>> with Andrew having sent a patch already (iirc).
>> 
>> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com
>> 
>> Code is correct. Commit message needs rework, including in light of
>> this discovery. (I may eventually split it into two patches.)
>> 
>> Claudia, do you want to test with this patch?
>> 
>> I'm getting hunk failed in domctl.c applying to R4.1 default repo (fc31, Xen 4.13). I'll see if I
>> can fix it but bear with me, I'm new at this.
>> 
>> Marek: Would you by any chance be willing to merge this into a test branch on your repo, so the
>> rest of us can pull it directly into qubes-builder? It'll take you a fraction of the time it'll
>> take me, plus then zachm and awokd and anyone else can pull it also.
> 
> Here is one for Xen 4.13:
> https://github.com/QubesOS/qubes-vmm-xen/pull/71
> builder.conf snippet for qubes-builder:
> 
> BRANCH_vmm_xen=xen-4.13-amd-suspend
> GIT_URL_vmm_xen=https://github.com/marmarek/qubes-vmm-xen
> 
> This is already v2 patch from the other thread.

Thanks! For anyone else trying this, I also had to add NO_CHECK=vmm-xen vmm-xen-stubdom-legacy, I guess because there are no tags on that branch. The RPMs built successfully, and I'll be able to test them as soon as I get the latest R4.1 build downloaded and installed (I'm currently running 4.0).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen fails to resume on AMD Fam15h (and Fam17h?) because of CPUID mismatch

[Z M]

[-- Attachment #1: Type: text/plain, Size: 3184 bytes --]

Sorry to bump this ancient issue.. Claudia, did you try the RPMs out, or
did anyone else? I've had xen* and python*-xen excluded from yum/dnf for a
while on my R4.0 so I'm not getting patches on the Xen 4.8 branch. I guess
we need to wait til R4.1 to use it in production as it's built for Xen 4.13
though. Unless there's some way to put Xen 4.13 on R4.0 safely.



On Tue, Feb 11, 2020 at 10:20 PM Claudia <claudia1@disroot.org> wrote:

> February 11, 2020 9:09 PM, "Marek Marczykowski-Górecki" <
> marmarek@invisiblethingslab.com> wrote:
>
> > On Tue, Feb 11, 2020 at 12:59:22PM +0000, Claudia wrote:
> >
> >> February 10, 2020 12:14 PM, "Marek Marczykowski-Górecki" <
> marmarek@invisiblethingslab.com> wrote:
> >>
> >> On Mon, Feb 10, 2020 at 11:17:34AM +0000, Andrew Cooper wrote:
> >>
> >> On 10/02/2020 08:55, Jan Beulich wrote:
> >> On 10.02.2020 00:06, Marek Marczykowski-Górecki wrote:
> >> Hi,
> >>
> >> Multiple Qubes users have reported issues with resuming from S3 on AMD
> >> systems (Ryzen 2500U, Ryzen Pro 3700U, maybe more). The error message
> >> is:
> >>
> >> (XEN) CPU0: cap[ 1] is 7ed8320b (expected f6d8320b)
> >>
> >> If I read it right, this is:
> >> - OSXSAVE: 0 -> 1
> >> - HYPERVISOR: 1 -> 0
> >>
> >> Commenting out the panic on a failed recheck_cpu_features() in power.c
> >> makes the system work after resume, reportedly stable. But that doesn't
> >> sounds like a good idea generally.
> >>
> >> Is this difference a Xen fault (some missing MSR / other register
> >> restore on resume)? Or BIOS vendor / AMD, that could be worked around in
> >> Xen?
> >> The transition of the HYPERVISOR bit is definitely a Xen issue,
> >> with Andrew having sent a patch already (iirc).
> >>
> >>
> https://lore.kernel.org/xen-devel/20200127202121.2961-1-andrew.cooper3@citrix.com
> >>
> >> Code is correct. Commit message needs rework, including in light of
> >> this discovery. (I may eventually split it into two patches.)
> >>
> >> Claudia, do you want to test with this patch?
> >>
> >> I'm getting hunk failed in domctl.c applying to R4.1 default repo
> (fc31, Xen 4.13). I'll see if I
> >> can fix it but bear with me, I'm new at this.
> >>
> >> Marek: Would you by any chance be willing to merge this into a test
> branch on your repo, so the
> >> rest of us can pull it directly into qubes-builder? It'll take you a
> fraction of the time it'll
> >> take me, plus then zachm and awokd and anyone else can pull it also.
> >
> > Here is one for Xen 4.13:
> > https://github.com/QubesOS/qubes-vmm-xen/pull/71
> > builder.conf snippet for qubes-builder:
> >
> > BRANCH_vmm_xen=xen-4.13-amd-suspend
> > GIT_URL_vmm_xen=https://github.com/marmarek/qubes-vmm-xen
> >
> > This is already v2 patch from the other thread.
>
> Thanks! For anyone else trying this, I also had to add NO_CHECK=vmm-xen
> vmm-xen-stubdom-legacy, I guess because there are no tags on that branch.
> The RPMs built successfully, and I'll be able to test them as soon as I get
> the latest R4.1 build downloaded and installed (I'm currently running 4.0).
>


-- 
- - - - - -
Zachary Muller
http://muller.is/

[-- Attachment #2: Type: text/html, Size: 4550 bytes --]