XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

For some reason migration of HVM domUs from pre-4.14 to staging is broken.

It seems xc_cpuid_apply_policy gets nr_leaves=119 from XEN_SYSCTL_get_cpu_policy.
Later XEN_SYSCTL_cpu_policy_hvm_default truncates this value to 31.
As a result x86_cpuid_copy_to_buffer returns -ENOBUFS. It would copy up to 51 leaves in my testing.

I suspect commit 34990446ca91d9e201ff17bd4e3f43bfe3ad308e just revealed a logic error somewhere.

What is supposed to happen in libxl__srm_callout_callback_static_data_done when XGR_SDD_MISSING_CPUID is set?

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Andrew Cooper]

On 01/07/2021 11:56, Olaf Hering wrote:
> For some reason migration of HVM domUs from pre-4.14 to staging is broken.
>
> It seems xc_cpuid_apply_policy gets nr_leaves=119 from XEN_SYSCTL_get_cpu_policy.
> Later XEN_SYSCTL_cpu_policy_hvm_default truncates this value to 31.

119 is the maximum number of leaves to cover sizeof(struct cpuid_policy).

But no CPU hits both Intel and AMD max leaf, so we can unambiguously
represent the policy with fewer than 119 leaves.  31 is the count of
leaves with data in, on this particular system.

> As a result x86_cpuid_copy_to_buffer returns -ENOBUFS. It would copy up to 51 leaves in my testing.
>
> I suspect commit 34990446ca91d9e201ff17bd4e3f43bfe3ad308e just revealed a logic error somewhere.
>
> What is supposed to happen in libxl__srm_callout_callback_static_data_done when XGR_SDD_MISSING_CPUID is set?

One of the many totally broken things about Xen migration was that the
guests CPUID data never used to be part of the stream.  It was always
reinvented from scratch on the destination host.

This had fun properties such as the CPU's family/model/stepping changing
underfoot (I'm still amazed that VMs didn't complain more about this -
windows would happily install a new HAL driver and call itself done),
and less fun properties where some features just disappeared totally due
to changes in xc_cpuid_apply_policy() logic.

Starting in 4.14, after I'd added a "please get the guest's view of
CPUID" hypercall, the CPUID settings the VM saw move in the migrate
stream with it.

XGR_SDD_MISSING_CPUID is the signal of "this is a pre-4.14 VM with no
CPUID data", which means "you need to reinvent CPUID as the toolstack in
4.13 would have done".

This *ought* to be "grab default, edit a few details, send back to
Xen".  Where is the 51 leaves coming in?  Do you have cpuid= lines in
the xl cfg file?

~Andrew

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

Am Thu, 1 Jul 2021 12:19:24 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> This *ought* to be "grab default, edit a few details, send back to
> Xen".  Where is the 51 leaves coming in?  Do you have cpuid= lines in
> the xl cfg file?

No cpuid= or anything in domU.cfg or Xen cmdline.
If I understand x86_cpuid_copy_to_buffer correctly, it copies all of basic+extd and viridian+xen leaves, which is apparently more than 31.

Olaf 

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Andrew Cooper]

On 01/07/2021 12:36, Olaf Hering wrote:
> Am Thu, 1 Jul 2021 12:19:24 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> This *ought* to be "grab default, edit a few details, send back to
>> Xen".  Where is the 51 leaves coming in?  Do you have cpuid= lines in
>> the xl cfg file?
> No cpuid= or anything in domU.cfg or Xen cmdline.
> If I understand x86_cpuid_copy_to_buffer correctly, it copies all of basic+extd and viridian+xen leaves, which is apparently more than 31.

It will copy the subset based on things like max_leaf, etc.

Can you grab xen-cpuid -p from this system please?

~Andrew

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 8502 bytes --]

Am Thu, 1 Jul 2021 13:35:20 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> Can you grab xen-cpuid -p from this system please?

Two PowerEdge T320 with E5-2430L.

Xen reports there are maximum 119 leaves and 2 MSRs
Raw policy: 32 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:7fbee3ff:bfebfbff
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000005:ffffffff -> 00000040:00000040:00000003:00001120
  00000006:ffffffff -> 00000077:00000002:00000001:00000000
  00000007:00000000 -> 00000000:00000281:00000000:9c000400
  0000000a:ffffffff -> 07300403:00000000:00000000:00000603
  0000000b:00000000 -> 00000001:00000002:00000100:00000000
  0000000b:00000001 -> 00000005:0000000c:00000201:00000000
  0000000d:00000000 -> 00000007:00000340:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:2c100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000007:ffffffff -> 00000000:00000000:00000000:00000100
  80000008:ffffffff -> 0000302e:00000000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000
Host policy: 31 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:77bee3ff:bfebfbff
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000007:00000000 -> 00000000:00000281:00000000:9c000400
  0000000d:00000000 -> 00000007:00000000:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:2c100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000007:ffffffff -> 00000000:00000000:00000000:00000100
  80000008:ffffffff -> 0000302e:00000000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000
PV Max policy: 31 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:f6b82203:1fc9cbf5
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000007:00000000 -> 00000000:00000201:00000000:8c000400
  0000000d:00000000 -> 00000007:00000000:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:28100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000007:ffffffff -> 00000000:00000000:00000000:00000100
  80000008:ffffffff -> 0000302e:00001000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000
HVM Max policy: 31 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:f7ba2223:1fcbfbff
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000007:00000000 -> 00000000:00000281:00000000:9c000400
  0000000d:00000000 -> 00000007:00000000:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:2c100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000007:ffffffff -> 00000000:00000000:00000000:00000100
  80000008:ffffffff -> 0000302e:00101000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000
PV Default policy: 31 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:f6b82203:1fc9cbf5
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000007:00000000 -> 00000000:00000201:00000000:8c000400
  0000000d:00000000 -> 00000007:00000000:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:28100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000008:ffffffff -> 0000302e:00001000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000
HVM Default policy: 31 leaves, 2 MSRs
 CPUID:
  leaf     subleaf  -> eax      ebx      ecx      edx     
  00000000:ffffffff -> 0000000d:756e6547:6c65746e:49656e69
  00000001:ffffffff -> 000306e4:00200800:f7ba2203:1fcbfbff
  00000002:ffffffff -> 76036301:00f0b2ff:00000000:00ca0000
  00000004:00000000 -> 3c004121:01c0003f:0000003f:00000000
  00000004:00000001 -> 3c004122:01c0003f:0000003f:00000000
  00000004:00000002 -> 3c004143:01c0003f:000001ff:00000000
  00000004:00000003 -> 3c07c163:04c0003f:00002fff:00000006
  00000007:00000000 -> 00000000:00000281:00000000:9c000400
  0000000d:00000000 -> 00000007:00000000:00000340:00000000
  0000000d:00000001 -> 00000001:00000000:00000000:00000000
  0000000d:00000002 -> 00000100:00000240:00000000:00000000
  80000000:ffffffff -> 80000008:00000000:00000000:00000000
  80000001:ffffffff -> 00000000:00000000:00000001:2c100800
  80000002:ffffffff -> 20202020:746e4920:52286c65:65582029
  80000003:ffffffff -> 52286e6f:50432029:35452055:3334322d
  80000004:ffffffff -> 76204c30:20402032:30342e32:007a4847
  80000006:ffffffff -> 00000000:00000000:01006040:00000000
  80000008:ffffffff -> 0000302e:00101000:00000000:00000000
 MSRs:
  index    -> value           
  000000ce -> 0000000080000000
  0000010a -> 0000000000000000

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

Am Thu, 1 Jul 2021 12:19:24 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

>  Where is the 51 leaves coming in?

I suspect the bug is this line, and the magic "0x1c" value:

p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu);

In my testing basic remains at 13, while extd is truncated from 0x80000008 to 28.
I think it should be truncated to CPUID_GUEST_NR_EXTD_INTEL?

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Andrew Cooper]

On 01/07/2021 16:00, Olaf Hering wrote:
> Am Thu, 1 Jul 2021 12:19:24 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>>  Where is the 51 leaves coming in?
> I suspect the bug is this line, and the magic "0x1c" value:
>
> p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu);
>
> In my testing basic remains at 13, while extd is truncated from 0x80000008 to 28.
> I think it should be truncated to CPUID_GUEST_NR_EXTD_INTEL?

That logic is definitely incorrect, but using a different constant isn't
right.  I think it has been copied from elsewhere else and incorrectly
simplified.

I'll see about trying to fix it...

~Andrew

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

Am Thu, 1 Jul 2021 16:13:21 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> That logic is definitely incorrect, but using a different constant isn't
> right.

Yeah, that occurred to me after sending this email.
I think in the end the values should probably get closer to 51, not smaller than the current one.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Andrew Cooper]

On 01/07/2021 16:18, Olaf Hering wrote:
> Am Thu, 1 Jul 2021 16:13:21 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> That logic is definitely incorrect, but using a different constant isn't
>> right.
> Yeah, that occurred to me after sending this email.
> I think in the end the values should probably get closer to 51, not smaller than the current one.

Can you give this a spin?  (Sorry - I'm busy with another task right now
- I'll do a commit message in due course)

~Andrew

diff --git a/tools/libs/guest/xg_cpuid_x86.c
b/tools/libs/guest/xg_cpuid_x86.c
index e01d657e03..e1fa33d708 100644
--- a/tools/libs/guest/xg_cpuid_x86.c
+++ b/tools/libs/guest/xg_cpuid_x86.c
@@ -513,7 +513,7 @@ int xc_cpuid_apply_policy(xc_interface *xch,
uint32_t domid, bool restore,
         /* Clamp maximum leaves to the ones supported on 4.12. */
         p->basic.max_leaf = min(p->basic.max_leaf, 0xdu);
         p->feat.max_subleaf = 0;
-        p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu);
+        p->extd.max_leaf = 0x80000000 | min(p->extd.max_leaf & 0xffff,
0x1cu);
     }
 
     if ( featureset )

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

Am Thu, 1 Jul 2021 16:25:10 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> -        p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu);
> +        p->extd.max_leaf = 0x80000000 | min(p->extd.max_leaf & 0xffff,
> 0x1cu);

Yes, this fixes it for me. Thanks.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Andrew Cooper]

On 01/07/2021 17:23, Olaf Hering wrote:
> Am Thu, 1 Jul 2021 16:25:10 +0100
> schrieb Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> -        p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu);
>> +        p->extd.max_leaf = 0x80000000 | min(p->extd.max_leaf & 0xffff,
>> 0x1cu);
> Yes, this fixes it for me. Thanks.

Fantastic.  I'll try and do a proper patch in due course.

On further thought, I think I'm going to go with the slightly simpler

p->extd.max_leaf = min(p->extd.max_leaf, 0x8000001c);

This is a policy obtained directly from Xen, which means it is running
on a 64bit capable system, which means that 0x80000008 is the lowest
number which can possibly exist in extd.max_leaf.

~Andrew

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 173 bytes --]

Am Thu, 1 Jul 2021 17:28:25 +0100
schrieb Andrew Cooper <andrew.cooper3@citrix.com>:

> 0x8000001c

Dues this need to be tagged as unsigned with a trailing U?

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]