* [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
@ 2022-05-24 15:10 Igor Mammedov
2022-05-24 15:10 ` [PATCH 1/2] x86: cpu: make sure number of addressable IDs for processor cores meets the spec Igor Mammedov
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Igor Mammedov @ 2022-05-24 15:10 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, f4bug, yang.zhong
Igor Mammedov (2):
x86: cpu: make sure number of addressable IDs for processor cores
meets the spec
x86: cpu: fixup number of addressable IDs for logical processors
sharing cache
target/i386/cpu.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/2] x86: cpu: make sure number of addressable IDs for processor cores meets the spec
2022-05-24 15:10 [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
@ 2022-05-24 15:10 ` Igor Mammedov
2022-05-24 15:10 ` [PATCH 2/2] x86: cpu: fixup number of addressable IDs for logical processors sharing cache Igor Mammedov
` (2 subsequent siblings)
3 siblings, 0 replies; 12+ messages in thread
From: Igor Mammedov @ 2022-05-24 15:10 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, f4bug, yang.zhong
Accourding Intel's CPUID[EAX=04H] resulting bits 31 - 26 in EAX
should be:
"
**** The nearest power-of-2 integer that is not smaller than (1 + EAX[31:26]) is the number of unique
Core_IDs reserved for addressing different processor cores in a physical package. Core ID is a subset of
bits of the initial APIC ID.
"
ensure that values stored in EAX[31-26] always meets this condition.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
---
target/i386/cpu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 35c3475e6c..bbe37dce2e 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5279,7 +5279,7 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
/* QEMU gives out its own APIC IDs, never pass down bits 31..26. */
*eax &= ~0xFC000000;
if ((*eax & 31) && cs->nr_cores > 1) {
- *eax |= (cs->nr_cores - 1) << 26;
+ *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
}
} else if (cpu->vendor_cpuid_only && IS_AMD_CPU(env)) {
*eax = *ebx = *ecx = *edx = 0;
--
2.31.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/2] x86: cpu: fixup number of addressable IDs for logical processors sharing cache
2022-05-24 15:10 [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
2022-05-24 15:10 ` [PATCH 1/2] x86: cpu: make sure number of addressable IDs for processor cores meets the spec Igor Mammedov
@ 2022-05-24 15:10 ` Igor Mammedov
2022-05-24 15:19 ` [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
2022-05-31 12:44 ` Igor Mammedov
3 siblings, 0 replies; 12+ messages in thread
From: Igor Mammedov @ 2022-05-24 15:10 UTC (permalink / raw)
To: qemu-devel; +Cc: pbonzini, richard.henderson, f4bug, yang.zhong
When QEMU is started with '-cpu host,host-cache-info=on', it will
passthrough host's number of logical processors sharing cache and
number of processor cores in the physical package. QEMU already
fixes up the later to correctly reflect number of configured cores
for VM, however number of logical processors sharing cache is still
comes from host CPU, which confuses guest started with:
-machine q35,accel=kvm \
-cpu host,host-cache-info=on,l3-cache=off \
-smp 20,sockets=2,dies=1,cores=10,threads=1 \
-numa node,nodeid=0,memdev=ram-node0 \
-numa node,nodeid=1,memdev=ram-node1 \
-numa cpu,socket-id=0,node-id=0 \
-numa cpu,socket-id=1,node-id=1
on 2 socket Xeon 4210R host with 10 cores per socket
with CPUID[04H]:
...
--- cache 3 ---
cache type = unified cache (3)
cache level = 0x3 (3)
self-initializing cache level = true
fully associative cache = false
maximum IDs for CPUs sharing cache = 0x1f (31)
maximum IDs for cores in pkg = 0xf (15)
...
that doesn't match number of logical processors VM was
configured with and as result RHEL 9.0 guest complains:
sched: CPU #10's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
WARNING: CPU: 10 PID: 0 at arch/x86/kernel/smpboot.c:421 topology_sane.isra.0+0x67/0x80
...
Call Trace:
set_cpu_sibling_map+0x176/0x590
start_secondary+0x5b/0x150
secondary_startup_64_no_verify+0xc2/0xcb
Fix it by capping max number of logical processors to vcpus/socket
as it was configured, which fixes the issue.
Signed-off-by: Igor Mammedov <imammedo@redhat.com>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2088311
---
PS:
capping to logical cpus/socket was arbitrarily chosen (maybe
it should be per die or something else but don't see that in spec)
---
target/i386/cpu.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index bbe37dce2e..ffb274dcf6 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -5276,10 +5276,22 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
/* cache info: needed for Core compatibility */
if (cpu->cache_info_passthrough) {
x86_cpu_get_cache_cpuid(index, count, eax, ebx, ecx, edx);
- /* QEMU gives out its own APIC IDs, never pass down bits 31..26. */
- *eax &= ~0xFC000000;
- if ((*eax & 31) && cs->nr_cores > 1) {
- *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
+ /*
+ * QEMU has its own number of cores/logical cpus,
+ * set 24..14, 31..26 bit to configured values
+ */
+ if (*eax & 31) {
+ int host_vcpus_per_cache = 1 + ((*eax & 0x3FFC000) >> 14);
+ int vcpus_per_socket = env->nr_dies * cs->nr_cores *
+ cs->nr_threads;
+ if (cs->nr_cores > 1) {
+ *eax &= ~0xFC000000;
+ *eax |= (pow2ceil(cs->nr_cores) - 1) << 26;
+ }
+ if (host_vcpus_per_cache > vcpus_per_socket) {
+ *eax &= ~0x3FFC000;
+ *eax |= (pow2ceil(vcpus_per_socket) - 1) << 14;
+ }
}
} else if (cpu->vendor_cpuid_only && IS_AMD_CPU(env)) {
*eax = *ebx = *ecx = *edx = 0;
--
2.31.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 15:10 [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
2022-05-24 15:10 ` [PATCH 1/2] x86: cpu: make sure number of addressable IDs for processor cores meets the spec Igor Mammedov
2022-05-24 15:10 ` [PATCH 2/2] x86: cpu: fixup number of addressable IDs for logical processors sharing cache Igor Mammedov
@ 2022-05-24 15:19 ` Igor Mammedov
2022-05-24 19:48 ` Moger, Babu
2022-05-31 12:44 ` Igor Mammedov
3 siblings, 1 reply; 12+ messages in thread
From: Igor Mammedov @ 2022-05-24 15:19 UTC (permalink / raw)
To: qemu-devel; +Cc: babu.moger
On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <imammedo@redhat.com> wrote:
CCing AMD folks as that might be of interest to them
> Igor Mammedov (2):
> x86: cpu: make sure number of addressable IDs for processor cores
> meets the spec
> x86: cpu: fixup number of addressable IDs for logical processors
> sharing cache
>
> target/i386/cpu.c | 20 ++++++++++++++++----
> 1 file changed, 16 insertions(+), 4 deletions(-)
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 15:19 ` [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
@ 2022-05-24 19:48 ` Moger, Babu
2022-05-24 23:23 ` Alejandro Jimenez
2022-05-25 7:05 ` Igor Mammedov
0 siblings, 2 replies; 12+ messages in thread
From: Moger, Babu @ 2022-05-24 19:48 UTC (permalink / raw)
To: Igor Mammedov, qemu-devel
On 5/24/22 10:19, Igor Mammedov wrote:
> On Tue, 24 May 2022 11:10:18 -0400
> Igor Mammedov <imammedo@redhat.com> wrote:
>
> CCing AMD folks as that might be of interest to them
I am trying to recreate the bug on my AMD system here.. Seeing this message..
qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
is ambiguous
Here is my command line..
#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
nic -nographic -machine q35,accel=kvm -cpu
host,host-cache-info=on,l3-cache=off -smp
20,sockets=2,dies=1,cores=10,threads=1 -numa
node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
Am I missing something?
>
>> Igor Mammedov (2):
>> x86: cpu: make sure number of addressable IDs for processor cores
>> meets the spec
>> x86: cpu: fixup number of addressable IDs for logical processors
>> sharing cache
>>
>> target/i386/cpu.c | 20 ++++++++++++++++----
>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>
--
Thanks
Babu Moger
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 19:48 ` Moger, Babu
@ 2022-05-24 23:23 ` Alejandro Jimenez
2022-05-25 19:56 ` Moger, Babu
2022-05-25 7:05 ` Igor Mammedov
1 sibling, 1 reply; 12+ messages in thread
From: Alejandro Jimenez @ 2022-05-24 23:23 UTC (permalink / raw)
To: babu.moger; +Cc: Joao Martins, Igor Mammedov, QEMU Developers
On 5/24/2022 3:48 PM, Moger, Babu wrote:
>
> On 5/24/22 10:19, Igor Mammedov wrote:
>> On Tue, 24 May 2022 11:10:18 -0400
>> Igor Mammedov <imammedo@redhat.com> wrote:
>>
>> CCing AMD folks as that might be of interest to them
>
> I am trying to recreate the bug on my AMD system here.. Seeing this message..
>
> qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
> is ambiguous
>
> Here is my command line..
>
> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
> nic -nographic -machine q35,accel=kvm -cpu
> host,host-cache-info=on,l3-cache=off -smp
> 20,sockets=2,dies=1,cores=10,threads=1 -numa
> node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
> cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
>
> Am I missing something?
Hi Babu,
Hopefully this will help you reproduce the issue if you are testing on
Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year,
while I was working on patches for encoding the topology CPUID leaf in
different Zen platforms.
What I found from my experiments on Milan, is that the warning will
appear whenever the NUMA topology requested in QEMU cmdline assigns a
number of CPUs to each node that is smaller than the default # of CPUs
sharing a LLC on the host platform. In short, on a Milan host where we
have 16 CPUs sharing a CCX:
# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7,128-135
If a guest is launched with the following arguments:
-cpu host,+topoext \
-smp cpus=64,cores=32,threads=2,sockets=1 \
-numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
-numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
-numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
-numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \
it assigns 8 cpus to each NUMA node, causing the error above to be
displayed.
Note that ultimately the guest topology is built based on the NUMA
information, so the LLC domains on the guest only end up spanning a
single NUMA node. e.g.:
# cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
0-7
Hope that helps,
Alejandro
>
>
>>
>>> Igor Mammedov (2):
>>> x86: cpu: make sure number of addressable IDs for processor cores
>>> meets the spec
>>> x86: cpu: fixup number of addressable IDs for logical processors
>>> sharing cache
>>>
>>> target/i386/cpu.c | 20 ++++++++++++++++----
>>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 19:48 ` Moger, Babu
2022-05-24 23:23 ` Alejandro Jimenez
@ 2022-05-25 7:05 ` Igor Mammedov
2022-05-25 20:04 ` Moger, Babu
1 sibling, 1 reply; 12+ messages in thread
From: Igor Mammedov @ 2022-05-25 7:05 UTC (permalink / raw)
To: Moger, Babu; +Cc: qemu-devel
On Tue, 24 May 2022 14:48:29 -0500
"Moger, Babu" <babu.moger@amd.com> wrote:
> On 5/24/22 10:19, Igor Mammedov wrote:
> > On Tue, 24 May 2022 11:10:18 -0400
> > Igor Mammedov <imammedo@redhat.com> wrote:
> >
> > CCing AMD folks as that might be of interest to them
>
> I am trying to recreate the bug on my AMD system here.. Seeing this message..
>
> qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
> is ambiguous
>
> Here is my command line..
>
> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
> nic -nographic -machine q35,accel=kvm -cpu
> host,host-cache-info=on,l3-cache=off -smp
> 20,sockets=2,dies=1,cores=10,threads=1 -numa
> node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
> cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
>
> Am I missing something?
Yep, sorry I've omitted -object memory-backend-foo definitions for
ram-node0 and ram-node1
one can use any memory backend, it doesn't really matter in this case,
for example following should do:
-object memory-backend-ram,id=ram-node0,size=2G \
-object memory-backend-ram,id=ram-node1,size=2G
>
>
> >
> >> Igor Mammedov (2):
> >> x86: cpu: make sure number of addressable IDs for processor cores
> >> meets the spec
> >> x86: cpu: fixup number of addressable IDs for logical processors
> >> sharing cache
> >>
> >> target/i386/cpu.c | 20 ++++++++++++++++----
> >> 1 file changed, 16 insertions(+), 4 deletions(-)
> >>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 23:23 ` Alejandro Jimenez
@ 2022-05-25 19:56 ` Moger, Babu
2022-05-25 21:20 ` Alejandro Jimenez
0 siblings, 1 reply; 12+ messages in thread
From: Moger, Babu @ 2022-05-25 19:56 UTC (permalink / raw)
To: Alejandro Jimenez; +Cc: Joao Martins, Igor Mammedov, QEMU Developers
On 5/24/22 18:23, Alejandro Jimenez wrote:
> On 5/24/2022 3:48 PM, Moger, Babu wrote:
>>
>> On 5/24/22 10:19, Igor Mammedov wrote:
>>> On Tue, 24 May 2022 11:10:18 -0400
>>> Igor Mammedov <imammedo@redhat.com> wrote:
>>>
>>> CCing AMD folks as that might be of interest to them
>>
>> I am trying to recreate the bug on my AMD system here.. Seeing this
>> message..
>>
>> qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
>> is ambiguous
>>
>> Here is my command line..
>>
>> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
>> nic -nographic -machine q35,accel=kvm -cpu
>> host,host-cache-info=on,l3-cache=off -smp
>> 20,sockets=2,dies=1,cores=10,threads=1 -numa
>> node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
>> cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
>>
>> Am I missing something?
> Hi Babu,
>
> Hopefully this will help you reproduce the issue if you are testing on
> Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year,
> while I was working on patches for encoding the topology CPUID leaf in
> different Zen platforms.
>
> What I found from my experiments on Milan, is that the warning will
> appear whenever the NUMA topology requested in QEMU cmdline assigns a
> number of CPUs to each node that is smaller than the default # of CPUs
> sharing a LLC on the host platform. In short, on a Milan host where we
> have 16 CPUs sharing a CCX:
Yes. I recreated the issue with this following command line.
#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
nic -nographic -machine q35,accel=kvm -cpu host,+topoext -smp
16,sockets=1,dies=1,cores=16,threads=1 -object
memory-backend-ram,id=ram-node0,size=2G -object
memory-backend-ram,id=ram-node1,size=2G -numa
node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa
node,nodeid=1,cpus=8-15,memdev=ram-node1
But solving this will be bit complicated. For AMD, this information comes
from CPUID 0x8000001d. But, when this cpuid is being populated we don't
have all the information about numa nodes etc..
But you can work-around it by modifying the command line by including
dies(dies=2 in this case) information. Something like this.
#qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
nic -nographic -machine q35,accel=kvm -cpu
host,+topoext,host-cache-info=on -smp
16,sockets=1,dies=2,cores=8,threads=1 -object
memory-backend-ram,id=ram-node0,size=2G -object
memory-backend-ram,id=ram-node1,size=2G -numa
node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa
node,nodeid=1,cpus=8-15,memdev=ram-node1
But this may not be acceptable solution in all the cases.
>
> # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
> 0-7,128-135
>
> If a guest is launched with the following arguments:
>
> -cpu host,+topoext \
> -smp cpus=64,cores=32,threads=2,sockets=1 \
> -numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
> -numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
> -numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
> -numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \
>
> it assigns 8 cpus to each NUMA node, causing the error above to be
> displayed.
>
> Note that ultimately the guest topology is built based on the NUMA
> information, so the LLC domains on the guest only end up spanning a
> single NUMA node. e.g.:
>
> # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
> 0-7
>
> Hope that helps,
> Alejandro
>>
>>
>>>
>>>> Igor Mammedov (2):
>>>> x86: cpu: make sure number of addressable IDs for processor cores
>>>> meets the spec
>>>> x86: cpu: fixup number of addressable IDs for logical processors
>>>> sharing cache
>>>>
>>>> target/i386/cpu.c | 20 ++++++++++++++++----
>>>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>>>
>
--
Thanks
Babu Moger
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-25 7:05 ` Igor Mammedov
@ 2022-05-25 20:04 ` Moger, Babu
0 siblings, 0 replies; 12+ messages in thread
From: Moger, Babu @ 2022-05-25 20:04 UTC (permalink / raw)
To: Igor Mammedov; +Cc: qemu-devel
On 5/25/22 02:05, Igor Mammedov wrote:
> On Tue, 24 May 2022 14:48:29 -0500
> "Moger, Babu" <babu.moger@amd.com> wrote:
>
>> On 5/24/22 10:19, Igor Mammedov wrote:
>>> On Tue, 24 May 2022 11:10:18 -0400
>>> Igor Mammedov <imammedo@redhat.com> wrote:
>>>
>>> CCing AMD folks as that might be of interest to them
>> I am trying to recreate the bug on my AMD system here.. Seeing this message..
>>
>> qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
>> is ambiguous
>>
>> Here is my command line..
>>
>> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
>> nic -nographic -machine q35,accel=kvm -cpu
>> host,host-cache-info=on,l3-cache=off -smp
>> 20,sockets=2,dies=1,cores=10,threads=1 -numa
>> node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
>> cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
>>
>> Am I missing something?
> Yep, sorry I've omitted -object memory-backend-foo definitions for
> ram-node0 and ram-node1
>
> one can use any memory backend, it doesn't really matter in this case,
> for example following should do:
> -object memory-backend-ram,id=ram-node0,size=2G \
> -object memory-backend-ram,id=ram-node1,size=2G
Thanks Igor. However these changes(patch 1 and 2) does not affect AMD
systems as far i can see.
Thanks
Babu
>
>>
>>>
>>>> Igor Mammedov (2):
>>>> x86: cpu: make sure number of addressable IDs for processor cores
>>>> meets the spec
>>>> x86: cpu: fixup number of addressable IDs for logical processors
>>>> sharing cache
>>>>
>>>> target/i386/cpu.c | 20 ++++++++++++++++----
>>>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>>>
--
Thanks
Babu Moger
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-25 19:56 ` Moger, Babu
@ 2022-05-25 21:20 ` Alejandro Jimenez
0 siblings, 0 replies; 12+ messages in thread
From: Alejandro Jimenez @ 2022-05-25 21:20 UTC (permalink / raw)
To: babu.moger; +Cc: Joao Martins, Igor Mammedov, QEMU Developers
On 5/25/2022 3:56 PM, Moger, Babu wrote:
>
> On 5/24/22 18:23, Alejandro Jimenez wrote:
>> On 5/24/2022 3:48 PM, Moger, Babu wrote:
>>>
>>> On 5/24/22 10:19, Igor Mammedov wrote:
>>>> On Tue, 24 May 2022 11:10:18 -0400
>>>> Igor Mammedov <imammedo@redhat.com> wrote:
>>>>
>>>> CCing AMD folks as that might be of interest to them
>>>
>>> I am trying to recreate the bug on my AMD system here.. Seeing this
>>> message..
>>>
>>> qemu-system-x86_64: -numa node,nodeid=0,memdev=ram-node0: memdev=ram-node0
>>> is ambiguous
>>>
>>> Here is my command line..
>>>
>>> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
>>> nic -nographic -machine q35,accel=kvm -cpu
>>> host,host-cache-info=on,l3-cache=off -smp
>>> 20,sockets=2,dies=1,cores=10,threads=1 -numa
>>> node,nodeid=0,memdev=ram-node0 -numa node,nodeid=1,memdev=ram-node1 -numa
>>> cpu,socket-id=0,node-id=0 -numa cpu,socket-id=1,node-id=1
>>>
>>> Am I missing something?
>> Hi Babu,
>>
>> Hopefully this will help you reproduce the issue if you are testing on
>> Milan/Genoa. Joao (CC'd) pointed out this warning to me late last year,
>> while I was working on patches for encoding the topology CPUID leaf in
>> different Zen platforms.
>>
>> What I found from my experiments on Milan, is that the warning will
>> appear whenever the NUMA topology requested in QEMU cmdline assigns a
>> number of CPUs to each node that is smaller than the default # of CPUs
>> sharing a LLC on the host platform. In short, on a Milan host where we
>> have 16 CPUs sharing a CCX:
>
> Yes. I recreated the issue with this following command line.
>
> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
> nic -nographic -machine q35,accel=kvm -cpu host,+topoext -smp
> 16,sockets=1,dies=1,cores=16,threads=1 -object
> memory-backend-ram,id=ram-node0,size=2G -object
> memory-backend-ram,id=ram-node1,size=2G -numa
> node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa
> node,nodeid=1,cpus=8-15,memdev=ram-node1
>
> But solving this will be bit complicated. For AMD, this information comes
> from CPUID 0x8000001d. But, when this cpuid is being populated we don't
> have all the information about numa nodes etc..
>
> But you can work-around it by modifying the command line by including
> dies(dies=2 in this case) information. Something like this.
Makes sense; using dies=2 makes it so the cache topology leaf is built
with 8cores/CCX, matching the # of NUMA nodes so all is well.
>
> #qemu-system-x86_64 -name rhel8 -m 4096 -hda vdisk.qcow2 -enable-kvm -net
> nic -nographic -machine q35,accel=kvm -cpu
> host,+topoext,host-cache-info=on -smp
> 16,sockets=1,dies=2,cores=8,threads=1 -object
> memory-backend-ram,id=ram-node0,size=2G -object
> memory-backend-ram,id=ram-node1,size=2G -numa
> node,nodeid=0,cpus=0-7,memdev=ram-node0 -numa
> node,nodeid=1,cpus=8-15,memdev=ram-node1
>
> But this may not be acceptable solution in all the cases.
This is not specific to host-cache-info behavior so it is probably
better to discuss it separately. With that being said...
The idea that I considered was to automatically calculate a value of
'dies' iff a explicit value was not requested via the '-smp' options,
instead of just using the current default of dies=1. i.e. automatically
mimic the host cache topology in the guest so that if we are running on
Rome, the guest OS sees 4cores/CCX, but when running on Milan it sees
8cores/CCX. This can be done by querying the host CPUID and using that
info to build the guest CPUID leaf in QEMU, similar to what Igor is
doing here but also adjusting the number of dies that is encoded.
I built prototype code that seemed to work correctly, but did not
consider the complication added by '-numa' options.
I think there is a much larger debate involved about what defaults are
"sane", so rather than derailing this thread more, I'll send a follow up
message in the future when I can take another look at the prototype
patches I have.
Thank you,
Alejandro
>
>>
>> # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
>> 0-7,128-135
>>
>> If a guest is launched with the following arguments:
>>
>> -cpu host,+topoext \
>> -smp cpus=64,cores=32,threads=2,sockets=1 \
>> -numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15 \
>> -numa node,nodeid=2,cpus=16-23 -numa node,nodeid=3,cpus=24-31 \
>> -numa node,nodeid=4,cpus=32-39 -numa node,nodeid=5,cpus=40-47 \
>> -numa node,nodeid=6,cpus=48-55 -numa node,nodeid=7,cpus=56-63 \
>>
>> it assigns 8 cpus to each NUMA node, causing the error above to be
>> displayed.
>>
>> Note that ultimately the guest topology is built based on the NUMA
>> information, so the LLC domains on the guest only end up spanning a
>> single NUMA node. e.g.:
>>
>> # cat /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list
>> 0-7
>>
>> Hope that helps,
>> Alejandro
>>>
>>>
>>>>
>>>>> Igor Mammedov (2):
>>>>> x86: cpu: make sure number of addressable IDs for processor cores
>>>>> meets the spec
>>>>> x86: cpu: fixup number of addressable IDs for logical processors
>>>>> sharing cache
>>>>>
>>>>> target/i386/cpu.c | 20 ++++++++++++++++----
>>>>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>>>>
>>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-24 15:10 [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
` (2 preceding siblings ...)
2022-05-24 15:19 ` [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
@ 2022-05-31 12:44 ` Igor Mammedov
2022-06-01 10:02 ` Paolo Bonzini
3 siblings, 1 reply; 12+ messages in thread
From: Igor Mammedov @ 2022-05-31 12:44 UTC (permalink / raw)
To: pbonzini; +Cc: qemu-devel
Paolo,
can you pick this up if it looks fine, please?
On Tue, 24 May 2022 11:10:18 -0400
Igor Mammedov <imammedo@redhat.com> wrote:
> Igor Mammedov (2):
> x86: cpu: make sure number of addressable IDs for processor cores
> meets the spec
> x86: cpu: fixup number of addressable IDs for logical processors
> sharing cache
>
> target/i386/cpu.c | 20 ++++++++++++++++----
> 1 file changed, 16 insertions(+), 4 deletions(-)
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on
2022-05-31 12:44 ` Igor Mammedov
@ 2022-06-01 10:02 ` Paolo Bonzini
0 siblings, 0 replies; 12+ messages in thread
From: Paolo Bonzini @ 2022-06-01 10:02 UTC (permalink / raw)
To: Igor Mammedov; +Cc: qemu-devel
On 5/31/22 14:44, Igor Mammedov wrote:
> Paolo,
> can you pick this up if it looks fine, please?
>
> On Tue, 24 May 2022 11:10:18 -0400
> Igor Mammedov <imammedo@redhat.com> wrote:
>
>> Igor Mammedov (2):
>> x86: cpu: make sure number of addressable IDs for processor cores
>> meets the spec
>> x86: cpu: fixup number of addressable IDs for logical processors
>> sharing cache
>>
>> target/i386/cpu.c | 20 ++++++++++++++++----
>> 1 file changed, 16 insertions(+), 4 deletions(-)
>>
>
Yup, queued now. Thanks,
Paolo
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-06-01 10:06 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-24 15:10 [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
2022-05-24 15:10 ` [PATCH 1/2] x86: cpu: make sure number of addressable IDs for processor cores meets the spec Igor Mammedov
2022-05-24 15:10 ` [PATCH 2/2] x86: cpu: fixup number of addressable IDs for logical processors sharing cache Igor Mammedov
2022-05-24 15:19 ` [PATCH 0/2] i386: fixup number of logical CPUs when host-cache-info=on Igor Mammedov
2022-05-24 19:48 ` Moger, Babu
2022-05-24 23:23 ` Alejandro Jimenez
2022-05-25 19:56 ` Moger, Babu
2022-05-25 21:20 ` Alejandro Jimenez
2022-05-25 7:05 ` Igor Mammedov
2022-05-25 20:04 ` Moger, Babu
2022-05-31 12:44 ` Igor Mammedov
2022-06-01 10:02 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).