[Bug 1856335] [NEW] Cache Layout wrong on many Zen Arch CPUs

[Bug 1856335] [NEW] Cache Layout wrong on many Zen Arch CPUs

[Damir]

Public bug reported:

AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems to
always map Cache ass if it was an 4-Core per CCX CPU, which is
incorrect, and costs upwards 30% performance (more realistically 10%) in
L3 Cache Layout aware applications.

Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='8' threads='1'/>

In windows, coreinfo reports correctly:

****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

On a 3-CCX CPU (3960X /w 6 cores and no SMT):

 <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='6' threads='1'/>

in windows, coreinfo reports incorrectly:

****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64


Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm. 

With newer Qemu there is a fix (that does behave correctly) in using the dies parameter: 
 <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

The problem is that the dies are exposed differently than how AMD does
it natively, they are exposed to Windows as sockets, which means, you
can't ever have a machine with more than two CCX (6 cores) as Windows
only supports two sockets. (Should this be reported as a separate bug?)

** Affects: qemu
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  
  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm. 

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter: 
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, you
  can't ever have a machine with more than two CCX (6 cores) as Windows
  only supports two sockets. (Should this be reported as a separate
  bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Damir]

** Description changed:

  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems to
  always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%) in
  L3 Cache Layout aware applications.
  
  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):
  
-   <cpu mode='custom' match='exact' check='full'>
-     <model fallback='forbid'>EPYC-IBPB</model>
-     <vendor>AMD</vendor>
-     <topology sockets='1' cores='8' threads='1'/>
+   <cpu mode='custom' match='exact' check='full'>
+     <model fallback='forbid'>EPYC-IBPB</model>
+     <vendor>AMD</vendor>
+     <topology sockets='1' cores='8' threads='1'/>
  
  In windows, coreinfo reports correctly:
  
  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64
  
  On a 3-CCX CPU (3960X /w 6 cores and no SMT):
  
-  <cpu mode='custom' match='exact' check='full'>
-     <model fallback='forbid'>EPYC-IBPB</model>
-     <vendor>AMD</vendor>
-     <topology sockets='1' cores='6' threads='1'/>
+  <cpu mode='custom' match='exact' check='full'>
+     <model fallback='forbid'>EPYC-IBPB</model>
+     <vendor>AMD</vendor>
+     <topology sockets='1' cores='6' threads='1'/>
  
  in windows, coreinfo reports incorrectly:
  
  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64
  
+ Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.
  
- Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm. 
- 
- With newer Qemu there is a fix (that does behave correctly) in using the dies parameter: 
-  <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>
+ With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
+  <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>
  
  The problem is that the dies are exposed differently than how AMD does
- it natively, they are exposed to Windows as sockets, which means, you
- can't ever have a machine with more than two CCX (6 cores) as Windows
- only supports two sockets. (Should this be reported as a separate bug?)
+ it natively, they are exposed to Windows as sockets, which means, that
+ if you are nto a business user, you can't ever have a machine with more
+ than two CCX (6 cores) as consumer versions of Windows only supports two
+ sockets. (Should this be reported as a separate bug?)

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Damir]

Hi,

I've since confirmed that this bug also exist (as expected) on Linux
guests, as well as Zen1 EPYC 7401 CPUs, to make sure this wasn't a
problem with the detection of the newer consumer platform.

Basically it seems (looking at the code with layman eyes) that as long
as you have a topology that is dividable by 4 or 8, it will always
result in the wrong topology being exposed to the guest, even when the
correct option can be built (12, 24 core CPUs, although, it would be
great if we could support 9 Core VM CPus as that is a reasonable use
case for VMs (3 CCXs of 3 Cores for a total of 9 (or 18 SMT threads)).

Pinging the author and committer of the TopoEXT feature / EPYC cpu model
as they should probably know best how to solve this issue.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Damir]

This is the commit I am referencing:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8f4202fb1080f86958782b1fca0bf0279f67d136

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Damir,
  We normally test Linux guests here. Can you please give me exact qemu command line. Even the SMP parameters(sockets,cores,threads,dies) will also work. I will try to recreate it locally first.
Give me example what works and what does not work.

I have recently sent few more patches to fix another bug. Please check if this makes any difference.
https://patchwork.kernel.org/cover/11272063/
https://lore.kernel.org/qemu-devel/157541968844.46157.17994918142533791313.stgit@naples-babu.amd.com/

This should apply cleanly on git://github.com/ehabkost/qemu.git (branch
x86-next)

Note: I will be on vacation until first week of Jan. Responses will be
delayed.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Same problem for Ryzen 9 3900X. There should be 4x L3 caches, but there
are only 3.

Same results with "host-passthrough" and "EPYC-IBPB". Windows doesn't
recognize the correct L3 cache layout.

>From coreinfo.exe:

Logical Processor to Cache Map:
**----------------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
********----------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
--------**--------------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------------  Unified Cache       5, Level 2,  512 KB, Assoc   8, LineSize  64
--------********--------  Unified Cache       6, Level 3,   16 MB, Assoc  16, LineSize  64
----------**------------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----------  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
--------------**--------  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--------  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--------  Unified Cache       9, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**------  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
----------------********  Unified Cache      11, Level 3,   16 MB, Assoc  16, LineSize  64
------------------**----  Data Cache          9, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Instruction Cache   9, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Unified Cache      12, Level 2,  512 KB, Assoc   8, LineSize  64
--------------------**--  Data Cache         10, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------**--  Instruction Cache  10, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------**--  Unified Cache      13, Level 2,  512 KB, Assoc   8, LineSize  64
----------------------**  Data Cache         11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Instruction Cache  11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Unified Cache      14, Level 2,  512 KB, Assoc   8, LineSize  64

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

AMD does not use dies. For AMD dies is normally set to 1. You probably
have to pass dies in some other ways. Did you try the latest qemu v 5.0?
Please try it.

Qemu expects the user to configure the topology based on their requirement.
 
Try replacing <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/> 
with <qemu:arg value='cores=6,threads=1,dies=1,sockets=1'/>

You can also use the numa configuration. There are multiple ways you can
achieve your required configuration.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Damir, Example of how to use numa configuration.
-smp 16,maxcpus=16,cores=16,threads=1,sockets=1 -numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15

This will help to put all the cores in correct L3 boundary. I strongly
suggest to use the latest qemu release.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

It could be an issue of how the kernel presents the CPU topology.

Hardware: AMD Ryzen 3900X 12 core 24 threads (SMT)
Host: Kernel 5.6.6, QEMU 4.2

virsh capabilities | grep "cpu id"
            <cpu id='0' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='3' socket_id='0' core_id='4' siblings='3,15'/>
            <cpu id='4' socket_id='0' core_id='5' siblings='4,16'/>
            <cpu id='5' socket_id='0' core_id='6' siblings='5,17'/>
            <cpu id='6' socket_id='0' core_id='8' siblings='6,18'/>
            <cpu id='7' socket_id='0' core_id='9' siblings='7,19'/>
            <cpu id='8' socket_id='0' core_id='10' siblings='8,20'/>
            <cpu id='9' socket_id='0' core_id='12' siblings='9,21'/>
            <cpu id='10' socket_id='0' core_id='13' siblings='10,22'/>
            <cpu id='11' socket_id='0' core_id='14' siblings='11,23'/>
            <cpu id='12' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='13' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='14' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='15' socket_id='0' core_id='4' siblings='3,15'/>
            <cpu id='16' socket_id='0' core_id='5' siblings='4,16'/>
            <cpu id='17' socket_id='0' core_id='6' siblings='5,17'/>
            <cpu id='18' socket_id='0' core_id='8' siblings='6,18'/>
            <cpu id='19' socket_id='0' core_id='9' siblings='7,19'/>
            <cpu id='20' socket_id='0' core_id='10' siblings='8,20'/>
            <cpu id='21' socket_id='0' core_id='12' siblings='9,21'/>
            <cpu id='22' socket_id='0' core_id='13' siblings='10,22'/>
            <cpu id='23' socket_id='0' core_id='14' siblings='11,23'/>

See how cpu id=3 gets core id=4, and cpu id=6 gets core id=8, etc.

cat /sys/devices/system/cpu/cpu2/topology/core_id
2

cat /sys/devices/system/cpu/cpu3/topology/core_id
4

However, the association of CPU IDs to L3 caches seems to be correct:

echo "Level  CPU list";for file in /sys/devices/system/cpu/cpu*/cache/index3; do echo $(cat $file/id) "    " $(cat $file/shared_cpu_list); done | sort --version-sort
Level  CPU list
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23

There are 4 L3 caches with the correct CPU lists (6 CPUs/threads each).

Is it possible that this weird CPU ID enumeration is causing the
confusion?

Haven't had a chance to check out QEMU 5.0, but hope to do that today.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Finally installed QEMU 5.0.0.154 - still the same. QEMU doesn't
recognize the L3 caches and still lists 3 L3 caches instead of 4 with 3
cores/6 threads.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Here the vm.log with the qemu command line (shortened):

2020-05-03 18:23:38.674+0000: starting up libvirt version: 5.10.0, qemu
version: 5.0.50v5.0.0-154-g2ef486e76d-dirty, kernel: 5.4.36-1-MANJARO

-machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,invtsc=on,hypervisor=on,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-vendor-id=AuthenticAMD,hv-frequencies,hv-crash,kvm=off,host-cache-info=on,l3-cache=off \
-m 49152 \
-mem-prealloc \
-mem-path /dev/hugepages/libvirt/qemu/1-win10 \
-overcommit mem-lock=off \
-smp 24,sockets=1,cores=12,threads=2 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot menu=off,strict=on \

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Hi Seiger,
I am not an expert on libvirt. I mostly use qemu command line for my test. I was able to achieve the 3960X configuration with the following command line. 

# qemu-system-x86_64 -name rhel7  -m 16384 -smp
24,cores=12,threads=2,sockets=1 -hda vdisk.qcow2 -enable-kvm -net nic
-net bridge,br=virbr0,helper=/usr/libexec/qemu-bridge-helper -cpu
host,+topoext -nographic -numa node,nodeid=0,cpus=0-5 -numa
node,nodeid=1,cpus=6-11 -numa node,nodeid=2,cpus=12-17 -numa
node,nodeid=3,cpus=18-23

Basically qemu does not have all the information to build the topology
for every configuration. It depends on  libvirt for that information.
See if this combination works for you.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Hello Babu,

Thanks for the reply and the QEMU command line. I will try to implement
it in the XML.

So essentially what you do is to define each group of cpus and associate
them with a numa node:

-numa node,nodeid=0,cpus=0-5 -numa node,nodeid=1,cpus=6-11 -numa
node,nodeid=2,cpus=12-17 -numa node,nodeid=3,cpus=18-23

Haven't tried it but that might work. Do you need QEMU 5.0 for this to
work, or is 4.2 OK?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Yes. Sieger. Please install 5.0 it should work fine. I am not sure about
4.2.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Damir]

Hello,

I took a look today at the layouts when using 1950X (which previously
worked, and yes, admittedly, I am using Windows / coreinfo), and any
basic config (previously something simple as Sockets=1,Cores=8, Theads=1
(now also Dies=1) worked, but now, the topology presents as if all cores
share L3, and that each two cores share L1C/L1D/L2, like if they were
smt-siblings. I would call this a serious regression.

I don't think using Numa Nodes is an ok way to solve this (especially
not when at least for 4CCX CPUs, this worked flawlessly before), as that
will make numa-aware applications start taking note of numa nodes, and
possibly do wierd things (plus, it introduces more configuration where
it was not needed before).

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

I upgraded to QEMU emulator version 5.0.50
Using q35-5.1 (the latest) and the following libvirt configuration:

  <memory unit="KiB">50331648</memory>
  <currentMemory unit="KiB">50331648</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">24</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="6" cpuset="3"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <vcpupin vcpu="8" cpuset="4"/>
    <vcpupin vcpu="9" cpuset="16"/>
    <vcpupin vcpu="10" cpuset="5"/>
    <vcpupin vcpu="11" cpuset="17"/>
    <vcpupin vcpu="12" cpuset="6"/>
    <vcpupin vcpu="13" cpuset="18"/>
    <vcpupin vcpu="14" cpuset="7"/>
    <vcpupin vcpu="15" cpuset="19"/>
    <vcpupin vcpu="16" cpuset="8"/>
    <vcpupin vcpu="17" cpuset="20"/>
    <vcpupin vcpu="18" cpuset="9"/>
    <vcpupin vcpu="19" cpuset="21"/>
    <vcpupin vcpu="20" cpuset="10"/>
    <vcpupin vcpu="21" cpuset="22"/>
    <vcpupin vcpu="22" cpuset="11"/>
    <vcpupin vcpu="23" cpuset="23"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.1">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="no"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <vendor_id state="on" value="AuthenticAMD"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="12" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="hypervisor"/>
    <feature policy="require" name="topoext"/>
    <numa>
      <cell id="0" cpus="0-2,12-14" memory="12582912" unit="KiB"/>
      <cell id="1" cpus="3-5,15-17" memory="12582912" unit="KiB"/>
      <cell id="2" cpus="6-8,18-20" memory="12582912" unit="KiB"/>
      <cell id="3" cpus="9-11,21-23" memory="12582912" unit="KiB"/>
    </numa>
  </cpu>

...

/var/log/libvirt/qemu/win10.log:

-machine pc-q35-5.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,invtsc=on,hypervisor=on,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-vendor-id=AuthenticAMD,hv-frequencies,hv-crash,kvm=off,host-cache-info=on,l3-cache=off \
-m 49152 \
-overcommit mem-lock=off \
-smp 24,sockets=1,cores=12,threads=2 \
-mem-prealloc \
-mem-path /dev/hugepages/libvirt/qemu/3-win10 \
-numa node,nodeid=0,cpus=0-2,cpus=12-14,mem=12288 \
-numa node,nodeid=1,cpus=3-5,cpus=15-17,mem=12288 \
-numa node,nodeid=2,cpus=6-8,cpus=18-20,mem=12288 \
-numa node,nodeid=3,cpus=9-11,cpus=21-23,mem=12288 \
...

For some reason I always get l3-cache=off.

CoreInfo.exe in Windows 10 then produces the following report
(shortened):

Logical to Physical Processor Map:
**----------------------  Physical Processor 0 (Hyperthreaded)
--*---------------------  Physical Processor 1
---*--------------------  Physical Processor 2
----**------------------  Physical Processor 3 (Hyperthreaded)
------**----------------  Physical Processor 4 (Hyperthreaded)
--------*---------------  Physical Processor 5
---------*--------------  Physical Processor 6
----------**------------  Physical Processor 7 (Hyperthreaded)
------------**----------  Physical Processor 8 (Hyperthreaded)
--------------*---------  Physical Processor 9
---------------*--------  Physical Processor 10
----------------**------  Physical Processor 11 (Hyperthreaded)
------------------**----  Physical Processor 12 (Hyperthreaded)
--------------------*---  Physical Processor 13
---------------------*--  Physical Processor 14
----------------------**  Physical Processor 15 (Hyperthreaded)

Logical Processor to Socket Map:
************************  Socket 0

Logical Processor to NUMA Node Map:
***---------***---------  NUMA Node 0
---***---------***------  NUMA Node 1
------***---------***---  NUMA Node 2
---------***---------***  NUMA Node 3

Approximate Cross-NUMA Node Access Cost (relative to fastest):
     00  01  02  03
00: 1.4 1.2 1.1 1.2
01: 1.1 1.1 1.3 1.1
02: 1.0 1.1 1.0 1.2
03: 1.1 1.2 1.2 1.2

Logical Processor to Cache Map:
**----------------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
***---------------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--*---------------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--*---------------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--*---------------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
---*--------------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
---*--------------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
---*--------------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
---***------------------  Unified Cache       4, Level 3,   16 MB, Assoc  16, LineSize  64
----**------------------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Unified Cache       5, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       7, Level 3,   16 MB, Assoc  16, LineSize  64
--------*---------------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
--------*---------------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
--------*---------------  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
--------*---------------  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64
---------*--------------  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
---------*--------------  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
---------*--------------  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
---------***------------  Unified Cache      11, Level 3,   16 MB, Assoc  16, LineSize  64
----------**------------  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Unified Cache      12, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----------  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Unified Cache      13, Level 2,  512 KB, Assoc   8, LineSize  64
------------***---------  Unified Cache      14, Level 3,   16 MB, Assoc  16, LineSize  64
--------------*---------  Data Cache          9, Level 1,   32 KB, Assoc   8, LineSize  64
--------------*---------  Instruction Cache   9, Level 1,   32 KB, Assoc   8, LineSize  64
--------------*---------  Unified Cache      15, Level 2,  512 KB, Assoc   8, LineSize  64
---------------*--------  Data Cache         10, Level 1,   32 KB, Assoc   8, LineSize  64
---------------*--------  Instruction Cache  10, Level 1,   32 KB, Assoc   8, LineSize  64
---------------*--------  Unified Cache      16, Level 2,  512 KB, Assoc   8, LineSize  64
---------------*--------  Unified Cache      17, Level 3,   16 MB, Assoc  16, LineSize  64
----------------**------  Data Cache         11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Instruction Cache  11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      18, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      19, Level 3,   16 MB, Assoc  16, LineSize  64
------------------**----  Data Cache         12, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Instruction Cache  12, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Unified Cache      20, Level 2,  512 KB, Assoc   8, LineSize  64
------------------***---  Unified Cache      21, Level 3,   16 MB, Assoc  16, LineSize  64
--------------------*---  Data Cache         13, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------*---  Instruction Cache  13, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------*---  Unified Cache      22, Level 2,  512 KB, Assoc   8, LineSize  64
---------------------*--  Data Cache         14, Level 1,   32 KB, Assoc   8, LineSize  64
---------------------*--  Instruction Cache  14, Level 1,   32 KB, Assoc   8, LineSize  64
---------------------*--  Unified Cache      23, Level 2,  512 KB, Assoc   8, LineSize  64
---------------------***  Unified Cache      24, Level 3,   16 MB, Assoc  16, LineSize  64
----------------------**  Data Cache         15, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Instruction Cache  15, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Unified Cache      25, Level 2,  512 KB, Assoc   8, LineSize  64

Logical Processor to Group Map:
************************  Group 0


The above result is even further away from the actual L3 cache configuration.

So numatune doesn't produce the expected outcome.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

Same problem here on 5.0 and 3900x (3 cores per CCX). And as stated
before - declaring NUMA nodes is definitely not the right solution if
the aim is to emulate the host CPU as close as possible.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

The problem is that disabled cores are not taken into account.. ALL Zen2
CPUs have L3 cache group per CCX and every CCX has 4 cores, the problem
is that some cores in each CCX (1 for 6 and 12-core CPUs, 2 for 3100)
are disabled for some models, but they still use their core ids (as can
be seen in virsh capabilities | grep "cpu id" output in above comments).
Looking at target/i386/cpu.c:5529, this is not taken into account.

Maybe the cleanest way to fix this is to emulate the host topology by
also skipping disabled core ids in the VM? That way, die offset will
actually match the real host CPU topology...

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

A workaround for Linux VMs is to disable CPUs (and setting their
number/pinnings accordingly, e.g. every 4th (and 3rd for 3100) core is
going to be 'dummy' and disabled system-wide) by e.g. echo 0 >
/sys/devices/system/cpu/cpu3/online

No good workaround for Windows VMs exists, as far as I know - the best
you can do is setting affinity to specific process(es) and avoid the
'dummy' CPUs, but I am not aware of any possibility to disable specific
CPUs (only limiting the overall number).

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Damir]

Hi Jan,

Problem for me now is why does every config (I can figure out) now
result in SMT on/L3 across all cores which is obviously never true on
Zen except if you have only less than 4 cores, 8 cores should always
result in 2 L3 Caches, and so should 16 Threads /w 8+SMT. This worked in
my initial post.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Latest qemu has removed all the hard coded configurations for AMD. It is
leaving everything to customize.  One way is to configure is using numa
nodes. This will make sure cpus under one numa node share same L3. Then
pin the correct host cpus to guest cpus using vcpupin. I would change
this -numa node,nodeid=0,cpus=0-2,cpus=12-14,mem=12288 to -numa
node,nodeid=0,cpus=0-2,cpus=3-5,mem=12288. Then have vcpupin map the
correct host cpu to guest cpu. Check if this works for you. Can you
please post lscpu output from host for everybody's understanding?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

No, creating artificial NUMA nodes is, simply put, never a good solution
for CPUs that operate as a single NUMA node - which is the case for all
Zen2 CPUs (except maybe EPYCs? not sure about those).

You may workaround the L3 issue that way, but hit many new bugs/problems
by introducing multiple NUMA nodes, _especially_ on Windows VMs, because
that OS has crappy NUMA handling and multitude of bugs related to it -
which was one of the major reasons why even Zen2 Threadrippers are now
single NUMA node (e.g. https://www.servethehome.com/wp-
content/uploads/2019/11/AMD-Ryzen-Threadripper-3960X-Topology.png ).

The host CPU architecture should be replicated as closely as possible on
the VM and for Zen2 CPUs with 4 cores per CCX, _this already works
perfectly_ - there are no problems on
3300X/3700(X)/3800X/3950X/3970X/3990X.

There is, unfortunately, no way to customize/specify the "disabled" CPU
cores in QEMU, and therefore no way to emulate 1 NUMA node + L3 cache
per 2/3 cores - only to passthrough the cache config from host, which is
unfortunately not done correctly for CPUs with disabled cores (but
again, works perfectly for CPUs with all 4 cores enabled per CCX).

lscpu:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          24
On-line CPU(s) list:             0-23
Thread(s) per core:              2
Core(s) per socket:              12
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           113
Model name:                      AMD Ryzen 9 3900X 12-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         2972.127
CPU max MHz:                     3800.0000
CPU min MHz:                     2200.0000
BogoMIPS:                        7602.55
Virtualization:                  AMD-V
L1d cache:                       384 KiB
L1i cache:                       384 KiB
L2 cache:                        6 MiB
L3 cache:                        64 MiB
NUMA node0 CPU(s):               0-23
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonsto
                                 p_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a mi
                                 salignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase b
                                 mi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru
                                  wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca


But the important thing has already been posted here in previous comments - notice the skipped core ids belonging to the disabled cores:

virsh capabilities | grep "cpu id":
<cpu id='0' socket_id='0' core_id='0' siblings='0,12'/>
<cpu id='1' socket_id='0' core_id='1' siblings='1,13'/>
<cpu id='2' socket_id='0' core_id='2' siblings='2,14'/>
<cpu id='3' socket_id='0' core_id='4' siblings='3,15'/>
<cpu id='4' socket_id='0' core_id='5' siblings='4,16'/>
<cpu id='5' socket_id='0' core_id='6' siblings='5,17'/>
<cpu id='6' socket_id='0' core_id='8' siblings='6,18'/>
<cpu id='7' socket_id='0' core_id='9' siblings='7,19'/>
<cpu id='8' socket_id='0' core_id='10' siblings='8,20'/>
<cpu id='9' socket_id='0' core_id='12' siblings='9,21'/>
<cpu id='10' socket_id='0' core_id='13' siblings='10,22'/>
<cpu id='11' socket_id='0' core_id='14' siblings='11,23'/>
<cpu id='12' socket_id='0' core_id='0' siblings='0,12'/>
<cpu id='13' socket_id='0' core_id='1' siblings='1,13'/>
<cpu id='14' socket_id='0' core_id='2' siblings='2,14'/>
<cpu id='15' socket_id='0' core_id='4' siblings='3,15'/>
<cpu id='16' socket_id='0' core_id='5' siblings='4,16'/>
<cpu id='17' socket_id='0' core_id='6' siblings='5,17'/>
<cpu id='18' socket_id='0' core_id='8' siblings='6,18'/>
<cpu id='19' socket_id='0' core_id='9' siblings='7,19'/>
<cpu id='20' socket_id='0' core_id='10' siblings='8,20'/>
<cpu id='21' socket_id='0' core_id='12' siblings='9,21'/>
<cpu id='22' socket_id='0' core_id='13' siblings='10,22'/>
<cpu id='23' socket_id='0' core_id='14' siblings='11,23'/>

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

Damir:
Hm, must be some misconfiguration, then. My config for Linux VMs to utilize 3 out of the 4 CCXs. Important parts of the libvirt domain XML:

  <vcpu placement="static">24</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="3"/>
    <vcpupin vcpu="1" cpuset="15"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="16"/>
    <vcpupin vcpu="4" cpuset="5"/>
    <vcpupin vcpu="5" cpuset="17"/>
    <vcpupin vcpu="6" cpuset="0,12"/>
    <vcpupin vcpu="7" cpuset="0,12"/>
    <vcpupin vcpu="8" cpuset="6"/>
    <vcpupin vcpu="9" cpuset="18"/>
    <vcpupin vcpu="10" cpuset="7"/>
    <vcpupin vcpu="11" cpuset="19"/>
    <vcpupin vcpu="12" cpuset="8"/>
    <vcpupin vcpu="13" cpuset="20"/>
    <vcpupin vcpu="14" cpuset="0,12"/>
    <vcpupin vcpu="15" cpuset="0,12"/>
    <vcpupin vcpu="16" cpuset="9"/>
    <vcpupin vcpu="17" cpuset="21"/>
    <vcpupin vcpu="18" cpuset="10"/>
    <vcpupin vcpu="19" cpuset="22"/>
    <vcpupin vcpu="20" cpuset="11"/>
    <vcpupin vcpu="21" cpuset="23"/>
    <vcpupin vcpu="22" cpuset="0,12"/>
    <vcpupin vcpu="23" cpuset="0,12"/>
    <emulatorpin cpuset="1,13"/>
    <iothreadpin iothread="1" cpuset="2,14"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/ccxtest-clone_VARS.fd</nvram>
  </os>
.
.
.
  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,host-cache-info=on,-amd-stibp"/>
  </qemu:commandline>

The CPUs with cpuset="0,12" are disabled once booted. The host-cache-
info=on is the part that makes sure that the cache config is passed to
the VM (but unfortunately does not take disabled cores into account,
which results in incorrect config). The qemu:commandline is added
because I need to add -amd-stibp, otherwise I wouldn't be able to boot.
This overrides most parts in the <cpu> XML part.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

"The CPUs with cpuset="0,12" are disabled once booted. The host-cache-
info=on is the part that makes sure that the cache config is passed to
the VM (but unfortunately does not take disabled cores into account,
which results in incorrect config). The qemu:commandline is added
because I need to add -amd-stibp, otherwise I wouldn't be able to boot.
This overrides most parts in the <cpu> XML part."

Is there a XML equivalent for host-cache-info=on ?

Will that work with model EPYC-IBPB as well?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Babu Moger]

Sieger, I am not an expert on XML. So, I dont know. Qemu probably cannot
handle disabled cores. I am still trying to learn more about this
problem.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

With regard to Jan's comment earlier and the virsh capabilities listing
the cores and siblings, also note the following lines from virsh
capabilities for a 3900X CPU:

    <cache>
      <bank id='0' level='3' type='both' size='16' unit='MiB' cpus='0-2,12-14'/>
      <bank id='1' level='3' type='both' size='16' unit='MiB' cpus='3-5,15-17'/>
      <bank id='2' level='3' type='both' size='16' unit='MiB' cpus='6-8,18-20'/>
      <bank id='3' level='3' type='both' size='16' unit='MiB' cpus='9-11,21-23'/>
    </cache>

virsh capabilities is perfectly able to identify the L3 cache structure
and associate the right cpus. It would be ideal to just use the above
output inside the libvirt domain configuration to "manually" define the
L3 cache, or something to that effect on the qemu command line.

Users could then decide to pin only part of the cpus, usually a multiple
of 6 (in the case of the 3900X) to align with the CCX.

I'm now on kernel 5.6.11 and QEMU v5.0.0.r533.gdebe78ce14-1 (from Arch
Linux AUR qemu-git), running q35-5.1. I will try the host-passthrough
with host-cache-info=on option Jan posted. Question - is host-cache-
info=on the same as <cache mode="passthrough"/> under <cpu mode=host-
passthrough...?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

<cache mode="passthrough"/>

adds "host-cache-info=on,l3-cache=off"

to the qemu -cpu args

I believe l3-cache=off is useless with host-cache-info=on

So <cache mode="passthrough"/> should do what you want.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Thanks Jan. I had some new hardware/software issues combined with the
QEMU 5.0.. issues that had my Windows VM crash after some minutes.

I totally overlooked the following:
    <vcpupin vcpu="6" cpuset="0,12"/>
    <vcpupin vcpu="7" cpuset="0,12"/>

So I guess you posted to answer to this:
https://www.reddit.com/r/VFIO/comments/erwzrg/think_i_found_a_workaround_to_get_l3_cache_shared/

As it's late, I'll try tomorrow. Sorry for all the confusion but I had a
real tough time with this Ryzen build.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Jan, I tried your suggestion but it didn't make a difference. Here is my
current setup:

h/w: AMD Ryzen 9 3900X
kernel: 5.4
QEMU: 5.0.0-6
Chipset selection: Q35-5.0

Configuration: host-passthrough, cache enabled

Use CoreInfo.exe inside Windows. The problem is this:

Logical Processor to Cache Map:
**---------------------- Data Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64
**---------------------- Instruction Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64
**---------------------- Unified Cache 0, Level 2, 512 KB, Assoc 8, LineSize 64
********---------------- Unified Cache 1, Level 3, 16 MB, Assoc 16, LineSize 64

The last line above should be as follows:

******------------------ Unified Cache 0, Level 3, 16 MB, Assoc 16,
LineSize 64

The cache is supposed to be associated with 3 cores a 2 threads in group
0. Yet it shows 8 (2x4) vcpus inside a cache that is associated with the
next group.

In total, I always get 3 L3 caches instead of 4 L4 caches for my 12
cores / 24 threads. Also see my next post.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

This is the CPU cache layout as shown by lscpu -a -e

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0          yes 3800.0000 2200.0000
  1    0      0    1 1:1:1:0          yes 3800.0000 2200.0000
  2    0      0    2 2:2:2:0          yes 3800.0000 2200.0000
  3    0      0    3 3:3:3:1          yes 3800.0000 2200.0000
  4    0      0    4 4:4:4:1          yes 3800.0000 2200.0000
  5    0      0    5 5:5:5:1          yes 3800.0000 2200.0000
  6    0      0    6 6:6:6:2          yes 3800.0000 2200.0000
  7    0      0    7 7:7:7:2          yes 3800.0000 2200.0000
  8    0      0    8 8:8:8:2          yes 3800.0000 2200.0000
  9    0      0    9 9:9:9:3          yes 3800.0000 2200.0000
 10    0      0   10 10:10:10:3       yes 3800.0000 2200.0000
 11    0      0   11 11:11:11:3       yes 3800.0000 2200.0000
 12    0      0    0 0:0:0:0          yes 3800.0000 2200.0000
 13    0      0    1 1:1:1:0          yes 3800.0000 2200.0000
 14    0      0    2 2:2:2:0          yes 3800.0000 2200.0000
 15    0      0    3 3:3:3:1          yes 3800.0000 2200.0000
 16    0      0    4 4:4:4:1          yes 3800.0000 2200.0000
 17    0      0    5 5:5:5:1          yes 3800.0000 2200.0000
 18    0      0    6 6:6:6:2          yes 3800.0000 2200.0000
 19    0      0    7 7:7:7:2          yes 3800.0000 2200.0000
 20    0      0    8 8:8:8:2          yes 3800.0000 2200.0000
 21    0      0    9 9:9:9:3          yes 3800.0000 2200.0000
 22    0      0   10 10:10:10:3       yes 3800.0000 2200.0000
 23    0      0   11 11:11:11:3       yes 3800.0000 2200.0000

I was trying to allocate cache using the cachetune feature in libvirt,
but it turns out to be either misleading or much too complicated to be
usable. Here is what I tried:

  <vcpu placement="static">24</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="6" cpuset="3"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <vcpupin vcpu="8" cpuset="4"/>
    <vcpupin vcpu="9" cpuset="16"/>
    <vcpupin vcpu="10" cpuset="5"/>
    <vcpupin vcpu="11" cpuset="17"/>
    <vcpupin vcpu="12" cpuset="6"/>
    <vcpupin vcpu="13" cpuset="18"/>
    <vcpupin vcpu="14" cpuset="7"/>
    <vcpupin vcpu="15" cpuset="19"/>
    <vcpupin vcpu="16" cpuset="8"/>
    <vcpupin vcpu="17" cpuset="20"/>
    <vcpupin vcpu="18" cpuset="9"/>
    <vcpupin vcpu="19" cpuset="21"/>
    <vcpupin vcpu="20" cpuset="10"/>
    <vcpupin vcpu="21" cpuset="22"/>
    <vcpupin vcpu="22" cpuset="11"/>
    <vcpupin vcpu="23" cpuset="23"/>
    <cachetune vcpus="0-2,12-14">
      <cache id="0" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="0-2,12-14"/>
    </cachetune>
    <cachetune vcpus="3-5,15-17">
      <cache id="1" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="3-5,15-17"/>
    </cachetune>
    <cachetune vcpus="6-8,18-20">
      <cache id="2" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="6-8,18-20"/>
    </cachetune>
    <cachetune vcpus="9-11,21-23">
      <cache id="3" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="9-11,21-23"/>
    </cachetune>
  </cputune>

Unfortunately it gives the following error when I try to start the VM:

Error starting domain: internal error: Missing or inconsistent resctrl
info for memory bandwidth allocation

I have resctrl mounted like this:

mount -t resctrl resctrl /sys/fs/resctrl

This error leads to the following description on how to allocate memory
bandwith: https://software.intel.com/content/www/us/en/develop/articles
/use-intel-resource-director-technology-to-allocate-memory-
bandwidth.html

I think this is over the top and perhaps I'm trying the wrong approach.
All I can say is that every suggestion I've seen and tried so far has
led me to one conclusion: QEMU does NOT support the L3 cache layout of
the new ZEN 2 arch CPUs such as the Ryzen 9 3900X.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

h-sieger,
that is a misunderstanding, read my comment carefully again:
"A workaround for Linux VMs is to disable CPUs (and setting their number/pinnings accordingly, e.g. every 4th (and 3rd for 3100) core is going to be 'dummy' and disabled system-wide) by e.g. echo 0 > /sys/devices/system/cpu/cpu3/online

No good workaround for Windows VMs exists, as far as I know - the best
you can do is setting affinity to specific process(es) and avoid the
'dummy' CPUs, but I am not aware of any possibility to disable specific
CPUs (only limiting the overall number)."

I do NOT have a fix - only a very ugly workaround for Linux guests only
- I cannot fix the cache layout, but on Linux, I can get around that by
adding dummy CPUs that I then disable in the guest during startup, so
they are not used - effectively making sure that only the correct 6
vCPUs / 3 cores are used. On Windows, you cannot do that, AFAIK.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Thanks for clarifying, Jan.

In the meantime I tried a number of so-called solutions published on
Reddit and other places, none of which seems to work.

So if I understand it correctly, there is currently no solution to the
incorrect l3 cache layout for Zen architecture CPUs. At best a
workaround for Linux guests.

I hope somebody is looking into that.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Thanks for clarifying, Jan.

In the meantime I tried a number of so-called solutions published on
Reddit and other places, none of which seems to work.

So if I understand it correctly, there is currently no solution to the
incorrect l3 cache layout for Zen architecture CPUs. At best a
workaround for Linux guests.

I hope somebody is looking into that.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

The problem is caused by the fact that with Ryzen CPUs with disabled
cores, the APIC IDs are not sequential on host - in order for cache
topology to be configured properly, there is a 'hole' in APIC ID and
core ID numbering (I have added full output of cpuid for my 3900X).
Unfortunately, adding holes to the numbering is the only way to achieve
what is needed for 3 cores per CCX as CPUID Fn8000_001D_EAX
NumSharingCache parameter rounds to  powers of two (for Ryzen 3100 with
2 cores per CCX, lowering NumSharingCache should also work, correctly
setting the L3 cache cores with their IDs still being sequential).

A small hack in x86_apicid_from_topo_ids() in include/hw/i386/topology.h
can introduce a correct numbering (at least if you do not have epyc set
as your cpu, then _epyc variant of the functions are used). But to fix
this properly will probably require some thought - maybe introduce the
ability to assign APIC IDs directly somehow? Or the ability to specify
the 'holes' somehow in the -smt param, or maybe -cpu host,topoext=on
should do this automatically? I don't know...

e.g. For 3 core per CCX CPUs, to fix this, at
include/hw/i386/topology.h:220 change:

(topo_ids->core_id << apicid_core_offset(topo_info)) |

to

((topo_ids->core_id + (topo_ids->core_id / 3)) <<
apicid_core_offset(topo_info)) |


The cache topology is now correct (-cpu host,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,host-cache-info=on -smp 18,sockets=1,dies=1,cores=9,threads=2), even in Windows:

Logical Processor to Cache Map:
**----------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
******------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
------******------  Unified Cache       5, Level 3,   16 MB, Assoc  16, LineSize  64
--------**--------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
----------**------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
------------******  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64


** Attachment added: "cpuid.txt"
   https://bugs.launchpad.net/qemu/+bug/1856335/+attachment/5383184/+files/cpuid.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

@Jan: this coreinfo output looks good.

I finally managed to get the core /cache alignment right, I believe:

  <vcpu placement="static" current="24">32</vcpu>
  <vcpus>
    <vcpu id="0" enabled="yes" hotpluggable="no"/>
    <vcpu id="1" enabled="yes" hotpluggable="yes"/>
    <vcpu id="2" enabled="yes" hotpluggable="yes"/>
    <vcpu id="3" enabled="yes" hotpluggable="yes"/>
    <vcpu id="4" enabled="yes" hotpluggable="yes"/>
    <vcpu id="5" enabled="yes" hotpluggable="yes"/>
    <vcpu id="6" enabled="no" hotpluggable="yes"/>
    <vcpu id="7" enabled="no" hotpluggable="yes"/>
    <vcpu id="8" enabled="yes" hotpluggable="yes"/>
    <vcpu id="9" enabled="yes" hotpluggable="yes"/>
    <vcpu id="10" enabled="yes" hotpluggable="yes"/>
    <vcpu id="11" enabled="yes" hotpluggable="yes"/>
    <vcpu id="12" enabled="yes" hotpluggable="yes"/>
    <vcpu id="13" enabled="yes" hotpluggable="yes"/>
    <vcpu id="14" enabled="no" hotpluggable="yes"/>
    <vcpu id="15" enabled="no" hotpluggable="yes"/>
    <vcpu id="16" enabled="yes" hotpluggable="yes"/>
    <vcpu id="17" enabled="yes" hotpluggable="yes"/>
    <vcpu id="18" enabled="yes" hotpluggable="yes"/>
    <vcpu id="19" enabled="yes" hotpluggable="yes"/>
    <vcpu id="20" enabled="yes" hotpluggable="yes"/>
    <vcpu id="21" enabled="yes" hotpluggable="yes"/>
    <vcpu id="22" enabled="no" hotpluggable="yes"/>
    <vcpu id="23" enabled="no" hotpluggable="yes"/>
    <vcpu id="24" enabled="yes" hotpluggable="yes"/>
    <vcpu id="25" enabled="yes" hotpluggable="yes"/>
    <vcpu id="26" enabled="yes" hotpluggable="yes"/>
    <vcpu id="27" enabled="yes" hotpluggable="yes"/>
    <vcpu id="28" enabled="yes" hotpluggable="yes"/>
    <vcpu id="29" enabled="yes" hotpluggable="yes"/>
    <vcpu id="30" enabled="no" hotpluggable="yes"/>
    <vcpu id="31" enabled="no" hotpluggable="yes"/>
  </vcpus>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="8" cpuset="3"/>
    <vcpupin vcpu="9" cpuset="15"/>
    <vcpupin vcpu="10" cpuset="4"/>
    <vcpupin vcpu="11" cpuset="16"/>
    <vcpupin vcpu="12" cpuset="5"/>
    <vcpupin vcpu="13" cpuset="17"/>
    <vcpupin vcpu="16" cpuset="6"/>
    <vcpupin vcpu="17" cpuset="18"/>
    <vcpupin vcpu="18" cpuset="7"/>
    <vcpupin vcpu="19" cpuset="19"/>
    <vcpupin vcpu="20" cpuset="8"/>
    <vcpupin vcpu="21" cpuset="20"/>
    <vcpupin vcpu="24" cpuset="9"/>
    <vcpupin vcpu="25" cpuset="21"/>
    <vcpupin vcpu="26" cpuset="10"/>
    <vcpupin vcpu="27" cpuset="22"/>
    <vcpupin vcpu="28" cpuset="11"/>
    <vcpupin vcpu="29" cpuset="23"/>
  </cputune>

...
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" dies="1" cores="16" threads="2"/>
    <cache mode="passthrough"/>


The Windows Coreinfo output is this:

Logical to Physical Processor Map:
**----------------  Physical Processor 0 (Hyperthreaded)
--**--------------  Physical Processor 1 (Hyperthreaded)
----**------------  Physical Processor 2 (Hyperthreaded)
------**----------  Physical Processor 3 (Hyperthreaded)
--------**--------  Physical Processor 4 (Hyperthreaded)
----------**------  Physical Processor 5 (Hyperthreaded)
------------**----  Physical Processor 6 (Hyperthreaded)
--------------**--  Physical Processor 7 (Hyperthreaded)
----------------**  Physical Processor 8 (Hyperthreaded)

Logical Processor to Socket Map:
******************  Socket 0

Logical Processor to NUMA Node Map:
******************  NUMA Node 0

No NUMA nodes.

Logical Processor to Cache Map:
**----------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
******------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
------******------  Unified Cache       5, Level 3,   16 MB, Assoc  16, LineSize  64
--------**--------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
----------**------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
------------******  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64
--------------**--  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**  Unified Cache      11, Level 2,  512 KB, Assoc   8, LineSize  64

Logical Processor to Group Map:
******************  Group 0


Haven't been able to test if it performs as expected. Need to do that.

Of course it would be great if QEMU was patched to recognize correct CCX
alignment as I'm not sure if and what will be the penalty of this weird
setup.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Jan Klos]

Yep, I read the Reddit thread, had no idea this was possible.

Still, both solutions are ugly workarounds and it would be nice to fix
this properly. But at least I don't have to patch and compile QEMU on my
own anymore.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Sanjay Basu]

h-sieger,
Your XML gave me very significant performance gains.
Is there any way to do this with more than 24 assigned cores?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

@sanjaybmd

I'm glad to read that it worked for you. In fact, since I posted the XML
I didn't have the time to do benchmarking, now my motherboard is dead
and I have to wait for repair/replacement.

Do you have any data to quantify the performance gain?

As to the number of cores, you will notice that my 3900X has only 12
physical cores, that is 24 threads. Yet I assigned 32 vcpus in total. 8
of them are disabled. This is to align the vcpus to the actual CCX
topology of 3 cores per CCX.

QEMU thinks the cores per CCX should be a multiple of 2, e.g. 2, 4, etc.
cores. So I assign 4 cores = 8 vcpus, and disable 2 vcpus to simulate
the actual topology.

If your CPU has more cores, you could scale it up. Be aware that the
3950X should not have this issue as it has 4 cores per CCX, if I
remember correctly.

Note: I took this idea from a Reddit post (see link somewhere above).

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Sanjay Basu]

h-sieger, 
I did some testing with geekbench 5:

baseline multicore score = 12733
https://browser.geekbench.com/v5/cpu/3069626

score with <cache="passthrough"> option = 12775
https://browser.geekbench.com/v5/cpu/3069415

best score with your xml above = 16960
https://browser.geekbench.com/v5/cpu/3066003

I'm running a 3960x and it is 3 cores per CCX so your xml above works well. I'm just now learning about all this so I'm still trying to figure out how to modify your xml to assign more cores. Anyway, I'm getting better performance out of my Windows 10 VM now assigning 24 vcpu as opposed to the 32 that I was assigning before!
By the way, I tried to email you directly because I'm not sure this is appropriate discussion for this bug report but I could not create an account on your website (captcha was malfunctioning). Hope you you can fix that soon.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Heiko Sieger]

Sanjay,

You can just increase the number of vcpus, such as:

<vcpu placement="static" current="48">64</vcpu>

then continue to define the vcpus:

    <vcpu id="32" enabled="yes" hotpluggable="yes"/>
    <vcpu id="33" enabled="yes" hotpluggable="yes"/>
    <vcpu id="34" enabled="yes" hotpluggable="yes"/>
    <vcpu id="35" enabled="yes" hotpluggable="yes"/>
    <vcpu id="36" enabled="yes" hotpluggable="yes"/>
    <vcpu id="37" enabled="yes" hotpluggable="yes"/>
    <vcpu id="38" enabled="no" hotpluggable="yes"/>
    <vcpu id="39" enabled="no" hotpluggable="yes"/>
    <vcpu id="40" enabled="yes" hotpluggable="yes"/>
    <vcpu id="41" enabled="yes" hotpluggable="yes"/>
    <vcpu id="42" enabled="yes" hotpluggable="yes"/>
    <vcpu id="43" enabled="yes" hotpluggable="yes"/>
    <vcpu id="44" enabled="yes" hotpluggable="yes"/>
    <vcpu id="45" enabled="yes" hotpluggable="yes"/>
    <vcpu id="46" enabled="no" hotpluggable="yes"/>
    <vcpu id="47" enabled="no" hotpluggable="yes"/>
    <vcpu id="48" enabled="yes" hotpluggable="yes"/>
    <vcpu id="49" enabled="yes" hotpluggable="yes"/>
    <vcpu id="50" enabled="yes" hotpluggable="yes"/>
    <vcpu id="51" enabled="yes" hotpluggable="yes"/>
    <vcpu id="52" enabled="yes" hotpluggable="yes"/>
    <vcpu id="53" enabled="yes" hotpluggable="yes"/>
    <vcpu id="54" enabled="no" hotpluggable="yes"/>
    <vcpu id="55" enabled="no" hotpluggable="yes"/>
    <vcpu id="56" enabled="yes" hotpluggable="yes"/>
    <vcpu id="57" enabled="yes" hotpluggable="yes"/>
    <vcpu id="58" enabled="yes" hotpluggable="yes"/>
    <vcpu id="59" enabled="yes" hotpluggable="yes"/>
    <vcpu id="60" enabled="yes" hotpluggable="yes"/>
    <vcpu id="61" enabled="yes" hotpluggable="yes"/>
    <vcpu id="62" enabled="no" hotpluggable="yes"/>
    <vcpu id="63" enabled="no" hotpluggable="yes"/>

(6x enabled=yes, then 2x enabled=no.)

You will get more vcpu ids than you have threads, but since you disable
16 out of 64, you will have 48 active.

vcpupin should continue as follows:

    <vcpupin vcpu="32" cpuset="24"/>
    <vcpupin vcpu="33" cpuset="36"/>
    <vcpupin vcpu="34" cpuset="25"/>
    <vcpupin vcpu="35" cpuset="37"/>
    <vcpupin vcpu="36" cpuset="26"/>
    <vcpupin vcpu="37" cpuset="38"/>
    <vcpupin vcpu="40" cpuset="27"/>
    <vcpupin vcpu="41" cpuset="39"/>
    <vcpupin vcpu="42" cpuset="28"/>
    <vcpupin vcpu="43" cpuset="40"/>
    <vcpupin vcpu="44" cpuset="29"/>
    <vcpupin vcpu="45" cpuset="41"/>
    <vcpupin vcpu="48" cpuset="30"/>
    <vcpupin vcpu="49" cpuset="42"/>
    <vcpupin vcpu="50" cpuset="31"/>
    <vcpupin vcpu="51" cpuset="43"/>
    <vcpupin vcpu="52" cpuset="32"/>
    <vcpupin vcpu="53" cpuset="44"/>
    <vcpupin vcpu="56" cpuset="33"/>
    <vcpupin vcpu="57" cpuset="45"/>
    <vcpupin vcpu="58" cpuset="34"/>
    <vcpupin vcpu="59" cpuset="46"/>
    <vcpupin vcpu="60" cpuset="35"/>
    <vcpupin vcpu="61" cpuset="47"/>

This is if you pin all vcpus to the VM, which may not be the best thing
to do. The maximum number of vcpus you can pin on a Threadripper 3960X
are 48.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Thomas Huth]

The QEMU project is currently considering to move its bug tracking to
another system. For this we need to know which bugs are still valid
and which could be closed already. Thus we are setting older bugs to
"Incomplete" now.

If you still think this bug report here is valid, then please switch
the state back to "New" within the next 60 days, otherwise this report
will be marked as "Expired". Or please mark it as "Fix Released" if
the problem has been solved with a newer version of QEMU already.

Thank you and sorry for the inconvenience.


** Changed in: qemu
       Status: New => Incomplete

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  Incomplete

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions

[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  Expired

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions