[Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs

From: Heiko Sieger <1856335@bugs.launchpad.net>
To: qemu-devel@nongnu.org
Subject: [Bug 1856335] Re: Cache Layout wrong on many Zen Arch CPUs
Date: Wed, 29 Jul 2020 02:23:39 -0000	[thread overview]
Message-ID: <159598941967.4737.7676730076336265460.malone@soybean.canonical.com> (raw)
In-Reply-To: 157625616239.22064.10423897892496347105.malonedeb@gac.canonical.com

Sanjay,

You can just increase the number of vcpus, such as:

<vcpu placement="static" current="48">64</vcpu>

then continue to define the vcpus:

    <vcpu id="32" enabled="yes" hotpluggable="yes"/>
    <vcpu id="33" enabled="yes" hotpluggable="yes"/>
    <vcpu id="34" enabled="yes" hotpluggable="yes"/>
    <vcpu id="35" enabled="yes" hotpluggable="yes"/>
    <vcpu id="36" enabled="yes" hotpluggable="yes"/>
    <vcpu id="37" enabled="yes" hotpluggable="yes"/>
    <vcpu id="38" enabled="no" hotpluggable="yes"/>
    <vcpu id="39" enabled="no" hotpluggable="yes"/>
    <vcpu id="40" enabled="yes" hotpluggable="yes"/>
    <vcpu id="41" enabled="yes" hotpluggable="yes"/>
    <vcpu id="42" enabled="yes" hotpluggable="yes"/>
    <vcpu id="43" enabled="yes" hotpluggable="yes"/>
    <vcpu id="44" enabled="yes" hotpluggable="yes"/>
    <vcpu id="45" enabled="yes" hotpluggable="yes"/>
    <vcpu id="46" enabled="no" hotpluggable="yes"/>
    <vcpu id="47" enabled="no" hotpluggable="yes"/>
    <vcpu id="48" enabled="yes" hotpluggable="yes"/>
    <vcpu id="49" enabled="yes" hotpluggable="yes"/>
    <vcpu id="50" enabled="yes" hotpluggable="yes"/>
    <vcpu id="51" enabled="yes" hotpluggable="yes"/>
    <vcpu id="52" enabled="yes" hotpluggable="yes"/>
    <vcpu id="53" enabled="yes" hotpluggable="yes"/>
    <vcpu id="54" enabled="no" hotpluggable="yes"/>
    <vcpu id="55" enabled="no" hotpluggable="yes"/>
    <vcpu id="56" enabled="yes" hotpluggable="yes"/>
    <vcpu id="57" enabled="yes" hotpluggable="yes"/>
    <vcpu id="58" enabled="yes" hotpluggable="yes"/>
    <vcpu id="59" enabled="yes" hotpluggable="yes"/>
    <vcpu id="60" enabled="yes" hotpluggable="yes"/>
    <vcpu id="61" enabled="yes" hotpluggable="yes"/>
    <vcpu id="62" enabled="no" hotpluggable="yes"/>
    <vcpu id="63" enabled="no" hotpluggable="yes"/>

(6x enabled=yes, then 2x enabled=no.)

You will get more vcpu ids than you have threads, but since you disable
16 out of 64, you will have 48 active.

vcpupin should continue as follows:

    <vcpupin vcpu="32" cpuset="24"/>
    <vcpupin vcpu="33" cpuset="36"/>
    <vcpupin vcpu="34" cpuset="25"/>
    <vcpupin vcpu="35" cpuset="37"/>
    <vcpupin vcpu="36" cpuset="26"/>
    <vcpupin vcpu="37" cpuset="38"/>
    <vcpupin vcpu="40" cpuset="27"/>
    <vcpupin vcpu="41" cpuset="39"/>
    <vcpupin vcpu="42" cpuset="28"/>
    <vcpupin vcpu="43" cpuset="40"/>
    <vcpupin vcpu="44" cpuset="29"/>
    <vcpupin vcpu="45" cpuset="41"/>
    <vcpupin vcpu="48" cpuset="30"/>
    <vcpupin vcpu="49" cpuset="42"/>
    <vcpupin vcpu="50" cpuset="31"/>
    <vcpupin vcpu="51" cpuset="43"/>
    <vcpupin vcpu="52" cpuset="32"/>
    <vcpupin vcpu="53" cpuset="44"/>
    <vcpupin vcpu="56" cpuset="33"/>
    <vcpupin vcpu="57" cpuset="45"/>
    <vcpupin vcpu="58" cpuset="34"/>
    <vcpupin vcpu="59" cpuset="46"/>
    <vcpupin vcpu="60" cpuset="35"/>
    <vcpupin vcpu="61" cpuset="47"/>

This is if you pin all vcpus to the VM, which may not be the best thing
to do. The maximum number of vcpus you can pin on a Threadripper 3960X
are 48.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1856335

Title:
  Cache Layout wrong on many Zen Arch CPUs

Status in QEMU:
  New

Bug description:
  AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems
  to always map Cache ass if it was an 4-Core per CCX CPU, which is
  incorrect, and costs upwards 30% performance (more realistically 10%)
  in L3 Cache Layout aware applications.

  Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT):

    <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='8' threads='1'/>

  In windows, coreinfo reports correctly:

  ****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

  On a 3-CCX CPU (3960X /w 6 cores and no SMT):

   <cpu mode='custom' match='exact' check='full'>
      <model fallback='forbid'>EPYC-IBPB</model>
      <vendor>AMD</vendor>
      <topology sockets='1' cores='6' threads='1'/>

  in windows, coreinfo reports incorrectly:

  ****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
  ----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64

  Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm.

  With newer Qemu there is a fix (that does behave correctly) in using the dies parameter:
   <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

  The problem is that the dies are exposed differently than how AMD does
  it natively, they are exposed to Windows as sockets, which means, that
  if you are nto a business user, you can't ever have a machine with
  more than two CCX (6 cores) as consumer versions of Windows only
  supports two sockets. (Should this be reported as a separate bug?)

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1856335/+subscriptions