linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Len Brown <lenb@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: X86 ML <x86@kernel.org>,
	linux-kernel@vger.kernel.org, Len Brown <len.brown@intel.com>,
	linux-doc@vger.kernel.org
Subject: Re: [PATCH 03/11] x86 topology: Add CPUID.1F multi-die/package support
Date: Thu, 28 Feb 2019 10:59:19 -0500	[thread overview]
Message-ID: <CAJvTdKk6B0rWMukNv3OKimSvbDqP+c83dTQpO4kFvbL7ETGPvg@mail.gmail.com> (raw)
In-Reply-To: <20190226135426.GU32477@hirez.programming.kicks-ass.net>

On Tue, Feb 26, 2019 at 8:54 AM Peter Zijlstra <peterz@infradead.org> wrote:

> > > It would've been nice to have the CPUID instruction 1F leaf reference
> > > 3B-3.9 in the SDM, and maybe mention this here too.
> >
> > I didn't mention SDM sections because they change -- leaving stale
> > pointers in our commit messages.  The SDM is re-published 4 times per
> > year.
>
> Yah, I know. Which is why I keep all SDMs. So if you say, book 3 section
> 8 of Jul'17, I can find it :-)

The SDM is like software -- usually (but not always) you are better
off with the latest version:-)

> > Cache enumeration in Leaf-4 is totally unchanged.
> > ACPI NUMA tables are totally unchanged.
>
> Sure; and yet Sub-NUMA-Clustering broke stuff in interesting ways. I'm
> trying to get a feel for how these levels will interact with all that.
>
> Before that SNC stuff, caches had never spanned NODEs (and I still
> think that is 'creative' at best).

Yeah, SNC is sort of a curve ball.  I guess it made enough stuff run better that
it is available as an option.  But it doesn't help everything, so it is disabled
by default...

I think from a scheduler point of view, sticking with the output of
CPUID.4 for the cache topology, and the ACPI tables for the
node topology/distances, is the right strategy.

> > From a scheduler point of view, imagine that a SKX system with 4 die
> > in 4 packages was mechanically re-designed so that those 4 die resided
> > in 2 double-sized packages.
> >
> > They may have tweaked the links between the die, but logically it is
> > identical and compatible, and the legacy kernel will function
> > properly.
>
> This example has LLC in die and yes that works.
>
> But I can imagine things like L2 in tile and L3 across tiles but within
> DIE and then it _might_ make sense to still consider the tile for
> scheduling.
>
> Another option is having the LLC off die; also not unheard of.
>
> And then there's many creative and slightly crazy ways this can all be
> combined :/

If any of those crazy things happen,  CPUID.B/CPUID.1F are not
going to help software understand it -- CPUID.4 and the NUMA tables
are the tool of choice.

> > So the effect of Leaf B,1F is that it defines the scope of MSRs.  eg.
> > what processors does a die-scope MSR cover.  That is why the rest of
> > the patch is about sysfs topology, and about package MSR scope.
> >
> > Yes, there will be more exotic MSR situations in future products --
> > the first ones are pretty simple -- something  called a
> > package-scope-MSR  in the SDM today becomes a die-scope-MSR in this
> > generation on a multi-die/package system.
>
> Yes :-(
>
> > It also reflects how many packages appear in sysfs, and this can
> > effect licensing of some kinds of software.
>
> That's just plain insanity and we should not let that affect our sysfs
> interfaces.

This change isn't made for compatibility with per-package licensing.
Indeed, vendors, who license  based on package-count need to
be made aware that on a system with multi-die/package, they'll
see their package count go _down_ as a result of this change.
Thankfully, I'm told that per-package licensing is quite rare --
most stuff that cares has moved to per-CPU.

I think a good semantic side effect of this series is that it
maintains the invariant that a physical package and a socket are synonymous.
While we don't use the word "socket" in Linux anymore, the industry
broadly assume that the two are synonyms.  And people expect that a
physical package really is a physical package -- you can see it,
buy it in a box, and hold it in your hand.

Functionally, the bottom line is that it allows software to discover topology
levels that previously needed to be discovered by looking up family/model,
in the past, which was sort of annoying.  The things that care are
things that care about MSR scope.   Thankfully, the list of things that care
about MSR scope is quite finite.

thanks,
-Len




--
Len Brown, Intel Open Source Technology Center

  reply	other threads:[~2019-02-28 15:59 UTC|newest]

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-19  3:40 [PATCH 0/11] multi-die/package support Len Brown
2019-02-19  3:40 ` [PATCH 01/11] x86 topology: fix doc typo Len Brown
2019-02-19  3:40   ` [PATCH 02/11] topolgy: simplify cputopology.txt formatting and wording Len Brown
     [not found]     ` <9108bd98-e9f4-fee3-80c7-72d540c48291@infradead.org>
2019-02-19 20:33       ` [linux-drivers-review] " Brown, Len
2019-02-19  3:40   ` [PATCH 03/11] x86 topology: Add CPUID.1F multi-die/package support Len Brown
2019-02-19 16:49     ` Liang, Kan
2019-02-19 19:27       ` Brown, Len
2019-02-20  2:59     ` Like Xu
2019-02-20  6:10       ` Len Brown
2019-02-20 10:55     ` Peter Zijlstra
2019-02-20 15:08       ` Len Brown
2019-02-26 13:54         ` Peter Zijlstra
2019-02-28 15:59           ` Len Brown [this message]
2019-02-28 17:56             ` Peter Zijlstra
2019-02-24 10:04     ` Brice Goglin
2019-02-25  5:31       ` Like Xu
2019-02-25  8:08       ` Brown, Len
2019-02-19  3:40   ` [PATCH 04/11] cpu topology: export die_id Len Brown
2019-02-19  3:40   ` [PATCH 05/11] x86 topology: export die_siblings Len Brown
2019-02-19 16:56     ` Liang, Kan
2019-02-19 18:43       ` Brown, Len
2019-02-19 19:33         ` Liang, Kan
2019-02-20 10:58           ` Peter Zijlstra
2019-02-20 21:52     ` Brice Goglin
2019-02-21  7:41       ` Len Brown
2019-02-21  8:38         ` Brice Goglin
2019-02-19  3:40   ` [PATCH 06/11] x86 topology: define topology_unique_die_id() Len Brown
2019-02-19  3:40   ` [PATCH 07/11] powercap/intel_rapl: simplify rapl_find_package() Len Brown
2019-02-19  9:11     ` Rafael J. Wysocki
2019-02-19  3:40   ` [PATCH 08/11] powercap/intel_rapl: Support multi-die/package Len Brown
2019-02-19  9:10     ` Rafael J. Wysocki
2019-02-20 11:02     ` Peter Zijlstra
2019-02-21  5:44       ` Len Brown
2019-02-26  4:41         ` Len Brown
2019-02-26  6:55           ` Zhang Rui
2019-02-19  3:40   ` [PATCH 09/11] powercap/intel_rapl: update rapl domain name and debug messages Len Brown
2019-02-19  9:10     ` Rafael J. Wysocki
2019-02-19  3:40   ` [PATCH 10/11] thermal/x86_pkg_temp_thermal: Support multi-die/package Len Brown
2019-02-19  3:40   ` [PATCH 11/11] hwmon/coretemp: " Len Brown
2019-02-19 16:46     ` Guenter Roeck

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAJvTdKk6B0rWMukNv3OKimSvbDqP+c83dTQpO4kFvbL7ETGPvg@mail.gmail.com \
    --to=lenb@kernel.org \
    --cc=len.brown@intel.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=peterz@infradead.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).