From: Tony Luck <tony.luck@intel.com>
To: James Morse <james.morse@arm.com>
Cc: Reinette Chatre <reinette.chatre@intel.com>,
"Wieczor-Retman, Maciej" <maciej.wieczor-retman@intel.com>,
"Yu, Fenghua" <fenghua.yu@intel.com>,
Shuah Khan <shuah@kernel.org>,
"ilpo.jarvinen@linux.intel.com" <ilpo.jarvinen@linux.intel.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-kselftest@vger.kernel.org"
<linux-kselftest@vger.kernel.org>
Subject: Re: [PATCH 4/4] selftests/resctrl: Adjust SNC support messages
Date: Fri, 8 Mar 2024 10:42:59 -0800 [thread overview]
Message-ID: <ZetcM9GO2PH6SC0j@agluck-desk3> (raw)
In-Reply-To: <eacdc287-24bd-4137-85c8-df055cfd78b1@arm.com>
On Fri, Mar 08, 2024 at 06:06:45PM +0000, James Morse wrote:
> Hi guys,
>
> On 07/03/2024 23:16, Tony Luck wrote:
> > On Thu, Mar 07, 2024 at 02:39:08PM -0800, Reinette Chatre wrote:
> >> Thank you for the example. I find that significantly easier to
> >> understand than a single number in a generic "nodes_per_l3_cache".
> >> Especially with potential confusion surrounding inconsistent "nodes"
> >> between allocation and monitoring.
> >>
> >> How about domain_cpu_list and domain_cpu_map ?
>
> > Like this (my test system doesn't have SNC, so all domains are the same):
> >
> > $ cd /sys/fs/resctrl/info/
> > $ grep . */domain*
> > L3/domain_cpu_list:0: 0-35,72-107
> > L3/domain_cpu_list:1: 36-71,108-143
> > L3/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
> > L3/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
> > L3_MON/domain_cpu_list:0: 0-35,72-107
> > L3_MON/domain_cpu_list:1: 36-71,108-143
> > L3_MON/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
> > L3_MON/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
> > MB/domain_cpu_list:0: 0-35,72-107
> > MB/domain_cpu_list:1: 36-71,108-143
> > MB/domain_cpu_map:0: 0000,00000fff,ffffff00,0000000f,ffffffff
> > MB/domain_cpu_map:1: ffff,fffff000,000000ff,fffffff0,00000000
>
> This duplicates the information in /sys/devices/system/cpu/cpuX/cache/indexY ... is this
> really because that information is, er, wrong on SNC systems. Is it possible to fix that?
On an SNC system the resctrl domain for L3_MON becomes the SNC node
instead of the L3 cache instance. With 2, 3, or 4 SNC nodes per L3.
Even without the SNC issue this duplication may be a useful
convienience. On Intel to get from a resctrl domain is a multi-step
process to first find which of the indexY directories has level=3
and then look for the "id" that matches the domain.
> >From Tony's earlier description of how SNC changes things, the MB controls remain
> per-socket. To me it feels less invasive to fix the definition of L3 on these platforms to
> describe how it behaves (assuming that is possible), and define a new 'MB' that is NUMA
> scoped.
> This direction of redefining L3 means /sys/fs/resctrl and /sys/devices have different
> views of 'the' cache hierarchy.
I almost went partly in that direction when I started this epic voyage.
The "almost" part was to change the names of the monitoring directories
under mon_data from (legacy non-SNC system):
$ ls -l mon_data
total 0
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_L3_00
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_L3_01
to (2 socket, SNC=2 system):
$ ls -l mon_data
total 0
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_NODE_00
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_NODE_01
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_NODE_02
dr-xr-xr-x. 2 root root 0 Mar 8 10:31 mon_NODE_03
While that is in some ways a more accurate view, it breaks a lot of
legacy monitoring applications that expect the "L3" names.
> (I also think that this be over the threshold on 'funny machines look funny' - but I bet
> someone builds an arm machine that looks like this too!)
-Tony
next prev parent reply other threads:[~2024-03-08 18:43 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-06 10:38 [PATCH 0/4] SNC support for resctrl selftests Maciej Wieczor-Retman
2024-03-06 10:39 ` [PATCH 1/4] selftests/resctrl: Adjust effective L3 cache size with SNC enabled Maciej Wieczor-Retman
2024-03-06 10:39 ` [PATCH 2/4] selftests/resctrl: SNC support for CMT Maciej Wieczor-Retman
2024-03-08 13:53 ` Ilpo Järvinen
2024-03-08 13:59 ` Ilpo Järvinen
2024-03-13 10:23 ` Maciej Wieczor-Retman
2024-03-06 10:39 ` [PATCH 3/4] selftests/resctrl: SNC support for MBM Maciej Wieczor-Retman
2024-03-08 14:07 ` Ilpo Järvinen
2024-03-13 10:26 ` Maciej Wieczor-Retman
2024-03-06 10:39 ` [PATCH 4/4] selftests/resctrl: Adjust SNC support messages Maciej Wieczor-Retman
2024-03-06 21:54 ` Luck, Tony
2024-03-07 9:25 ` Maciej Wieczor-Retman
2024-03-07 17:18 ` Luck, Tony
2024-03-07 17:41 ` Reinette Chatre
2024-03-07 17:57 ` Luck, Tony
2024-03-07 19:52 ` Reinette Chatre
2024-03-07 21:14 ` Luck, Tony
2024-03-07 22:39 ` Reinette Chatre
2024-03-07 23:16 ` Tony Luck
2024-03-07 23:41 ` Reinette Chatre
2024-03-08 18:06 ` James Morse
2024-03-08 18:42 ` Tony Luck [this message]
2024-03-15 18:02 ` James Morse
2024-03-18 19:15 ` Reinette Chatre
2024-03-18 19:34 ` Luck, Tony
2024-03-18 20:32 ` Reinette Chatre
2024-03-18 20:47 ` Luck, Tony
2024-03-18 21:04 ` Luck, Tony
2024-03-18 21:26 ` Reinette Chatre
2024-03-18 22:00 ` Luck, Tony
2024-03-18 22:43 ` Reinette Chatre
2024-03-18 21:23 ` Reinette Chatre
2024-03-18 22:04 ` Luck, Tony
2024-03-19 21:01 ` Peter Newman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZetcM9GO2PH6SC0j@agluck-desk3 \
--to=tony.luck@intel.com \
--cc=fenghua.yu@intel.com \
--cc=ilpo.jarvinen@linux.intel.com \
--cc=james.morse@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=maciej.wieczor-retman@intel.com \
--cc=reinette.chatre@intel.com \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).