From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Babu Moger <babu.moger@amd.com>
Cc: corbet@lwn.net, reinette.chatre@intel.com, tglx@linutronix.de,
mingo@redhat.com, bp@alien8.de, fenghua.yu@intel.com,
dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com,
paulmck@kernel.org, akpm@linux-foundation.org,
quic_neeraju@quicinc.com, rdunlap@infradead.org,
damien.lemoal@opensource.wdc.com, songmuchun@bytedance.com,
peterz@infradead.org, jpoimboe@kernel.org, pbonzini@redhat.com,
chang.seok.bae@intel.com, pawan.kumar.gupta@linux.intel.com,
jmattson@google.com, daniel.sneddon@linux.intel.com,
sandipan.das@amd.com, tony.luck@intel.com, james.morse@arm.com,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
eranian@google.com
Subject: Re: [PATCH v7 12/12] Documentation/x86: Update resctrl.rst for new features
Date: Tue, 18 Oct 2022 10:24:27 +0700 [thread overview]
Message-ID: <Y04ca9ZgS44tBsTk@debian.me> (raw)
In-Reply-To: <166604565437.5345.14867510739122394750.stgit@bmoger-ubuntu>
[-- Attachment #1: Type: text/plain, Size: 8964 bytes --]
On Mon, Oct 17, 2022 at 05:27:34PM -0500, Babu Moger wrote:
> Update the documentation for the new features:
> 1. Slow Memory Bandwidth allocation (SMBA).
> With this feature, the QOS enforcement policies can be applied
> to the external slow memory connected to the host. QOS enforcement
> is accomplished by assigning a Class Of Service (COS) to a processor
> and specifying allocations or limits for that COS for each resource
> to be allocated.
>
> 2. Bandwidth Monitoring Event Configuration (BMEC).
> The bandwidth monitoring events mbm_total_bytes and mbm_local_bytes
> are set to count all the total and local reads/writes respectively.
> With the introduction of slow memory, the two counters are not
> enough to count all the different types of memory events. With the
> feature BMEC, the users have the option to configure mbm_total_bytes
> and mbm_local_bytes to count the specific type of events.
>
> Also add configuration instructions with examples.
>
> Signed-off-by: Babu Moger <babu.moger@amd.com>
> ---
> Documentation/x86/resctrl.rst | 139 ++++++++++++++++++++++++++++++++++++++++-
> 1 file changed, 137 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/x86/resctrl.rst b/Documentation/x86/resctrl.rst
> index 71a531061e4e..d0b4e1a2cb8d 100644
> --- a/Documentation/x86/resctrl.rst
> +++ b/Documentation/x86/resctrl.rst
> @@ -17,14 +17,16 @@ AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).
> This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
> flag bits:
>
> -============================================= ================================
> +=============================================== ================================
> RDT (Resource Director Technology) Allocation "rdt_a"
> CAT (Cache Allocation Technology) "cat_l3", "cat_l2"
> CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"
> CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"
> MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"
> MBA (Memory Bandwidth Allocation) "mba"
> -============================================= ================================
> +SMBA (Slow Memory Bandwidth Allocation) "smba"
> +BMEC (Bandwidth Monitoring Event Configuration) "bmec"
> +=============================================== ================================
>
> To use the feature mount the file system::
>
> @@ -161,6 +163,79 @@ with the following files:
> "mon_features":
> Lists the monitoring events if
> monitoring is enabled for the resource.
> + Example::
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mon_features
> + llc_occupancy
> + mbm_total_bytes
> + mbm_local_bytes
> +
> + If the system supports Bandwidth Monitoring Event
> + Configuration (BMEC), then the bandwidth events will
> + be configurable. The output will be::
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mon_features
> + llc_occupancy
> + mbm_total_bytes
> + mbm_total_config
> + mbm_local_bytes
> + mbm_local_config
> +
> +"mbm_total_config", "mbm_local_config":
> + These files contain the current event configuration for the events
> + mbm_total_bytes and mbm_local_bytes, respectively, when the
> + Bandwidth Monitoring Event Configuration (BMEC) feature is supported.
> + The event configuration settings are domain specific and will affect
> + all the CPUs in the domain.
> +
> + Following are the types of events supported:
> +
> + ==== ========================================================
> + Bits Description
> + ==== ========================================================
> + 6 Dirty Victims from the QOS domain to all types of memory
> + 5 Reads to slow memory in the non-local NUMA domain
> + 4 Reads to slow memory in the local NUMA domain
> + 3 Non-temporal writes to non-local NUMA domain
> + 2 Non-temporal writes to local NUMA domain
> + 1 Reads to memory in the non-local NUMA domain
> + 0 Reads to memory in the local NUMA domain
> + ==== ========================================================
> +
> + By default, the mbm_total_bytes configuration is set to 0x7f to count
> + all the event types and the mbm_local_bytes configuration is set to
> + 0x15 to count all the local memory events.
> +
> + Examples:
> +
> + * To view the current configuration::
> + ::
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_total_config
> + 0=0x7f;1=0x7f;2=0x7f;3=0x7f
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_local_config
> + 0=0x15;1=0x15;3=0x15;4=0x15
> +
> + * To change the mbm_total_bytes to count only reads on domain 0,
> + the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary
> + (in hexadecimal 0x33):
> + ::
> +
> + # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_config
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_total_config
> + 0=0x33;1=0x7f;2=0x7f;3=0x7f
> +
> + * To change the mbm_local_bytes to count all the slow memory reads
> + on domain 1, the bits 4 and 5 needs to be set, which is 110000b
> + in binary (in hexadecimal 0x30):
> + ::
> +
> + # echo "1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_config
> +
> + # cat /sys/fs/resctrl/info/L3_MON/mbm_local_config
> + 0=0x15;1=0x30;3=0x15;4=0x15
>
> "max_threshold_occupancy":
> Read/write file provides the largest value (in
> @@ -464,6 +539,26 @@ Memory bandwidth domain is L3 cache.
>
> MB:<cache_id0>=bw_MBps0;<cache_id1>=bw_MBps1;...
>
> +Slow Memory bandwidth Allocation (SMBA)
> +---------------------------------------
> +AMD hardwares support Slow Memory bandwidth Allocation (SMBA) feature.
> +Currently, CXL.memory is the only supported "slow" memory device.
> +With the support of SMBA, the hardware enables bandwidth allocation
> +on the slow memory devices. If there are multiple such devices in the
> +system, the throttling logic groups all the slow sources together
> +and applies the limit on them as a whole.
> +
> +The presence of SMBA (with CXL.memory) is independent of slow memory
> +devices presence. If there is no such devices on the system, then
> +setting the configuring SMBA will have no impact on the performance
> +of the system.
> +
> +The bandwidth domain for slow memory is L3 cache. Its schemata file
> +is formatted as:
> +::
> +
> + SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
> +
> Reading/writing the schemata file
> ---------------------------------
> Reading the schemata file will show the state of all resources
> @@ -479,6 +574,46 @@ which you wish to change. E.g.
> L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
> L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
>
> +Reading/writing the schemata file (on AMD systems)
> +--------------------------------------------------
> +Reading the schemata file will show the current bandwidth limit on all
> +domains. The allocated resources are in multiples of one eighth GB/s.
> +When writing to the file, you need to specify what cache id you wish to
> +configure the bandwidth limit.
> +
> +For example, to allocate 2GB/s limit on the first cache id:
> +
> +::
> +
> + # cat schemata
> + MB:0=2048;1=2048;2=2048;3=2048
> + L3:0=ffff;1=ffff;2=ffff;3=ffff
> +
> + # echo "MB:1=16" > schemata
> + # cat schemata
> + MB:0=2048;1= 16;2=2048;3=2048
> + L3:0=ffff;1=ffff;2=ffff;3=ffff
> +
> +Reading/writing the schemata file (on AMD systems) with SMBA feature
> +--------------------------------------------------------------------
> +Reading and writing the schemata file is the same as without SMBA in
> +above section.
> +
> +For example, to allocate 8GB/s limit on the first cache id:
> +
> +::
> +
> + # cat schemata
> + SMBA:0=2048;1=2048;2=2048;3=2048
> + MB:0=2048;1=2048;2=2048;3=2048
> + L3:0=ffff;1=ffff;2=ffff;3=ffff
> +
> + # echo "SMBA:1=64" > schemata
> + # cat schemata
> + SMBA:0=2048;1= 64;2=2048;3=2048
> + MB:0=2048;1=2048;2=2048;3=2048
> + L3:0=ffff;1=ffff;2=ffff;3=ffff
> +
> Cache Pseudo-Locking
> ====================
> CAT enables a user to specify the amount of cache space that an
>
>
Finally LGTM, thanks.
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
next prev parent reply other threads:[~2022-10-18 3:34 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-17 22:25 [PATCH v7 00/12] x86/resctrl: Support for AMD QoS new features Babu Moger
2022-10-17 22:26 ` [PATCH v7 01/12] x86/cpufeatures: Add Slow Memory Bandwidth Allocation feature flag Babu Moger
2022-10-27 18:49 ` Reinette Chatre
2022-10-27 19:17 ` Moger, Babu
2022-10-17 22:26 ` [PATCH v7 02/12] x86/resctrl: Add a new resource type RDT_RESOURCE_SMBA Babu Moger
2022-10-17 22:26 ` [PATCH v7 03/12] x86/cpufeatures: Add Bandwidth Monitoring Event Configuration feature flag Babu Moger
2022-10-17 22:26 ` [PATCH v7 04/12] x86/resctrl: Include new features in command line options Babu Moger
2022-10-17 22:26 ` [PATCH v7 05/12] x86/resctrl: Detect and configure Slow Memory Bandwidth allocation Babu Moger
2022-10-25 23:43 ` Reinette Chatre
2022-10-26 19:07 ` Moger, Babu
2022-10-26 20:23 ` Reinette Chatre
2022-10-27 15:30 ` Moger, Babu
2022-10-27 18:37 ` Reinette Chatre
2022-10-28 15:16 ` Moger, Babu
2022-10-17 22:26 ` [PATCH v7 06/12] x86/resctrl: Introduce data structure to support monitor configuration Babu Moger
2022-10-25 23:45 ` Reinette Chatre
2022-10-26 19:25 ` Moger, Babu
2022-10-17 22:26 ` [PATCH v7 07/12] x86/resctrl: Add sysfs interface to read mbm_total_bytes event configuration Babu Moger
2022-10-25 23:47 ` Reinette Chatre
2022-10-26 19:36 ` Moger, Babu
2022-10-17 22:27 ` [PATCH v7 08/12] x86/resctrl: Add sysfs interface to read mbm_local_bytes " Babu Moger
2022-10-17 22:27 ` [PATCH v7 09/12] x86/resctrl: Add sysfs interface to write mbm_total_bytes " Babu Moger
2022-10-25 23:48 ` Reinette Chatre
2022-10-26 19:52 ` Moger, Babu
2022-10-17 22:27 ` [PATCH v7 10/12] x86/resctrl: Add sysfs interface to write mbm_local_bytes " Babu Moger
2022-10-17 22:27 ` [PATCH v7 11/12] x86/resctrl: Replace smp_call_function_many() with on_each_cpu_mask() Babu Moger
2022-10-17 22:27 ` [PATCH v7 12/12] Documentation/x86: Update resctrl.rst for new features Babu Moger
2022-10-18 3:24 ` Bagas Sanjaya [this message]
2022-10-25 23:50 ` Reinette Chatre
2022-10-26 20:00 ` Moger, Babu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y04ca9ZgS44tBsTk@debian.me \
--to=bagasdotme@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=babu.moger@amd.com \
--cc=bp@alien8.de \
--cc=chang.seok.bae@intel.com \
--cc=corbet@lwn.net \
--cc=damien.lemoal@opensource.wdc.com \
--cc=daniel.sneddon@linux.intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=eranian@google.com \
--cc=fenghua.yu@intel.com \
--cc=hpa@zytor.com \
--cc=james.morse@arm.com \
--cc=jmattson@google.com \
--cc=jpoimboe@kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=paulmck@kernel.org \
--cc=pawan.kumar.gupta@linux.intel.com \
--cc=pbonzini@redhat.com \
--cc=peterz@infradead.org \
--cc=quic_neeraju@quicinc.com \
--cc=rdunlap@infradead.org \
--cc=reinette.chatre@intel.com \
--cc=sandipan.das@amd.com \
--cc=songmuchun@bytedance.com \
--cc=tglx@linutronix.de \
--cc=tony.luck@intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).