Re: [RFD] resctrl: reassigning a running container's CTRL_MON group

From: Reinette Chatre <reinette.chatre@intel.com>
To: Peter Newman <peternewman@google.com>
Cc: Tony Luck <tony.luck@intel.com>,
	"Yu, Fenghua" <fenghua.yu@intel.com>,
	"Eranian, Stephane" <eranian@google.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	"James Morse" <james.morse@arm.com>,
	Babu Moger <Babu.Moger@amd.com>,
	Gaurang Upasani <gupasani@google.com>
Subject: Re: [RFD] resctrl: reassigning a running container's CTRL_MON group
Date: Thu, 20 Oct 2022 12:08:43 -0700	[thread overview]
Message-ID: <da89ea90-8453-f848-38d1-a14195faa95e@intel.com> (raw)
In-Reply-To: <CALPaoCj8ps2YRH5T1eCRY-8MLwGi25cfbd0KNtjYXTq5baAXOQ@mail.gmail.com>

On 10/20/2022 1:48 AM, Peter Newman wrote:
> On Thu, Oct 20, 2022 at 1:54 AM Reinette Chatre
> <reinette.chatre@intel.com> wrote:
>> It is still not clear to me how palatable this will be on Arm systems.
>> This solution also involves changing the CLOSID/PARTID like your original
>> proposal and James highlighted that it would "mess up the bandwidth counters"
>> because of the way PARTID.PMG is used for monitoring. Perhaps even a new
>> PMG would need to be assigned during such a monitor group move. One requirement
>> for this RFD was to keep usage counts intact and from what I understand
>> this will not be possible on Arm systems.  There could be software mechanisms
>> to help reduce the noise during the transition. For example, some new limbo
>> mechanism that avoids re-assigning the old PARTID.PMG, while perhaps still
>> using the old PARTID.PMG to read usage counts for a while? Or would the
>> guidance just be that the counters will have some noise after the move?
> 
> I'm going to have to follow up on the details of this in James's thread.
> It sounded like we probably won't be able to create enough mon_groups
> under a single control group for the rename feature to even be useful.
> Rather, we expect the PARTID counts to be so much larger than the PMG
> counts that creating more mon_groups to reduce the number of control
> groups wouldn't make sense.
> 
> At least in our use case, we're literally creating "classes of service"
> to prioritize memory traffic, so we want a small number of control
> groups to represent the small number of priority levels, but enough
> RMIDs to count every job's traffic independently. For MPAM to support
> this MBM/MBA use case in exactly this fashion, we'd have to develop the
> monitors-not-matching-on-PARTID use case better in the MPAM
> architecture. But before putting much effort into that, I'd want to know
> if there's any payoff beyond being able to use resctrl the same way on
> both implementations.

If the expectation is that PARTID counts are very high then how about
a solution where multiple PARTIDs are associated with the same CTRL_MON group?
A CTRL_MON group presents a resource allocation to user space, CLOSIDs/PARTIDs
are not exposed. So using multiple PARTIDs for a resource group (all with the
same allocation) seems conceptually ok to me. (Please note, I did not do an
audit to see if there are any hidden assumption or look into lifting required
to support his.)

So, if a user moves a MON group to a new CTRL_MON group, if there are no
PARTID.PMG available in the destination CTRL_MON group to support the move
then one of the free PARTID can be used, automatically assigned with the
allocation of the destination CTRL_MON, and a new monitor group created using
the new PMG range brought with the new PARTID.

There may also be a way to guide resctrl to do something like this (use
available PARTID) when a user creates a new MON group. This may be a way
to address the earlier concern of how applications can decide to create
lots of MON groups vs CTRL_MON groups.

Reinette