RE: About add an A64FX cache control function into resctrl

["tan.shaopeng@fujitsu.com" <tan.shaopeng@fujitsu.com>]

Hi Reinette,

> > > Sorry, I have not explained A64FX's sector cache function well yet.
> > > I think I need explain this function from different perspective.
> >
> > You have explained the A64FX's sector cache function well. I have also
> > read both specs to understand it better. It appears to me that you are
> > not considering the resctrl architecture as part of your solution but
> > instead just forcing your architecture onto the resctrl filesystem.
> > For example, in resctrl the resource groups are not just a directory
> > structure but has significance in what is being represented within the
> > directory (a class of service). The files within a resource group's
> > directory build on that. From your side I have not seen any effort in
> > aligning the sector cache function with the resctrl architecture but instead
> you are just changing resctrl interface to match the A64FX architecture.
> >
> > Could you please take a moment to understand what resctrl is and how
> > it could be mapped to A64FX in a coherent way?
> 
> Previously, my idea is based on how to make instructions use different sectors
> in one task. After I studied resctrl, to utilize resctrl architecture on A64FX, I
> think it’s better to assign one sector to one task. Thanks for your idea that
> "sectors" could be considered the same as the resctrl "classes of service".
> 
> Based on your idea, I am considering the implementation details.
> In this email, I will explain the outline of new proposal, and then please allow
> me to confirm a few technologies about resctrl.

Could you give me some comments & advices?

Best regards,
Tan Shaopeng

> The outline of my proposal is as follows.
> - Add a sector function equivalent to Intel's CAT function into resctrl.
>   (divide shared L2 cache into multiple partitions for multiple cores use)
> - Allocate one sector to one resource group (one CLOSID). Since one
>   core can only be assigned to one resource group, on A64FX each core
>   only uses one sector at a time.
> - Disable A64FX's HPC tag address override function. We only set each
>   core's default sector value according to closid(default sector ID=CLOSID).
> - No L1 cache control since L1 cache is not shared for cores. It is not
>   necessary to add L1 cache interface for schemata file.
> - No need to update schemata interface. Resctrl's L2 cache interface
>   (L2: <cache_id0> = <cbm>; <cache_id1> = <cbm>; ...)
>   will be used as it is. However, on A64FX, <cbm> does not indicate
>   the position of cache partition, only indicate the number of
>   cache ways (size).
> 
> This is the smallest start of incorporating sector cache function into resctrl. I
> will consider if we could add more sector cache features into resctrl (e.g.
> selecting different sectors from one task) after finishing this.
> 
> (some questions are below)
> 
> > >
> > >> On 5/17/2021 1:31 AM, tan.shaopeng@fujitsu.com wrote:
> >
> > > --------
> > > A64FX NUMA-PE-Cache Architecture:
> > > NUMA0:
> > >    PE0:
> > >      L1sector0,L1sector1,L1sector2,L1sector3
> > >    PE1:
> > >      L1sector0,L1sector1,L1sector2,L1sector3
> > >    ...
> > >    PE11:
> > >      L1sector0,L1sector1,L1sector2,L1sector3
> > >
> > >    L2sector0,1/L2sector2,3
> > > NUMA1:
> > >    PE0:
> > >      L1sector0,L1sector1,L1sector2,L1sector3
> > >    ...
> > >    PE11:
> > >      L1sector0,L1sector1,L1sector2,L1sector3
> > >
> > >    L2sector0,1/L2sector2,3
> > > NUMA2:
> > >    ...
> > > NUMA3:
> > >    ...
> > > --------
> > > In A64FX processor, one L1 sector cache capacity setting register is
> > > only for one PE and not shared among PEs. L2 sector cache maximum
> > > capacity setting registers are shared among PEs in same NUMA, and it
> > > is to be noted that changing these registers in one PE influences other PE.
> >
> > Understood. cache affinity is familiar to resctrl. When a CPU becomes
> > online it is discovered which caches/resources it has affinity to.
> > Resources then have CPU mask associated with them to indicate on which
> > CPU a register could be changed to configure the resource/cache. See
> > domain_add_cpu() and struct rdt_domain.
> 
> Is the following understanding correct?
> Struct rdt_domain is a group of online CPUs that share a same cache instance.
> When a CPU is online(resctrl initialization), the domain_add_cpu() function
> add the online cpu to corresponding rdt_domain (in rdt_resource:domains list).
> For example, if there are
> 4 L2 cache instances, then there will be 4 rdt_domain in the list and each CPU
> is assigned to corresponding rdt_domain.
> 
> The set values of cache/memory are stored in the *ctrl_val array (indexed by
> CLOSID) of struct rdt_domain. For example, in CAT function, the CBM value of
> CLOSID=x is stored in ctrl_val [x].
> When we create a resource group and write set values of cache into the
> schemata file, the update_domains() function updates the CBM value to
> ctrl_val [CLOSID = resource group ID] in rdt_domain and updates the CBM
> value to CBM register(MSR_IA32_Lx_CBM_BASE).
> 
> > > The number of ways for L2 Sector ID (0,1 or 2,3) can be set through
> > > any PEs in same NUMA. The sector ID 0,1 and 2,3 are not available at
> > > the same time in same NUMA.
> > >
> > >
> > > I think, in your idea, a resource group will be created for each sector ID.
> > > (> "sectors" could be considered the same as the resctrl "classes of
> > > service") Then, an example of resource group is created as follows.
> > > ・ L1: NUMAX-PEY-L1sector0 (X = 0,1,2,3.Y = 0,1,2 ... 11),
> > > ・ L2: NUMAX-L2sector0 (X = 0,1,2,3)
> > >
> > > In this example, sector with same ID(0) of all PEs is allocated to
> > > resource group. The L1D caches are numbered from
> > > NUMA0_PE0-L1sector0(0) to NUMA4_PE11-L1sector0(47) and the L2
> > caches
> > > numbered from
> > > NUMA0-L2sector0(0) to NUM4-L2sector0(3).
> > > (NUMA number X is from 0-4, PE number Y is from 0-11)
> > > (1) The number of ways of NUMAX-PEY-L1sector0 can be set
> independently
> > >      for each PEs (0-47). When run a task on this resource group,
> > >      we cannot control on which PE the task is running on and how many
> > >      cache ways the task is using.
> >
> > resctrl does not control the affinity on which PE/CPU a task is run.
> > resctrl is an interface with which to configure how resources are
> > allocated on the system. resctrl could thus provide interface with
> > which each sector of each cache instance is assigned a number of cache
> ways.
> > resctrl also provides an interface to assign a task with a class of
> > service (sector id?). Through this the task obtains access to all
> > resources that is allocated to the particular class of service (sector
> > id?). Depending on which CPU the task is running it may indeed
> > experience different performance if the sector id it is running with
> > does not have the same allocations on all cache instances. The affinity of the
> task needs to be managed separately using for example taskset.
> > Please see Documentation/x86/resctrl.rst "Examples for RDT allocation
> usage"
> 
> In resctrl_sched_in(), there are comments as follow:
>   /*
>  * If this task has a closid/rmid assigned, use it.
>   * Else use the closid/rmid assigned to this cpu.
>   */
> I thought when we write PID to tasks file, this task (PID) will only run on the
> CPUs which are specified in cpus file in the same resource group. So, the
> task_struct's closid and cpu's closid is the same.
> When task's closid is different from cpu's closid?
> 
> 
> Best regards,
> Tan Shaopeng


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel