Re: About add an A64FX cache control function into resctrl

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Tan Shaopeng,

On 7/21/2021 1:10 AM, tan.shaopeng@fujitsu.com wrote:
> Hi Reinette,
> 
>> On 7/7/2021 4:26 AM, tan.shaopeng@fujitsu.com wrote:
>>>>> Sorry, I have not explained A64FX's sector cache function well yet.
>>>>> I think I need explain this function from different perspective.
>>>>
>>>> You have explained the A64FX's sector cache function well. I have
>>>> also read both specs to understand it better. It appears to me that
>>>> you are not considering the resctrl architecture as part of your
>>>> solution but instead just forcing your architecture onto the resctrl
>>>> filesystem. For example, in resctrl the resource groups are not just
>>>> a directory structure but has significance in what is being
>>>> represented within the directory (a class of service). The files
>>>> within a resource group's directory build on that. From your side I
>>>> have not seen any effort in aligning the sector cache function with the
>> resctrl architecture but instead you are just changing resctrl interface to match
>> the A64FX architecture.
>>>>
>>>> Could you please take a moment to understand what resctrl is and how
>>>> it could be mapped to A64FX in a coherent way?
>>>
>>> Previously, my idea is based on how to make instructions use different
>>> sectors in one task. After I studied resctrl, to utilize resctrl
>>> architecture on A64FX, I think it’s better to assign one sector to one
>>> task. Thanks for your idea that "sectors" could be considered the same
>>> as the resctrl "classes of service".
>>>
>>> Based on your idea, I am considering the implementation details.
>>> In this email, I will explain the outline of new proposal, and then
>>> please allow me to confirm a few technologies about resctrl.
>>>
>>> The outline of my proposal is as follows.
>>> - Add a sector function equivalent to Intel's CAT function into resctrl.
>>>     (divide shared L2 cache into multiple partitions for multiple cores
>>> use)
>>> - Allocate one sector to one resource group (one CLOSID). Since one
>>>     core can only be assigned to one resource group, on A64FX each core
>>>     only uses one sector at a time.
>>
>> ok, so a sector is a portion of cache and matches with what can be represented
>> with a resource group.
>>
>> The second part of your comment is not clear to me. In the first part you
>> mention: "one core can only be assigned to one resource group" - this seems to
>> indicate some static assignment between cores and sectors and if this is the
> 
> Sorry, does "static assignment between cores and sectors" mean
> each core always use a fixed sector id? For example, core 0 always
> use sector 0 at any case. It is not.
> 
>> case this needs more thinking since the current implementation assumes that
>> any core that can access the cache can access all resource groups associated
>> with that cache. On the other hand, you mention "on A64FX each core only uses
>> one sector at a time" - this now sounds dynamic and is how resctrl works since
>> the CPU is assigned a single class of service to indicate all resources
>> accessible to it.
> 
> It is correct. Each core can be assigned to any resource group, and
> each core only uses one sector at a time. Additionally, which sector
> each core uses depends on the resource group (class of service) ID.

Thank you for clarifying. From what I understand this could be supported  
by existing resctrl flows.

...

>>> In resctrl_sched_in(), there are comments as follow:
>>>     /*
>>>    * If this task has a closid/rmid assigned, use it.
>>>     * Else use the closid/rmid assigned to this cpu.
>>>     */
>>> I thought when we write PID to tasks file, this task (PID) will only
>>> run on the CPUs which are specified in cpus file in the same resource
>>> group. So, the task_struct's closid and cpu's closid is the same.
>>> When task's closid is different from cpu's closid?
>>
>> resctrl does not manage the affinity of tasks.
>>
>> Tony recently summarized the cpus file very well to me: The actual semantics of
>> the CPUs file is to associate a CLOSid for a task that is in the default resctrl
>> group ? while it is running on one of the listed CPUs.
>>
>> To answer your question the task's closid could be different from the CPU's
>> closid if the task's closid is 0 while it is running on a CPU that is in the cpus file
>> of a non-default resource group.
>>
>> You can see a summary of the decision flow in section "Resource allocation
>> rules" in Documentation/x86/resctrl.rst
>>
>> The "cpus" file was created in support of the real-time use cases. In these use
>> cases a group of CPUs can be designated as supporting the real-time work and
>> with their own resource group and assigned the needed resources to do the
>> real-time work. A real-time task can then be started with affinity to those CPUs
>> and dynamically any kernel threads (that will be started on the same CPU)
>> doing work on behalf of this task would be able to use the resources set aside
>> for the real-time work.
> 
> Thanks for your explanation. I understood it.
> 
> I will implement this sector function, and if I have other questions,
> please allow me to mail you.

I will help where I can. You may also be interested in the work James is  
busy with. See his latest series at
https://lore.kernel.org/lkml/20210715173043.14222-1-james.morse@arm.com/

Reinette