resctrl2 - status

resctrl2 - status

[Luck, Tony]

I've posted before about my belief that it is time to re-invent the
Linux resctrl implementation to make it easier for CPU architectures
with different underlying resource control and monitoring capabilities to
implement those features without being unduly constrained by the quirks
of other architectural designs.

I'm super interested to hear from ARM folks on whether this can
be a suitable base for implementing MPAM.

I now have a reasonably complete example implementation that divides
the resctrl code into a an architecture neutral core layer that is
responsible for creating groups (directories) and associating tasks
with those groups. This layer also manages CPU hotplug events to
update domain lists for each control/monitor feature.

The architecture specific features are each implemented as loadable
modules that register with the core code to receive callbacks for
operations at the core level. E.g. modifying the schemata file for
a group results in callbacks to modules to parse changes to lines
that the module owns, and if all changes to all lines in the schemata
file are accepted by modules, there is a callback to instruct the
module to apply the changes (e.g. for X86 by updating MSRs).

Current snapshot of the code is here:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v65rc7

The commits aren't yet broken into bite-sized pieces. Everything is
in four patches (first two of them are huge).

de8bb88b3285 resctrl2: Architecture independent code
        Although this is big. The code is broken into fifteen file
        that each handle a specific part of the code. The largest
        (directory.c that handles mkdir/rmdir/rename) is only 481
        lines. Most are under 200 lines each.

98831f3edd3b resctrl2: Arch x86 modules for most of the legacy control/monitor functions
        This is also divided into code for each loadable module
        to implement just one X86 RDT feature each. At some point it
        could be broken into separate patches for each module.

d0f1b0a0a555 resctrl2: AMD Slow Memory Bandwidth Allocation
        I did put all the code for this feature into one patch. Should
        give some idea on how easy it is to use the core features
        to implement a specific control feature.

2a1dcfebba2d resctrl2: AMD Bandwidth Monitoring Event Configuration
        Ditto for this which just has to provide writable "info"
        files to allow a user to provide configuration options for
        an architectural feature. Needs a new hook to reset the
        MBM counter files when the configuration is changed to
        completely replicate the legacy resctrl functionality.


I don't have AMD hardware, so the AMD pieces are untested.


To try it out on X86 you need to pick CONFIG_RESCTRL2_FS=y
and then pick which modules to build. Full list is:

CONFIG_AMD_MBEC=m
CONFIG_AMD_SMBA=m
CONFIG_X86_RDT_IORDT=m
CONFIG_X86_RDT_L3_CAT=m
CONFIG_X86_RDT_L2_CAT=m
CONFIG_X86_RDT_L3_CDP=m
CONFIG_X86_RDT_L2_CDP=m
CONFIG_X86_RDT_SHOW_IDS=m
CONFIG_X86_RDT_L3_MBA=m
CONFIG_X86_RDT_L3_MBA_MBPS=m
CONFIG_X86_RDT_LLC_OCCUPANCY=m
CONFIG_X86_RDT_MBM_TOTAL_BYTES=m
CONFIG_X86_RDT_MBM_LOCAL_BYTES=m
CONFIG_X86_RDT_MBM_TOTAL_RATE=m
CONFIG_X86_RDT_MBM_LOCAL_RATE=m
CONFIG_X86_RDT_L2_PSEUDOLOCK=m
CONFIG_X86_RDT_L3_PSEUDOLOCK=m

After booting load the modules you want for the features you want
before (or after) mounting /sys/fs/resctrl.

There are no mount options. Just pick the right modules. E.g.

# modprobe rdt_l3_cat

for basic L3 cache control

# modprobe rdt_l3_cdp

for L3 cache control with code/data prioritization

# modprobe rdt_l3_pseudolock

for L3 cache control with pseudo cache locking support

-Tony

resctrl2 - status

[Luck, Tony]

I've posted before about my belief that it is time to re-invent the
Linux resctrl implementation to make it easier for CPU architectures
with different underlying resource control and monitoring capabilities to
implement those features without being unduly constrained by the quirks
of other architectural designs.

I'm super interested to hear from ARM folks on whether this can
be a suitable base for implementing MPAM.

I now have a reasonably complete example implementation that divides
the resctrl code into a an architecture neutral core layer that is
responsible for creating groups (directories) and associating tasks
with those groups. This layer also manages CPU hotplug events to
update domain lists for each control/monitor feature.

The architecture specific features are each implemented as loadable
modules that register with the core code to receive callbacks for
operations at the core level. E.g. modifying the schemata file for
a group results in callbacks to modules to parse changes to lines
that the module owns, and if all changes to all lines in the schemata
file are accepted by modules, there is a callback to instruct the
module to apply the changes (e.g. for X86 by updating MSRs).

Current snapshot of the code is here:

git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_v65rc7

The commits aren't yet broken into bite-sized pieces. Everything is
in four patches (first two of them are huge).

de8bb88b3285 resctrl2: Architecture independent code
        Although this is big. The code is broken into fifteen file
        that each handle a specific part of the code. The largest
        (directory.c that handles mkdir/rmdir/rename) is only 481
        lines. Most are under 200 lines each.

98831f3edd3b resctrl2: Arch x86 modules for most of the legacy control/monitor functions
        This is also divided into code for each loadable module
        to implement just one X86 RDT feature each. At some point it
        could be broken into separate patches for each module.

d0f1b0a0a555 resctrl2: AMD Slow Memory Bandwidth Allocation
        I did put all the code for this feature into one patch. Should
        give some idea on how easy it is to use the core features
        to implement a specific control feature.

2a1dcfebba2d resctrl2: AMD Bandwidth Monitoring Event Configuration
        Ditto for this which just has to provide writable "info"
        files to allow a user to provide configuration options for
        an architectural feature. Needs a new hook to reset the
        MBM counter files when the configuration is changed to
        completely replicate the legacy resctrl functionality.


I don't have AMD hardware, so the AMD pieces are untested.


To try it out on X86 you need to pick CONFIG_RESCTRL2_FS=y
and then pick which modules to build. Full list is:

CONFIG_AMD_MBEC=m
CONFIG_AMD_SMBA=m
CONFIG_X86_RDT_IORDT=m
CONFIG_X86_RDT_L3_CAT=m
CONFIG_X86_RDT_L2_CAT=m
CONFIG_X86_RDT_L3_CDP=m
CONFIG_X86_RDT_L2_CDP=m
CONFIG_X86_RDT_SHOW_IDS=m
CONFIG_X86_RDT_L3_MBA=m
CONFIG_X86_RDT_L3_MBA_MBPS=m
CONFIG_X86_RDT_LLC_OCCUPANCY=m
CONFIG_X86_RDT_MBM_TOTAL_BYTES=m
CONFIG_X86_RDT_MBM_LOCAL_BYTES=m
CONFIG_X86_RDT_MBM_TOTAL_RATE=m
CONFIG_X86_RDT_MBM_LOCAL_RATE=m
CONFIG_X86_RDT_L2_PSEUDOLOCK=m
CONFIG_X86_RDT_L3_PSEUDOLOCK=m

After booting load the modules you want for the features you want
before (or after) mounting /sys/fs/resctrl.

There are no mount options. Just pick the right modules. E.g.

# modprobe rdt_l3_cat

for basic L3 cache control

# modprobe rdt_l3_cdp

for L3 cache control with code/data prioritization

# modprobe rdt_l3_pseudolock

for L3 cache control with pseudo cache locking support

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/24/2023 11:10 AM, Luck, Tony wrote:

..

> After booting load the modules you want for the features you want
> before (or after) mounting /sys/fs/resctrl.

Could you please elaborate how user space is expected to
operate? For example, focusing on "load the modules you want",
sounds like user space is required to know what feature is supported
by the underlying hardware and load appropriate module in response.

In current resctrl the agreement is that the features are no longer
visible in /proc/cpuinfo if their support can be discovered via resctrl.
See, for example, how SMBA and BMEC does not appear in /proc/cpuinfo
because user space can learn about them via content in
/sys/fs/resctrl/info.
User space thus cannot rely on parsing /proc/cpuinfo to know which
modules to load. resctrl also supports module specific features,
like pseudo-locking making feature detection more complicated.
I'd expect that in the near future there will be a variety of ways
(beyond just running CPUID) in which features should be enumerated.
Is the expectation that user space needs to know how to enumerate
all the various features to know which modules can/should be loaded?

Alternatively, can user space just take a "load all resctrl modules
and see what sticks" (even modules of different architectures since
a user space may want to be generic) approach? 

This work is stated to "make it easier for CPU architectures
with different underlying resource control and monitoring capabilities to
implement those features without being unduly constrained by the quirks
of other architectural designs". It is not clear to me why making
the code modular requires everything to be modules. 

Finally, what is the plan to deal with current users that just mount
resctrl and expect to learn from it what features are supported?

> 
> There are no mount options. Just pick the right modules. E.g.
> 
> # modprobe rdt_l3_cat
> 
> for basic L3 cache control
> 
> # modprobe rdt_l3_cdp
> 
> for L3 cache control with code/data prioritization
> 
> # modprobe rdt_l3_pseudolock
> 
> for L3 cache control with pseudo cache locking support

Reinette

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/24/2023 11:10 AM, Luck, Tony wrote:

..

> After booting load the modules you want for the features you want
> before (or after) mounting /sys/fs/resctrl.

Could you please elaborate how user space is expected to
operate? For example, focusing on "load the modules you want",
sounds like user space is required to know what feature is supported
by the underlying hardware and load appropriate module in response.

In current resctrl the agreement is that the features are no longer
visible in /proc/cpuinfo if their support can be discovered via resctrl.
See, for example, how SMBA and BMEC does not appear in /proc/cpuinfo
because user space can learn about them via content in
/sys/fs/resctrl/info.
User space thus cannot rely on parsing /proc/cpuinfo to know which
modules to load. resctrl also supports module specific features,
like pseudo-locking making feature detection more complicated.
I'd expect that in the near future there will be a variety of ways
(beyond just running CPUID) in which features should be enumerated.
Is the expectation that user space needs to know how to enumerate
all the various features to know which modules can/should be loaded?

Alternatively, can user space just take a "load all resctrl modules
and see what sticks" (even modules of different architectures since
a user space may want to be generic) approach? 

This work is stated to "make it easier for CPU architectures
with different underlying resource control and monitoring capabilities to
implement those features without being unduly constrained by the quirks
of other architectural designs". It is not clear to me why making
the code modular requires everything to be modules. 

Finally, what is the plan to deal with current users that just mount
resctrl and expect to learn from it what features are supported?

> 
> There are no mount options. Just pick the right modules. E.g.
> 
> # modprobe rdt_l3_cat
> 
> for basic L3 cache control
> 
> # modprobe rdt_l3_cdp
> 
> for L3 cache control with code/data prioritization
> 
> # modprobe rdt_l3_pseudolock
> 
> for L3 cache control with pseudo cache locking support

Reinette

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: resctrl2 - status

[Luck, Tony]

> Alternatively, can user space just take a "load all resctrl modules
> and see what sticks" (even modules of different architectures since
> a user space may want to be generic) approach? 

This mostly works. Except for the cases where different modules access
the same underlying hardware, so can't be loaded together.

Examples:

rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
But this is already true ... they have to decide whether to pass the "-o cdp" option
to mount.

rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
with percentages, or with MB/sec values. Again the user already has to make this
decision when choosing mount options.


Maybe the "What resctrl options does this machine support?" question would be
best answered with a small utility?

-Tony

RE: resctrl2 - status

[Luck, Tony]

> Alternatively, can user space just take a "load all resctrl modules
> and see what sticks" (even modules of different architectures since
> a user space may want to be generic) approach? 

This mostly works. Except for the cases where different modules access
the same underlying hardware, so can't be loaded together.

Examples:

rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
But this is already true ... they have to decide whether to pass the "-o cdp" option
to mount.

rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
with percentages, or with MB/sec values. Again the user already has to make this
decision when choosing mount options.


Maybe the "What resctrl options does this machine support?" question would be
best answered with a small utility?

-Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 11:09 AM, Luck, Tony wrote:
>> Alternatively, can user space just take a "load all resctrl modules
>> and see what sticks" (even modules of different architectures since
>> a user space may want to be generic) approach? 
> 
> This mostly works. Except for the cases where different modules access
> the same underlying hardware, so can't be loaded together.
> 
> Examples:
> 
> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> But this is already true ... they have to decide whether to pass the "-o cdp" option
> to mount.
> 
> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> with percentages, or with MB/sec values. Again the user already has to make this
> decision when choosing mount options.
> 
> 
> Maybe the "What resctrl options does this machine support?" question would be
> best answered with a small utility?

A user space utility or a kernel provided utility? If it is a user space utility
I think it would end up needing to duplicate what the kernel is required to do
to know if a particular feature is supported. It seems appropriate that this
could be a kernel utility that can share this existing information with user
space. resctrl already supports the interface for this via /sys/fs/resctrl/info.

fyi ... as with previous attempts to discuss this work I find it difficult
to discuss this work when you are selective about what you want to discuss/answer
and just wipe the rest. Through this I understand that I am not your target
audience.

Reinette

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 11:09 AM, Luck, Tony wrote:
>> Alternatively, can user space just take a "load all resctrl modules
>> and see what sticks" (even modules of different architectures since
>> a user space may want to be generic) approach? 
> 
> This mostly works. Except for the cases where different modules access
> the same underlying hardware, so can't be loaded together.
> 
> Examples:
> 
> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> But this is already true ... they have to decide whether to pass the "-o cdp" option
> to mount.
> 
> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> with percentages, or with MB/sec values. Again the user already has to make this
> decision when choosing mount options.
> 
> 
> Maybe the "What resctrl options does this machine support?" question would be
> best answered with a small utility?

A user space utility or a kernel provided utility? If it is a user space utility
I think it would end up needing to duplicate what the kernel is required to do
to know if a particular feature is supported. It seems appropriate that this
could be a kernel utility that can share this existing information with user
space. resctrl already supports the interface for this via /sys/fs/resctrl/info.

fyi ... as with previous attempts to discuss this work I find it difficult
to discuss this work when you are selective about what you want to discuss/answer
and just wipe the rest. Through this I understand that I am not your target
audience.

Reinette

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: resctrl2 - status

[Luck, Tony]

> >> Alternatively, can user space just take a "load all resctrl modules
> >> and see what sticks" (even modules of different architectures since
> >> a user space may want to be generic) approach?
> >
> > This mostly works. Except for the cases where different modules access
> > the same underlying hardware, so can't be loaded together.
> >
> > Examples:
> >
> > rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> > But this is already true ... they have to decide whether to pass the "-o cdp" option
> > to mount.
> >
> > rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> > with percentages, or with MB/sec values. Again the user already has to make this
> > decision when choosing mount options.
> >
> >
> > Maybe the "What resctrl options does this machine support?" question would be
> > best answered with a small utility?
>
> A user space utility or a kernel provided utility? If it is a user space utility
> I think it would end up needing to duplicate what the kernel is required to do
> to know if a particular feature is supported. It seems appropriate that this
> could be a kernel utility that can share this existing information with user
> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.

I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
all features, a utility has access to all the CPUID leaves that contain the
details of each feature enumeration.

> fyi ... as with previous attempts to discuss this work I find it difficult
> to discuss this work when you are selective about what you want to discuss/answer
> and just wipe the rest. Through this I understand that I am not your target
> audience.

Not my intent. I value your input highly. I'm maybe too avid a follower of the
"trim your replies" school of e-mail etiquette. I thought I'd covered the gist
of your message.

I'll try to be more thorough in responding in the future.

-Tony

RE: resctrl2 - status

[Luck, Tony]

> >> Alternatively, can user space just take a "load all resctrl modules
> >> and see what sticks" (even modules of different architectures since
> >> a user space may want to be generic) approach?
> >
> > This mostly works. Except for the cases where different modules access
> > the same underlying hardware, so can't be loaded together.
> >
> > Examples:
> >
> > rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> > But this is already true ... they have to decide whether to pass the "-o cdp" option
> > to mount.
> >
> > rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> > with percentages, or with MB/sec values. Again the user already has to make this
> > decision when choosing mount options.
> >
> >
> > Maybe the "What resctrl options does this machine support?" question would be
> > best answered with a small utility?
>
> A user space utility or a kernel provided utility? If it is a user space utility
> I think it would end up needing to duplicate what the kernel is required to do
> to know if a particular feature is supported. It seems appropriate that this
> could be a kernel utility that can share this existing information with user
> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.

I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
all features, a utility has access to all the CPUID leaves that contain the
details of each feature enumeration.

> fyi ... as with previous attempts to discuss this work I find it difficult
> to discuss this work when you are selective about what you want to discuss/answer
> and just wipe the rest. Through this I understand that I am not your target
> audience.

Not my intent. I value your input highly. I'm maybe too avid a follower of the
"trim your replies" school of e-mail etiquette. I thought I'd covered the gist
of your message.

I'll try to be more thorough in responding in the future.

-Tony


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>> Alternatively, can user space just take a "load all resctrl modules
>>>> and see what sticks" (even modules of different architectures since
>>>> a user space may want to be generic) approach?
>>>
>>> This mostly works. Except for the cases where different modules access
>>> the same underlying hardware, so can't be loaded together.
>>>
>>> Examples:
>>>
>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>> to mount.
>>>
>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>> with percentages, or with MB/sec values. Again the user already has to make this
>>> decision when choosing mount options.
>>>
>>>
>>> Maybe the "What resctrl options does this machine support?" question would be
>>> best answered with a small utility?
>>
>> A user space utility or a kernel provided utility? If it is a user space utility
>> I think it would end up needing to duplicate what the kernel is required to do
>> to know if a particular feature is supported. It seems appropriate that this
>> could be a kernel utility that can share this existing information with user
>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> 
> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> all features, a utility has access to all the CPUID leaves that contain the
> details of each feature enumeration.

For x86 that may work (in some scenarios, see later) for now but as I understand
Arm would need a different solution where I believe the information is obtained
via ACPI. I think it is unnecessary to require user space to have parsers for
CPUID and ACPI if that same information needs to be parsed by the kernel and
there already exists an interface with which the information is communicated
from kernel to user space. Also, just because information CPUID shows a feature
is supported by the hardware does not mean that the kernel has support for that
feature. This could be because of a feature mismatch between user space and
kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
kernel parameter.

>> fyi ... as with previous attempts to discuss this work I find it difficult
>> to discuss this work when you are selective about what you want to discuss/answer
>> and just wipe the rest. Through this I understand that I am not your target
>> audience.
> 
> Not my intent. I value your input highly. I'm maybe too avid a follower of the
> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> of your message.
> 
> I'll try to be more thorough in responding in the future.

Two items from my previous email remain open:

First, why does making the code modular require everything to be loadable
modules?
I think that it is great that the code is modular. Ideally it will help to
support the other architectures. As you explain this modular design also
has the benefit that "modules" can be loaded and unloaded after resctrl mount.
Considering your example of MBA and MBA_MBps support ... if I understand
correctly with code being modular it enables changes from one to the other
after resctrl mount. User can start with MBA and then switch to MBA_MBps
without needing to unmount resctrl. What I do not understand is why does
the code being modular require everything to be modules? Why, for example,
could a user not interact with a resctrl file that enables the user to make
this switch from, for example, MBA to MBA_MBps? With this the existing
interfaces can remain to be respected, the existing mount parameters need
to remain anyway, while enabling future "more modular" usages.

Second, copied from my previous email, what is the plan to deal with current
users that just mount resctrl and expect to learn from it what features are
supported?

Reinette

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>> Alternatively, can user space just take a "load all resctrl modules
>>>> and see what sticks" (even modules of different architectures since
>>>> a user space may want to be generic) approach?
>>>
>>> This mostly works. Except for the cases where different modules access
>>> the same underlying hardware, so can't be loaded together.
>>>
>>> Examples:
>>>
>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>> to mount.
>>>
>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>> with percentages, or with MB/sec values. Again the user already has to make this
>>> decision when choosing mount options.
>>>
>>>
>>> Maybe the "What resctrl options does this machine support?" question would be
>>> best answered with a small utility?
>>
>> A user space utility or a kernel provided utility? If it is a user space utility
>> I think it would end up needing to duplicate what the kernel is required to do
>> to know if a particular feature is supported. It seems appropriate that this
>> could be a kernel utility that can share this existing information with user
>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> 
> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> all features, a utility has access to all the CPUID leaves that contain the
> details of each feature enumeration.

For x86 that may work (in some scenarios, see later) for now but as I understand
Arm would need a different solution where I believe the information is obtained
via ACPI. I think it is unnecessary to require user space to have parsers for
CPUID and ACPI if that same information needs to be parsed by the kernel and
there already exists an interface with which the information is communicated
from kernel to user space. Also, just because information CPUID shows a feature
is supported by the hardware does not mean that the kernel has support for that
feature. This could be because of a feature mismatch between user space and
kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
kernel parameter.

>> fyi ... as with previous attempts to discuss this work I find it difficult
>> to discuss this work when you are selective about what you want to discuss/answer
>> and just wipe the rest. Through this I understand that I am not your target
>> audience.
> 
> Not my intent. I value your input highly. I'm maybe too avid a follower of the
> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> of your message.
> 
> I'll try to be more thorough in responding in the future.

Two items from my previous email remain open:

First, why does making the code modular require everything to be loadable
modules?
I think that it is great that the code is modular. Ideally it will help to
support the other architectures. As you explain this modular design also
has the benefit that "modules" can be loaded and unloaded after resctrl mount.
Considering your example of MBA and MBA_MBps support ... if I understand
correctly with code being modular it enables changes from one to the other
after resctrl mount. User can start with MBA and then switch to MBA_MBps
without needing to unmount resctrl. What I do not understand is why does
the code being modular require everything to be modules? Why, for example,
could a user not interact with a resctrl file that enables the user to make
this switch from, for example, MBA to MBA_MBps? With this the existing
interfaces can remain to be respected, the existing mount parameters need
to remain anyway, while enabling future "more modular" usages.

Second, copied from my previous email, what is the plan to deal with current
users that just mount resctrl and expect to learn from it what features are
supported?

Reinette

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl modules
> >>>> and see what sticks" (even modules of different architectures since
> >>>> a user space may want to be generic) approach?
> >>>
> >>> This mostly works. Except for the cases where different modules access
> >>> the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>> to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>> with percentages, or with MB/sec values. Again the user already has to make this
> >>> decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" question would be
> >>> best answered with a small utility?
> >>
> >> A user space utility or a kernel provided utility? If it is a user space utility
> >> I think it would end up needing to duplicate what the kernel is required to do
> >> to know if a particular feature is supported. It seems appropriate that this
> >> could be a kernel utility that can share this existing information with user
> >> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> > all features, a utility has access to all the CPUID leaves that contain the
> > details of each feature enumeration.
> 
> For x86 that may work (in some scenarios, see later) for now but as I understand
> Arm would need a different solution where I believe the information is obtained
> via ACPI. I think it is unnecessary to require user space to have parsers for
> CPUID and ACPI if that same information needs to be parsed by the kernel and
> there already exists an interface with which the information is communicated
> from kernel to user space. Also, just because information CPUID shows a feature
> is supported by the hardware does not mean that the kernel has support for that
> feature. This could be because of a feature mismatch between user space and
> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> kernel parameter.

Agreed this is complex, and my initial resctrl2 proposal lacks
functionality in this area.

> >> fyi ... as with previous attempts to discuss this work I find it difficult
> >> to discuss this work when you are selective about what you want to discuss/answer
> >> and just wipe the rest. Through this I understand that I am not your target
> >> audience.
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a follower of the
> > "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> > of your message.
> > 
> > I'll try to be more thorough in responding in the future.
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be loadable
> modules?
> I think that it is great that the code is modular. Ideally it will help to
> support the other architectures. As you explain this modular design also
> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I understand
> correctly with code being modular it enables changes from one to the other
> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> without needing to unmount resctrl. What I do not understand is why does
> the code being modular require everything to be modules? Why, for example,
> could a user not interact with a resctrl file that enables the user to make
> this switch from, for example, MBA to MBA_MBps? With this the existing
> interfaces can remain to be respected, the existing mount parameters need
> to remain anyway, while enabling future "more modular" usages.

Lots of advantages to modules:
1) Only load what you need.
	- saves memory
	- reduces potential attack surface
	- may avoid periodic timers (e.g. for MBM overflow and
	  for LLC occupancy "limbo" mode).
2) If there is a security fix, can be deployed without a reboot.
3) Isolation between different features. 
	- Makes development and testing simpler

Sure some things like switching MBA to MBA_MBps mode by writing to
a control file are theoretically possible. But they would be far more
complex implementations with many possible oppurtunities for bugs.
I think Vikas made a good choice to make this a mount option rather
than selectable at run time.

> Second, copied from my previous email, what is the plan to deal with current
> users that just mount resctrl and expect to learn from it what features are
> supported?

Do such users exist? Resctrl is a sophisticated system management tool.
I'd expect system administrators deploying it are well aware of the
capabilities of the different types of systems in their data center.

But if I'm wrong, then I have to go back to figure out a way to
expose this information in a better way than randomly running "modprobe"
to see what sticks.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl modules
> >>>> and see what sticks" (even modules of different architectures since
> >>>> a user space may want to be generic) approach?
> >>>
> >>> This mostly works. Except for the cases where different modules access
> >>> the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>> to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>> with percentages, or with MB/sec values. Again the user already has to make this
> >>> decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" question would be
> >>> best answered with a small utility?
> >>
> >> A user space utility or a kernel provided utility? If it is a user space utility
> >> I think it would end up needing to duplicate what the kernel is required to do
> >> to know if a particular feature is supported. It seems appropriate that this
> >> could be a kernel utility that can share this existing information with user
> >> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> > all features, a utility has access to all the CPUID leaves that contain the
> > details of each feature enumeration.
> 
> For x86 that may work (in some scenarios, see later) for now but as I understand
> Arm would need a different solution where I believe the information is obtained
> via ACPI. I think it is unnecessary to require user space to have parsers for
> CPUID and ACPI if that same information needs to be parsed by the kernel and
> there already exists an interface with which the information is communicated
> from kernel to user space. Also, just because information CPUID shows a feature
> is supported by the hardware does not mean that the kernel has support for that
> feature. This could be because of a feature mismatch between user space and
> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> kernel parameter.

Agreed this is complex, and my initial resctrl2 proposal lacks
functionality in this area.

> >> fyi ... as with previous attempts to discuss this work I find it difficult
> >> to discuss this work when you are selective about what you want to discuss/answer
> >> and just wipe the rest. Through this I understand that I am not your target
> >> audience.
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a follower of the
> > "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> > of your message.
> > 
> > I'll try to be more thorough in responding in the future.
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be loadable
> modules?
> I think that it is great that the code is modular. Ideally it will help to
> support the other architectures. As you explain this modular design also
> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I understand
> correctly with code being modular it enables changes from one to the other
> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> without needing to unmount resctrl. What I do not understand is why does
> the code being modular require everything to be modules? Why, for example,
> could a user not interact with a resctrl file that enables the user to make
> this switch from, for example, MBA to MBA_MBps? With this the existing
> interfaces can remain to be respected, the existing mount parameters need
> to remain anyway, while enabling future "more modular" usages.

Lots of advantages to modules:
1) Only load what you need.
	- saves memory
	- reduces potential attack surface
	- may avoid periodic timers (e.g. for MBM overflow and
	  for LLC occupancy "limbo" mode).
2) If there is a security fix, can be deployed without a reboot.
3) Isolation between different features. 
	- Makes development and testing simpler

Sure some things like switching MBA to MBA_MBps mode by writing to
a control file are theoretically possible. But they would be far more
complex implementations with many possible oppurtunities for bugs.
I think Vikas made a good choice to make this a mount option rather
than selectable at run time.

> Second, copied from my previous email, what is the plan to deal with current
> users that just mount resctrl and expect to learn from it what features are
> supported?

Do such users exist? Resctrl is a sophisticated system management tool.
I'd expect system administrators deploying it are well aware of the
capabilities of the different types of systems in their data center.

But if I'm wrong, then I have to go back to figure out a way to
expose this information in a better way than randomly running "modprobe"
to see what sticks.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 1:54 PM, Tony Luck wrote:
> On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
>> On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>>>> Alternatively, can user space just take a "load all resctrl modules
>>>>>> and see what sticks" (even modules of different architectures since
>>>>>> a user space may want to be generic) approach?
>>>>>
>>>>> This mostly works. Except for the cases where different modules access
>>>>> the same underlying hardware, so can't be loaded together.
>>>>>
>>>>> Examples:
>>>>>
>>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>>>> to mount.
>>>>>
>>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>>>> with percentages, or with MB/sec values. Again the user already has to make this
>>>>> decision when choosing mount options.
>>>>>
>>>>>
>>>>> Maybe the "What resctrl options does this machine support?" question would be
>>>>> best answered with a small utility?
>>>>
>>>> A user space utility or a kernel provided utility? If it is a user space utility
>>>> I think it would end up needing to duplicate what the kernel is required to do
>>>> to know if a particular feature is supported. It seems appropriate that this
>>>> could be a kernel utility that can share this existing information with user
>>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
>>>
>>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
>>> all features, a utility has access to all the CPUID leaves that contain the
>>> details of each feature enumeration.
>>
>> For x86 that may work (in some scenarios, see later) for now but as I understand
>> Arm would need a different solution where I believe the information is obtained
>> via ACPI. I think it is unnecessary to require user space to have parsers for
>> CPUID and ACPI if that same information needs to be parsed by the kernel and
>> there already exists an interface with which the information is communicated
>> from kernel to user space. Also, just because information CPUID shows a feature
>> is supported by the hardware does not mean that the kernel has support for that
>> feature. This could be because of a feature mismatch between user space and
>> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
>> kernel parameter.
> 
> Agreed this is complex, and my initial resctrl2 proposal lacks
> functionality in this area.

Why is there a need to reinvent these parts?

>>>> fyi ... as with previous attempts to discuss this work I find it difficult
>>>> to discuss this work when you are selective about what you want to discuss/answer
>>>> and just wipe the rest. Through this I understand that I am not your target
>>>> audience.
>>>
>>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
>>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
>>> of your message.
>>>
>>> I'll try to be more thorough in responding in the future.
>>
>> Two items from my previous email remain open:
>>
>> First, why does making the code modular require everything to be loadable
>> modules?
>> I think that it is great that the code is modular. Ideally it will help to
>> support the other architectures. As you explain this modular design also
>> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
>> Considering your example of MBA and MBA_MBps support ... if I understand
>> correctly with code being modular it enables changes from one to the other
>> after resctrl mount. User can start with MBA and then switch to MBA_MBps
>> without needing to unmount resctrl. What I do not understand is why does
>> the code being modular require everything to be modules? Why, for example,
>> could a user not interact with a resctrl file that enables the user to make
>> this switch from, for example, MBA to MBA_MBps? With this the existing
>> interfaces can remain to be respected, the existing mount parameters need
>> to remain anyway, while enabling future "more modular" usages.
> 
> Lots of advantages to modules:
> 1) Only load what you need.
> 	- saves memory
> 	- reduces potential attack surface
> 	- may avoid periodic timers (e.g. for MBM overflow and
> 	  for LLC occupancy "limbo" mode).
> 2) If there is a security fix, can be deployed without a reboot.
> 3) Isolation between different features. 
> 	- Makes development and testing simpler
>

From what I understand (1) and (3) are accomplished through things
being modular. To transition smoothly it may be required for all
currently supported features to be loaded by default, with the
option to unload afterwards by user space that understands new
modular interfaces.

(2) does not need a module for each resource and feature supported
by resctrl. A single resctrl module would accomplish this and I
would expect it to be something everybody would like. James also
mentioned it being on his significant to-do list.
 
> Sure some things like switching MBA to MBA_MBps mode by writing to
> a control file are theoretically possible. But they would be far more
> complex implementations with many possible oppurtunities for bugs.
> I think Vikas made a good choice to make this a mount option rather
> than selectable at run time.
> 
>> Second, copied from my previous email, what is the plan to deal with current
>> users that just mount resctrl and expect to learn from it what features are
>> supported?
> 
> Do such users exist? Resctrl is a sophisticated system management tool.
> I'd expect system administrators deploying it are well aware of the
> capabilities of the different types of systems in their data center.
> 
> But if I'm wrong, then I have to go back to figure out a way to
> expose this information in a better way than randomly running "modprobe"
> to see what sticks.

I always start with intel-cmt-cat but I believe that the burden would be
on you to convince all that existing user space would not be impacted
by this change. If I understand correctly this implementation would
result in mounting resctrl to have an empty schemata and no resources
in the info directory. Without knowledge about how to enable resources
the user space could be expected to interpret that as no resources enabled
on the system.

Reinette

Re: resctrl2 - status

[Reinette Chatre]

Hi Tony,

On 8/25/2023 1:54 PM, Tony Luck wrote:
> On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
>> On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>>>> Alternatively, can user space just take a "load all resctrl modules
>>>>>> and see what sticks" (even modules of different architectures since
>>>>>> a user space may want to be generic) approach?
>>>>>
>>>>> This mostly works. Except for the cases where different modules access
>>>>> the same underlying hardware, so can't be loaded together.
>>>>>
>>>>> Examples:
>>>>>
>>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>>>> to mount.
>>>>>
>>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>>>> with percentages, or with MB/sec values. Again the user already has to make this
>>>>> decision when choosing mount options.
>>>>>
>>>>>
>>>>> Maybe the "What resctrl options does this machine support?" question would be
>>>>> best answered with a small utility?
>>>>
>>>> A user space utility or a kernel provided utility? If it is a user space utility
>>>> I think it would end up needing to duplicate what the kernel is required to do
>>>> to know if a particular feature is supported. It seems appropriate that this
>>>> could be a kernel utility that can share this existing information with user
>>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
>>>
>>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
>>> all features, a utility has access to all the CPUID leaves that contain the
>>> details of each feature enumeration.
>>
>> For x86 that may work (in some scenarios, see later) for now but as I understand
>> Arm would need a different solution where I believe the information is obtained
>> via ACPI. I think it is unnecessary to require user space to have parsers for
>> CPUID and ACPI if that same information needs to be parsed by the kernel and
>> there already exists an interface with which the information is communicated
>> from kernel to user space. Also, just because information CPUID shows a feature
>> is supported by the hardware does not mean that the kernel has support for that
>> feature. This could be because of a feature mismatch between user space and
>> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
>> kernel parameter.
> 
> Agreed this is complex, and my initial resctrl2 proposal lacks
> functionality in this area.

Why is there a need to reinvent these parts?

>>>> fyi ... as with previous attempts to discuss this work I find it difficult
>>>> to discuss this work when you are selective about what you want to discuss/answer
>>>> and just wipe the rest. Through this I understand that I am not your target
>>>> audience.
>>>
>>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
>>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
>>> of your message.
>>>
>>> I'll try to be more thorough in responding in the future.
>>
>> Two items from my previous email remain open:
>>
>> First, why does making the code modular require everything to be loadable
>> modules?
>> I think that it is great that the code is modular. Ideally it will help to
>> support the other architectures. As you explain this modular design also
>> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
>> Considering your example of MBA and MBA_MBps support ... if I understand
>> correctly with code being modular it enables changes from one to the other
>> after resctrl mount. User can start with MBA and then switch to MBA_MBps
>> without needing to unmount resctrl. What I do not understand is why does
>> the code being modular require everything to be modules? Why, for example,
>> could a user not interact with a resctrl file that enables the user to make
>> this switch from, for example, MBA to MBA_MBps? With this the existing
>> interfaces can remain to be respected, the existing mount parameters need
>> to remain anyway, while enabling future "more modular" usages.
> 
> Lots of advantages to modules:
> 1) Only load what you need.
> 	- saves memory
> 	- reduces potential attack surface
> 	- may avoid periodic timers (e.g. for MBM overflow and
> 	  for LLC occupancy "limbo" mode).
> 2) If there is a security fix, can be deployed without a reboot.
> 3) Isolation between different features. 
> 	- Makes development and testing simpler
>

From what I understand (1) and (3) are accomplished through things
being modular. To transition smoothly it may be required for all
currently supported features to be loaded by default, with the
option to unload afterwards by user space that understands new
modular interfaces.

(2) does not need a module for each resource and feature supported
by resctrl. A single resctrl module would accomplish this and I
would expect it to be something everybody would like. James also
mentioned it being on his significant to-do list.
 
> Sure some things like switching MBA to MBA_MBps mode by writing to
> a control file are theoretically possible. But they would be far more
> complex implementations with many possible oppurtunities for bugs.
> I think Vikas made a good choice to make this a mount option rather
> than selectable at run time.
> 
>> Second, copied from my previous email, what is the plan to deal with current
>> users that just mount resctrl and expect to learn from it what features are
>> supported?
> 
> Do such users exist? Resctrl is a sophisticated system management tool.
> I'd expect system administrators deploying it are well aware of the
> capabilities of the different types of systems in their data center.
> 
> But if I'm wrong, then I have to go back to figure out a way to
> expose this information in a better way than randomly running "modprobe"
> to see what sticks.

I always start with intel-cmt-cat but I believe that the burden would be
on you to convince all that existing user space would not be impacted
by this change. If I understand correctly this implementation would
result in mounting resctrl to have an empty schemata and no resources
in the info directory. Without knowledge about how to enable resources
the user space could be expected to interpret that as no resources enabled
on the system.

Reinette


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Fri, Aug 25, 2023 at 04:08:21PM -0700, Reinette Chatre wrote:
> Hi Tony,
> 
> On 8/25/2023 1:54 PM, Tony Luck wrote:
> > On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
> >> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>>>> Alternatively, can user space just take a "load all resctrl modules
> >>>>>> and see what sticks" (even modules of different architectures since
> >>>>>> a user space may want to be generic) approach?
> >>>>>
> >>>>> This mostly works. Except for the cases where different modules access
> >>>>> the same underlying hardware, so can't be loaded together.
> >>>>>
> >>>>> Examples:
> >>>>>
> >>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>>>> to mount.
> >>>>>
> >>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>>>> with percentages, or with MB/sec values. Again the user already has to make this
> >>>>> decision when choosing mount options.
> >>>>>
> >>>>>
> >>>>> Maybe the "What resctrl options does this machine support?" question would be
> >>>>> best answered with a small utility?
> >>>>
> >>>> A user space utility or a kernel provided utility? If it is a user space utility
> >>>> I think it would end up needing to duplicate what the kernel is required to do
> >>>> to know if a particular feature is supported. It seems appropriate that this
> >>>> could be a kernel utility that can share this existing information with user
> >>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> >>>
> >>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> >>> all features, a utility has access to all the CPUID leaves that contain the
> >>> details of each feature enumeration.
> >>
> >> For x86 that may work (in some scenarios, see later) for now but as I understand
> >> Arm would need a different solution where I believe the information is obtained
> >> via ACPI. I think it is unnecessary to require user space to have parsers for
> >> CPUID and ACPI if that same information needs to be parsed by the kernel and
> >> there already exists an interface with which the information is communicated
> >> from kernel to user space. Also, just because information CPUID shows a feature
> >> is supported by the hardware does not mean that the kernel has support for that
> >> feature. This could be because of a feature mismatch between user space and
> >> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> >> kernel parameter.
> > 
> > Agreed this is complex, and my initial resctrl2 proposal lacks
> > functionality in this area.
> 
> Why is there a need to reinvent these parts?

Perhaps there isn't ... see below.

> 
> >>>> fyi ... as with previous attempts to discuss this work I find it difficult
> >>>> to discuss this work when you are selective about what you want to discuss/answer
> >>>> and just wipe the rest. Through this I understand that I am not your target
> >>>> audience.
> >>>
> >>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
> >>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> >>> of your message.
> >>>
> >>> I'll try to be more thorough in responding in the future.
> >>
> >> Two items from my previous email remain open:
> >>
> >> First, why does making the code modular require everything to be loadable
> >> modules?
> >> I think that it is great that the code is modular. Ideally it will help to
> >> support the other architectures. As you explain this modular design also
> >> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> >> Considering your example of MBA and MBA_MBps support ... if I understand
> >> correctly with code being modular it enables changes from one to the other
> >> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> >> without needing to unmount resctrl. What I do not understand is why does
> >> the code being modular require everything to be modules? Why, for example,
> >> could a user not interact with a resctrl file that enables the user to make
> >> this switch from, for example, MBA to MBA_MBps? With this the existing
> >> interfaces can remain to be respected, the existing mount parameters need
> >> to remain anyway, while enabling future "more modular" usages.
> > 
> > Lots of advantages to modules:
> > 1) Only load what you need.
> > 	- saves memory
> > 	- reduces potential attack surface
> > 	- may avoid periodic timers (e.g. for MBM overflow and
> > 	  for LLC occupancy "limbo" mode).
> > 2) If there is a security fix, can be deployed without a reboot.
> > 3) Isolation between different features. 
> > 	- Makes development and testing simpler
> >
> 
> From what I understand (1) and (3) are accomplished through things
> being modular. To transition smoothly it may be required for all
> currently supported features to be loaded by default, with the
> option to unload afterwards by user space that understands new
> modular interfaces.
> 
> (2) does not need a module for each resource and feature supported
> by resctrl. A single resctrl module would accomplish this and I
> would expect it to be something everybody would like. James also
> mentioned it being on his significant to-do list.
>  
> > Sure some things like switching MBA to MBA_MBps mode by writing to
> > a control file are theoretically possible. But they would be far more
> > complex implementations with many possible oppurtunities for bugs.
> > I think Vikas made a good choice to make this a mount option rather
> > than selectable at run time.
> > 
> >> Second, copied from my previous email, what is the plan to deal with current
> >> users that just mount resctrl and expect to learn from it what features are
> >> supported?
> > 
> > Do such users exist? Resctrl is a sophisticated system management tool.
> > I'd expect system administrators deploying it are well aware of the
> > capabilities of the different types of systems in their data center.
> > 
> > But if I'm wrong, then I have to go back to figure out a way to
> > expose this information in a better way than randomly running "modprobe"
> > to see what sticks.
> 
> I always start with intel-cmt-cat but I believe that the burden would be
> on you to convince all that existing user space would not be impacted
> by this change. If I understand correctly this implementation would
> result in mounting resctrl to have an empty schemata and no resources
> in the info directory. Without knowledge about how to enable resources
> the user space could be expected to interpret that as no resources enabled
> on the system.

Reinette,

The basic issue is that my module based system has become less user
friendly. Requiring extra steps to get basic things works.

Luckily there is a simple solution. Make the modules for the basic
functions autoload.  E.g. for MBA:

+static const struct x86_cpu_id mba_feature[] = {
+       X86_MATCH_FEATURE(X86_FEATURE_MBA, 0),
+       { }
+};
+MODULE_DEVICE_TABLE(x86cpu, mba_feature);

Then immediately after booting the system looks like this:

$ lsmod | grep rdt
rdt_l3_mba             16384  0
rdt_mbm_local_bytes    12288  0
rdt_mbm_total_bytes    12288  0
rdt_llc_occupancy      12288  0
rdt_l3_cat             16384  0

And mounting resctrl:

$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ tree /sys/fs/resctrl/info
/sys/fs/resctrl/info
├── L3
│   ├── bit_usage
│   ├── cbm_mask
│   ├── min_cbm_bits
│   ├── num_closids
│   └── shareable_bits
├── L3_MON
│   ├── max_threshold_occupancy
│   ├── mbm_poll_threshold
│   ├── mon_features
│   └── num_rmids
├── last_cmd_status
└── MB
    ├── bandwidth_gran
    ├── delay_linear
    ├── min_bandwidth
    └── num_closids

3 directories, 14 files
$ cat /sys/fs/resctrl/schemata
MB: 0=0;1=0
L3: 0=fff;1=fff

Thanks for pushing me to search for this solution to make things
more compatible.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Fri, Aug 25, 2023 at 04:08:21PM -0700, Reinette Chatre wrote:
> Hi Tony,
> 
> On 8/25/2023 1:54 PM, Tony Luck wrote:
> > On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
> >> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>>>> Alternatively, can user space just take a "load all resctrl modules
> >>>>>> and see what sticks" (even modules of different architectures since
> >>>>>> a user space may want to be generic) approach?
> >>>>>
> >>>>> This mostly works. Except for the cases where different modules access
> >>>>> the same underlying hardware, so can't be loaded together.
> >>>>>
> >>>>> Examples:
> >>>>>
> >>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>>>> to mount.
> >>>>>
> >>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>>>> with percentages, or with MB/sec values. Again the user already has to make this
> >>>>> decision when choosing mount options.
> >>>>>
> >>>>>
> >>>>> Maybe the "What resctrl options does this machine support?" question would be
> >>>>> best answered with a small utility?
> >>>>
> >>>> A user space utility or a kernel provided utility? If it is a user space utility
> >>>> I think it would end up needing to duplicate what the kernel is required to do
> >>>> to know if a particular feature is supported. It seems appropriate that this
> >>>> could be a kernel utility that can share this existing information with user
> >>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> >>>
> >>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> >>> all features, a utility has access to all the CPUID leaves that contain the
> >>> details of each feature enumeration.
> >>
> >> For x86 that may work (in some scenarios, see later) for now but as I understand
> >> Arm would need a different solution where I believe the information is obtained
> >> via ACPI. I think it is unnecessary to require user space to have parsers for
> >> CPUID and ACPI if that same information needs to be parsed by the kernel and
> >> there already exists an interface with which the information is communicated
> >> from kernel to user space. Also, just because information CPUID shows a feature
> >> is supported by the hardware does not mean that the kernel has support for that
> >> feature. This could be because of a feature mismatch between user space and
> >> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> >> kernel parameter.
> > 
> > Agreed this is complex, and my initial resctrl2 proposal lacks
> > functionality in this area.
> 
> Why is there a need to reinvent these parts?

Perhaps there isn't ... see below.

> 
> >>>> fyi ... as with previous attempts to discuss this work I find it difficult
> >>>> to discuss this work when you are selective about what you want to discuss/answer
> >>>> and just wipe the rest. Through this I understand that I am not your target
> >>>> audience.
> >>>
> >>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
> >>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> >>> of your message.
> >>>
> >>> I'll try to be more thorough in responding in the future.
> >>
> >> Two items from my previous email remain open:
> >>
> >> First, why does making the code modular require everything to be loadable
> >> modules?
> >> I think that it is great that the code is modular. Ideally it will help to
> >> support the other architectures. As you explain this modular design also
> >> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> >> Considering your example of MBA and MBA_MBps support ... if I understand
> >> correctly with code being modular it enables changes from one to the other
> >> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> >> without needing to unmount resctrl. What I do not understand is why does
> >> the code being modular require everything to be modules? Why, for example,
> >> could a user not interact with a resctrl file that enables the user to make
> >> this switch from, for example, MBA to MBA_MBps? With this the existing
> >> interfaces can remain to be respected, the existing mount parameters need
> >> to remain anyway, while enabling future "more modular" usages.
> > 
> > Lots of advantages to modules:
> > 1) Only load what you need.
> > 	- saves memory
> > 	- reduces potential attack surface
> > 	- may avoid periodic timers (e.g. for MBM overflow and
> > 	  for LLC occupancy "limbo" mode).
> > 2) If there is a security fix, can be deployed without a reboot.
> > 3) Isolation between different features. 
> > 	- Makes development and testing simpler
> >
> 
> From what I understand (1) and (3) are accomplished through things
> being modular. To transition smoothly it may be required for all
> currently supported features to be loaded by default, with the
> option to unload afterwards by user space that understands new
> modular interfaces.
> 
> (2) does not need a module for each resource and feature supported
> by resctrl. A single resctrl module would accomplish this and I
> would expect it to be something everybody would like. James also
> mentioned it being on his significant to-do list.
>  
> > Sure some things like switching MBA to MBA_MBps mode by writing to
> > a control file are theoretically possible. But they would be far more
> > complex implementations with many possible oppurtunities for bugs.
> > I think Vikas made a good choice to make this a mount option rather
> > than selectable at run time.
> > 
> >> Second, copied from my previous email, what is the plan to deal with current
> >> users that just mount resctrl and expect to learn from it what features are
> >> supported?
> > 
> > Do such users exist? Resctrl is a sophisticated system management tool.
> > I'd expect system administrators deploying it are well aware of the
> > capabilities of the different types of systems in their data center.
> > 
> > But if I'm wrong, then I have to go back to figure out a way to
> > expose this information in a better way than randomly running "modprobe"
> > to see what sticks.
> 
> I always start with intel-cmt-cat but I believe that the burden would be
> on you to convince all that existing user space would not be impacted
> by this change. If I understand correctly this implementation would
> result in mounting resctrl to have an empty schemata and no resources
> in the info directory. Without knowledge about how to enable resources
> the user space could be expected to interpret that as no resources enabled
> on the system.

Reinette,

The basic issue is that my module based system has become less user
friendly. Requiring extra steps to get basic things works.

Luckily there is a simple solution. Make the modules for the basic
functions autoload.  E.g. for MBA:

+static const struct x86_cpu_id mba_feature[] = {
+       X86_MATCH_FEATURE(X86_FEATURE_MBA, 0),
+       { }
+};
+MODULE_DEVICE_TABLE(x86cpu, mba_feature);

Then immediately after booting the system looks like this:

$ lsmod | grep rdt
rdt_l3_mba             16384  0
rdt_mbm_local_bytes    12288  0
rdt_mbm_total_bytes    12288  0
rdt_llc_occupancy      12288  0
rdt_l3_cat             16384  0

And mounting resctrl:

$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ tree /sys/fs/resctrl/info
/sys/fs/resctrl/info
├── L3
│   ├── bit_usage
│   ├── cbm_mask
│   ├── min_cbm_bits
│   ├── num_closids
│   └── shareable_bits
├── L3_MON
│   ├── max_threshold_occupancy
│   ├── mbm_poll_threshold
│   ├── mon_features
│   └── num_rmids
├── last_cmd_status
└── MB
    ├── bandwidth_gran
    ├── delay_linear
    ├── min_bandwidth
    └── num_closids

3 directories, 14 files
$ cat /sys/fs/resctrl/schemata
MB: 0=0;1=0
L3: 0=fff;1=fff

Thanks for pushing me to search for this solution to make things
more compatible.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Reinette Chatre]

On 8/25/2023 6:11 PM, Tony Luck wrote:
> On Fri, Aug 25, 2023 at 04:08:21PM -0700, Reinette Chatre wrote:
>> Hi Tony,
>>
>> On 8/25/2023 1:54 PM, Tony Luck wrote:
>>> On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
>>>> On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>>>>>> Alternatively, can user space just take a "load all resctrl modules
>>>>>>>> and see what sticks" (even modules of different architectures since
>>>>>>>> a user space may want to be generic) approach?
>>>>>>>
>>>>>>> This mostly works. Except for the cases where different modules access
>>>>>>> the same underlying hardware, so can't be loaded together.
>>>>>>>
>>>>>>> Examples:
>>>>>>>
>>>>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>>>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>>>>>> to mount.
>>>>>>>
>>>>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>>>>>> with percentages, or with MB/sec values. Again the user already has to make this
>>>>>>> decision when choosing mount options.
>>>>>>>
>>>>>>>
>>>>>>> Maybe the "What resctrl options does this machine support?" question would be
>>>>>>> best answered with a small utility?
>>>>>>
>>>>>> A user space utility or a kernel provided utility? If it is a user space utility
>>>>>> I think it would end up needing to duplicate what the kernel is required to do
>>>>>> to know if a particular feature is supported. It seems appropriate that this
>>>>>> could be a kernel utility that can share this existing information with user
>>>>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
>>>>>
>>>>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
>>>>> all features, a utility has access to all the CPUID leaves that contain the
>>>>> details of each feature enumeration.
>>>>
>>>> For x86 that may work (in some scenarios, see later) for now but as I understand
>>>> Arm would need a different solution where I believe the information is obtained
>>>> via ACPI. I think it is unnecessary to require user space to have parsers for
>>>> CPUID and ACPI if that same information needs to be parsed by the kernel and
>>>> there already exists an interface with which the information is communicated
>>>> from kernel to user space. Also, just because information CPUID shows a feature
>>>> is supported by the hardware does not mean that the kernel has support for that
>>>> feature. This could be because of a feature mismatch between user space and
>>>> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
>>>> kernel parameter.
>>>
>>> Agreed this is complex, and my initial resctrl2 proposal lacks
>>> functionality in this area.
>>
>> Why is there a need to reinvent these parts?
> 
> Perhaps there isn't ... see below.
> 
>>
>>>>>> fyi ... as with previous attempts to discuss this work I find it difficult
>>>>>> to discuss this work when you are selective about what you want to discuss/answer
>>>>>> and just wipe the rest. Through this I understand that I am not your target
>>>>>> audience.
>>>>>
>>>>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
>>>>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
>>>>> of your message.
>>>>>
>>>>> I'll try to be more thorough in responding in the future.
>>>>
>>>> Two items from my previous email remain open:
>>>>
>>>> First, why does making the code modular require everything to be loadable
>>>> modules?
>>>> I think that it is great that the code is modular. Ideally it will help to
>>>> support the other architectures. As you explain this modular design also
>>>> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
>>>> Considering your example of MBA and MBA_MBps support ... if I understand
>>>> correctly with code being modular it enables changes from one to the other
>>>> after resctrl mount. User can start with MBA and then switch to MBA_MBps
>>>> without needing to unmount resctrl. What I do not understand is why does
>>>> the code being modular require everything to be modules? Why, for example,
>>>> could a user not interact with a resctrl file that enables the user to make
>>>> this switch from, for example, MBA to MBA_MBps? With this the existing
>>>> interfaces can remain to be respected, the existing mount parameters need
>>>> to remain anyway, while enabling future "more modular" usages.
>>>
>>> Lots of advantages to modules:
>>> 1) Only load what you need.
>>> 	- saves memory
>>> 	- reduces potential attack surface
>>> 	- may avoid periodic timers (e.g. for MBM overflow and
>>> 	  for LLC occupancy "limbo" mode).
>>> 2) If there is a security fix, can be deployed without a reboot.
>>> 3) Isolation between different features. 
>>> 	- Makes development and testing simpler
>>>
>>
>> From what I understand (1) and (3) are accomplished through things
>> being modular. To transition smoothly it may be required for all
>> currently supported features to be loaded by default, with the
>> option to unload afterwards by user space that understands new
>> modular interfaces.
>>
>> (2) does not need a module for each resource and feature supported
>> by resctrl. A single resctrl module would accomplish this and I
>> would expect it to be something everybody would like. James also
>> mentioned it being on his significant to-do list.
>>  
>>> Sure some things like switching MBA to MBA_MBps mode by writing to
>>> a control file are theoretically possible. But they would be far more
>>> complex implementations with many possible oppurtunities for bugs.
>>> I think Vikas made a good choice to make this a mount option rather
>>> than selectable at run time.
>>>
>>>> Second, copied from my previous email, what is the plan to deal with current
>>>> users that just mount resctrl and expect to learn from it what features are
>>>> supported?
>>>
>>> Do such users exist? Resctrl is a sophisticated system management tool.
>>> I'd expect system administrators deploying it are well aware of the
>>> capabilities of the different types of systems in their data center.
>>>
>>> But if I'm wrong, then I have to go back to figure out a way to
>>> expose this information in a better way than randomly running "modprobe"
>>> to see what sticks.
>>
>> I always start with intel-cmt-cat but I believe that the burden would be
>> on you to convince all that existing user space would not be impacted
>> by this change. If I understand correctly this implementation would
>> result in mounting resctrl to have an empty schemata and no resources
>> in the info directory. Without knowledge about how to enable resources
>> the user space could be expected to interpret that as no resources enabled
>> on the system.
> 
> Reinette,
> 
> The basic issue is that my module based system has become less user
> friendly. Requiring extra steps to get basic things works.
> 
> Luckily there is a simple solution. Make the modules for the basic
> functions autoload.  E.g. for MBA:
> 
> +static const struct x86_cpu_id mba_feature[] = {
> +       X86_MATCH_FEATURE(X86_FEATURE_MBA, 0),
> +       { }
> +};
> +MODULE_DEVICE_TABLE(x86cpu, mba_feature);
> 
> Then immediately after booting the system looks like this:
> 
> $ lsmod | grep rdt
> rdt_l3_mba             16384  0
> rdt_mbm_local_bytes    12288  0
> rdt_mbm_total_bytes    12288  0
> rdt_llc_occupancy      12288  0
> rdt_l3_cat             16384  0
> 
> And mounting resctrl:
> 
> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> $ tree /sys/fs/resctrl/info
> /sys/fs/resctrl/info
> ├── L3
> │   ├── bit_usage
> │   ├── cbm_mask
> │   ├── min_cbm_bits
> │   ├── num_closids
> │   └── shareable_bits
> ├── L3_MON
> │   ├── max_threshold_occupancy
> │   ├── mbm_poll_threshold
> │   ├── mon_features
> │   └── num_rmids
> ├── last_cmd_status
> └── MB
>     ├── bandwidth_gran
>     ├── delay_linear
>     ├── min_bandwidth
>     └── num_closids
> 
> 3 directories, 14 files
> $ cat /sys/fs/resctrl/schemata
> MB: 0=0;1=0
> L3: 0=fff;1=fff
> 
> Thanks for pushing me to search for this solution to make things
> more compatible.

heh ... sounds familiar to "To transition smoothly it may be required
for all currently supported features to be loaded by default". It is not
obvious to me how this also closes the other opens.

Reinette

Re: resctrl2 - status

[Reinette Chatre]

On 8/25/2023 6:11 PM, Tony Luck wrote:
> On Fri, Aug 25, 2023 at 04:08:21PM -0700, Reinette Chatre wrote:
>> Hi Tony,
>>
>> On 8/25/2023 1:54 PM, Tony Luck wrote:
>>> On Fri, Aug 25, 2023 at 01:20:22PM -0700, Reinette Chatre wrote:
>>>> On 8/25/2023 12:44 PM, Luck, Tony wrote:
>>>>>>>> Alternatively, can user space just take a "load all resctrl modules
>>>>>>>> and see what sticks" (even modules of different architectures since
>>>>>>>> a user space may want to be generic) approach?
>>>>>>>
>>>>>>> This mostly works. Except for the cases where different modules access
>>>>>>> the same underlying hardware, so can't be loaded together.
>>>>>>>
>>>>>>> Examples:
>>>>>>>
>>>>>>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
>>>>>>> But this is already true ... they have to decide whether to pass the "-o cdp" option
>>>>>>> to mount.
>>>>>>>
>>>>>>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
>>>>>>> with percentages, or with MB/sec values. Again the user already has to make this
>>>>>>> decision when choosing mount options.
>>>>>>>
>>>>>>>
>>>>>>> Maybe the "What resctrl options does this machine support?" question would be
>>>>>>> best answered with a small utility?
>>>>>>
>>>>>> A user space utility or a kernel provided utility? If it is a user space utility
>>>>>> I think it would end up needing to duplicate what the kernel is required to do
>>>>>> to know if a particular feature is supported. It seems appropriate that this
>>>>>> could be a kernel utility that can share this existing information with user
>>>>>> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
>>>>>
>>>>> I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
>>>>> all features, a utility has access to all the CPUID leaves that contain the
>>>>> details of each feature enumeration.
>>>>
>>>> For x86 that may work (in some scenarios, see later) for now but as I understand
>>>> Arm would need a different solution where I believe the information is obtained
>>>> via ACPI. I think it is unnecessary to require user space to have parsers for
>>>> CPUID and ACPI if that same information needs to be parsed by the kernel and
>>>> there already exists an interface with which the information is communicated
>>>> from kernel to user space. Also, just because information CPUID shows a feature
>>>> is supported by the hardware does not mean that the kernel has support for that
>>>> feature. This could be because of a feature mismatch between user space and
>>>> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
>>>> kernel parameter.
>>>
>>> Agreed this is complex, and my initial resctrl2 proposal lacks
>>> functionality in this area.
>>
>> Why is there a need to reinvent these parts?
> 
> Perhaps there isn't ... see below.
> 
>>
>>>>>> fyi ... as with previous attempts to discuss this work I find it difficult
>>>>>> to discuss this work when you are selective about what you want to discuss/answer
>>>>>> and just wipe the rest. Through this I understand that I am not your target
>>>>>> audience.
>>>>>
>>>>> Not my intent. I value your input highly. I'm maybe too avid a follower of the
>>>>> "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
>>>>> of your message.
>>>>>
>>>>> I'll try to be more thorough in responding in the future.
>>>>
>>>> Two items from my previous email remain open:
>>>>
>>>> First, why does making the code modular require everything to be loadable
>>>> modules?
>>>> I think that it is great that the code is modular. Ideally it will help to
>>>> support the other architectures. As you explain this modular design also
>>>> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
>>>> Considering your example of MBA and MBA_MBps support ... if I understand
>>>> correctly with code being modular it enables changes from one to the other
>>>> after resctrl mount. User can start with MBA and then switch to MBA_MBps
>>>> without needing to unmount resctrl. What I do not understand is why does
>>>> the code being modular require everything to be modules? Why, for example,
>>>> could a user not interact with a resctrl file that enables the user to make
>>>> this switch from, for example, MBA to MBA_MBps? With this the existing
>>>> interfaces can remain to be respected, the existing mount parameters need
>>>> to remain anyway, while enabling future "more modular" usages.
>>>
>>> Lots of advantages to modules:
>>> 1) Only load what you need.
>>> 	- saves memory
>>> 	- reduces potential attack surface
>>> 	- may avoid periodic timers (e.g. for MBM overflow and
>>> 	  for LLC occupancy "limbo" mode).
>>> 2) If there is a security fix, can be deployed without a reboot.
>>> 3) Isolation between different features. 
>>> 	- Makes development and testing simpler
>>>
>>
>> From what I understand (1) and (3) are accomplished through things
>> being modular. To transition smoothly it may be required for all
>> currently supported features to be loaded by default, with the
>> option to unload afterwards by user space that understands new
>> modular interfaces.
>>
>> (2) does not need a module for each resource and feature supported
>> by resctrl. A single resctrl module would accomplish this and I
>> would expect it to be something everybody would like. James also
>> mentioned it being on his significant to-do list.
>>  
>>> Sure some things like switching MBA to MBA_MBps mode by writing to
>>> a control file are theoretically possible. But they would be far more
>>> complex implementations with many possible oppurtunities for bugs.
>>> I think Vikas made a good choice to make this a mount option rather
>>> than selectable at run time.
>>>
>>>> Second, copied from my previous email, what is the plan to deal with current
>>>> users that just mount resctrl and expect to learn from it what features are
>>>> supported?
>>>
>>> Do such users exist? Resctrl is a sophisticated system management tool.
>>> I'd expect system administrators deploying it are well aware of the
>>> capabilities of the different types of systems in their data center.
>>>
>>> But if I'm wrong, then I have to go back to figure out a way to
>>> expose this information in a better way than randomly running "modprobe"
>>> to see what sticks.
>>
>> I always start with intel-cmt-cat but I believe that the burden would be
>> on you to convince all that existing user space would not be impacted
>> by this change. If I understand correctly this implementation would
>> result in mounting resctrl to have an empty schemata and no resources
>> in the info directory. Without knowledge about how to enable resources
>> the user space could be expected to interpret that as no resources enabled
>> on the system.
> 
> Reinette,
> 
> The basic issue is that my module based system has become less user
> friendly. Requiring extra steps to get basic things works.
> 
> Luckily there is a simple solution. Make the modules for the basic
> functions autoload.  E.g. for MBA:
> 
> +static const struct x86_cpu_id mba_feature[] = {
> +       X86_MATCH_FEATURE(X86_FEATURE_MBA, 0),
> +       { }
> +};
> +MODULE_DEVICE_TABLE(x86cpu, mba_feature);
> 
> Then immediately after booting the system looks like this:
> 
> $ lsmod | grep rdt
> rdt_l3_mba             16384  0
> rdt_mbm_local_bytes    12288  0
> rdt_mbm_total_bytes    12288  0
> rdt_llc_occupancy      12288  0
> rdt_l3_cat             16384  0
> 
> And mounting resctrl:
> 
> $ sudo mount -t resctrl resctrl /sys/fs/resctrl
> $ tree /sys/fs/resctrl/info
> /sys/fs/resctrl/info
> ├── L3
> │   ├── bit_usage
> │   ├── cbm_mask
> │   ├── min_cbm_bits
> │   ├── num_closids
> │   └── shareable_bits
> ├── L3_MON
> │   ├── max_threshold_occupancy
> │   ├── mbm_poll_threshold
> │   ├── mon_features
> │   └── num_rmids
> ├── last_cmd_status
> └── MB
>     ├── bandwidth_gran
>     ├── delay_linear
>     ├── min_bandwidth
>     └── num_closids
> 
> 3 directories, 14 files
> $ cat /sys/fs/resctrl/schemata
> MB: 0=0;1=0
> L3: 0=fff;1=fff
> 
> Thanks for pushing me to search for this solution to make things
> more compatible.

heh ... sounds familiar to "To transition smoothly it may be required
for all currently supported features to be loaded by default". It is not
obvious to me how this also closes the other opens.

Reinette

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Jonathan Cameron]

On Fri, 25 Aug 2023 13:20:22 -0700
Reinette Chatre <reinette.chatre@intel.com> wrote:

> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl modules
> >>>> and see what sticks" (even modules of different architectures since
> >>>> a user space may want to be generic) approach?  
> >>>
> >>> This mostly works. Except for the cases where different modules access
> >>> the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>> to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>> with percentages, or with MB/sec values. Again the user already has to make this
> >>> decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" question would be
> >>> best answered with a small utility?  
> >>
> >> A user space utility or a kernel provided utility? If it is a user space utility
> >> I think it would end up needing to duplicate what the kernel is required to do
> >> to know if a particular feature is supported. It seems appropriate that this
> >> could be a kernel utility that can share this existing information with user
> >> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.  
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> > all features, a utility has access to all the CPUID leaves that contain the
> > details of each feature enumeration.  
> 
> For x86 that may work (in some scenarios, see later) for now but as I understand
> Arm would need a different solution where I believe the information is obtained
> via ACPI. I think it is unnecessary to require user space to have parsers for
> CPUID and ACPI if that same information needs to be parsed by the kernel and
> there already exists an interface with which the information is communicated
> from kernel to user space. Also, just because information CPUID shows a feature
> is supported by the hardware does not mean that the kernel has support for that
> feature. This could be because of a feature mismatch between user space and
> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> kernel parameter.

James probably also has views on this, but I wouldn't expect userspace to go anywhere
near ACPI parsing. If the description of what MPAM features are supported (before
resctrl is running) is necessary, then we should add it to the existing
description of the various caches etc.  Maybe somewhere in
/sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
for the caches and
/sys/bus/node/devices/nodeX for the DRAM controllers.

I'm sure we can find somewhere sensible for other things covered by MPAM.
The MPAM table parsers and some early code to query features from the hardware should
be able to figure out what is needed.

Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.

Jonathan

> 
> >> fyi ... as with previous attempts to discuss this work I find it difficult
> >> to discuss this work when you are selective about what you want to discuss/answer
> >> and just wipe the rest. Through this I understand that I am not your target
> >> audience.  
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a follower of the
> > "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> > of your message.
> > 
> > I'll try to be more thorough in responding in the future.  
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be loadable
> modules?
> I think that it is great that the code is modular. Ideally it will help to
> support the other architectures. As you explain this modular design also
> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I understand
> correctly with code being modular it enables changes from one to the other
> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> without needing to unmount resctrl. What I do not understand is why does
> the code being modular require everything to be modules? Why, for example,
> could a user not interact with a resctrl file that enables the user to make
> this switch from, for example, MBA to MBA_MBps? With this the existing
> interfaces can remain to be respected, the existing mount parameters need
> to remain anyway, while enabling future "more modular" usages.
> 
> Second, copied from my previous email, what is the plan to deal with current
> users that just mount resctrl and expect to learn from it what features are
> supported?
> 
> Reinette
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

Re: resctrl2 - status

[Jonathan Cameron]

On Fri, 25 Aug 2023 13:20:22 -0700
Reinette Chatre <reinette.chatre@intel.com> wrote:

> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl modules
> >>>> and see what sticks" (even modules of different architectures since
> >>>> a user space may want to be generic) approach?  
> >>>
> >>> This mostly works. Except for the cases where different modules access
> >>> the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass the "-o cdp" option
> >>> to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control memory bandwidth
> >>> with percentages, or with MB/sec values. Again the user already has to make this
> >>> decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" question would be
> >>> best answered with a small utility?  
> >>
> >> A user space utility or a kernel provided utility? If it is a user space utility
> >> I think it would end up needing to duplicate what the kernel is required to do
> >> to know if a particular feature is supported. It seems appropriate that this
> >> could be a kernel utility that can share this existing information with user
> >> space. resctrl already supports the interface for this via /sys/fs/resctrl/info.  
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo doesn't show
> > all features, a utility has access to all the CPUID leaves that contain the
> > details of each feature enumeration.  
> 
> For x86 that may work (in some scenarios, see later) for now but as I understand
> Arm would need a different solution where I believe the information is obtained
> via ACPI. I think it is unnecessary to require user space to have parsers for
> CPUID and ACPI if that same information needs to be parsed by the kernel and
> there already exists an interface with which the information is communicated
> from kernel to user space. Also, just because information CPUID shows a feature
> is supported by the hardware does not mean that the kernel has support for that
> feature. This could be because of a feature mismatch between user space and
> kernel, or even some features disabled for use via the, for example "rdt=!l3cat",
> kernel parameter.

James probably also has views on this, but I wouldn't expect userspace to go anywhere
near ACPI parsing. If the description of what MPAM features are supported (before
resctrl is running) is necessary, then we should add it to the existing
description of the various caches etc.  Maybe somewhere in
/sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
for the caches and
/sys/bus/node/devices/nodeX for the DRAM controllers.

I'm sure we can find somewhere sensible for other things covered by MPAM.
The MPAM table parsers and some early code to query features from the hardware should
be able to figure out what is needed.

Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.

Jonathan

> 
> >> fyi ... as with previous attempts to discuss this work I find it difficult
> >> to discuss this work when you are selective about what you want to discuss/answer
> >> and just wipe the rest. Through this I understand that I am not your target
> >> audience.  
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a follower of the
> > "trim your replies" school of e-mail etiquette. I thought I'd covered the gist
> > of your message.
> > 
> > I'll try to be more thorough in responding in the future.  
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be loadable
> modules?
> I think that it is great that the code is modular. Ideally it will help to
> support the other architectures. As you explain this modular design also
> has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I understand
> correctly with code being modular it enables changes from one to the other
> after resctrl mount. User can start with MBA and then switch to MBA_MBps
> without needing to unmount resctrl. What I do not understand is why does
> the code being modular require everything to be modules? Why, for example,
> could a user not interact with a resctrl file that enables the user to make
> this switch from, for example, MBA to MBA_MBps? With this the existing
> interfaces can remain to be respected, the existing mount parameters need
> to remain anyway, while enabling future "more modular" usages.
> 
> Second, copied from my previous email, what is the plan to deal with current
> users that just mount resctrl and expect to learn from it what features are
> supported?
> 
> Reinette
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: resctrl2 - status

[Amit Singh Tomar]

-----Original Message-----
From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
Sent: Tuesday, August 29, 2023 3:53 PM
To: Reinette Chatre <reinette.chatre@intel.com>
Cc: Luck, Tony <tony.luck@intel.com>; Amit Singh Tomar <amitsinght@marvell.com>; Yu, Fenghua <fenghua.yu@intel.com>; james.morse@arm.com; George Cherian <gcherian@marvell.com>; robh@kernel.org; peternewman@google.com; Drew Fustini <dfustini@baylibre.com>; linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
Subject: [EXT] Re: resctrl2 - status

External Email

----------------------------------------------------------------------
On Fri, 25 Aug 2023 13:20:22 -0700
Reinette Chatre <reinette.chatre@intel.com> wrote:

> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl 
> >>>> modules and see what sticks" (even modules of different 
> >>>> architectures since a user space may want to be generic) approach?
> >>>
> >>> This mostly works. Except for the cases where different modules 
> >>> access the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass 
> >>> the "-o cdp" option to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control 
> >>> memory bandwidth with percentages, or with MB/sec values. Again 
> >>> the user already has to make this decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" 
> >>> question would be best answered with a small utility?
> >>
> >> A user space utility or a kernel provided utility? If it is a user 
> >> space utility I think it would end up needing to duplicate what the 
> >> kernel is required to do to know if a particular feature is 
> >> supported. It seems appropriate that this could be a kernel utility 
> >> that can share this existing information with user space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo 
> > doesn't show all features, a utility has access to all the CPUID 
> > leaves that contain the details of each feature enumeration.
> 
> For x86 that may work (in some scenarios, see later) for now but as I 
> understand Arm would need a different solution where I believe the 
> information is obtained via ACPI. I think it is unnecessary to require 
> user space to have parsers for CPUID and ACPI if that same information 
> needs to be parsed by the kernel and there already exists an interface 
> with which the information is communicated from kernel to user space. 
> Also, just because information CPUID shows a feature is supported by 
> the hardware does not mean that the kernel has support for that 
> feature. This could be because of a feature mismatch between user 
> space and kernel, or even some features disabled for use via the, for example "rdt=!l3cat", kernel parameter.

James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing. If the description of what MPAM features are supported (before resctrl is running) is necessary, then we should add it to the existing description of the various caches etc.  Maybe somewhere in /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
for the caches
and
/sys/bus/node/devices/nodeX for the DRAM controllers.
[>>] Or may be all the MPAM features list is unified at one place, something like this ?
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e

I'm sure we can find somewhere sensible for other things covered by MPAM.
The MPAM table parsers and some early code to query features from the hardware should be able to figure out what is needed.

Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.

Jonathan

> 
> >> fyi ... as with previous attempts to discuss this work I find it 
> >> difficult to discuss this work when you are selective about what 
> >> you want to discuss/answer and just wipe the rest. Through this I 
> >> understand that I am not your target audience.
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a 
> > follower of the "trim your replies" school of e-mail etiquette. I 
> > thought I'd covered the gist of your message.
> > 
> > I'll try to be more thorough in responding in the future.  
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be 
> loadable modules?
> I think that it is great that the code is modular. Ideally it will 
> help to support the other architectures. As you explain this modular 
> design also has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I 
> understand correctly with code being modular it enables changes from 
> one to the other after resctrl mount. User can start with MBA and then 
> switch to MBA_MBps without needing to unmount resctrl. What I do not 
> understand is why does the code being modular require everything to be 
> modules? Why, for example, could a user not interact with a resctrl 
> file that enables the user to make this switch from, for example, MBA 
> to MBA_MBps? With this the existing interfaces can remain to be 
> respected, the existing mount parameters need to remain anyway, while enabling future "more modular" usages.
> 
> Second, copied from my previous email, what is the plan to deal with 
> current users that just mount resctrl and expect to learn from it what 
> features are supported?
> 
> Reinette
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
> g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwICAg&c=nKjWec2b6R0mOyPaz7x
> tfQ&r=V_GK7jRuCHDErm6txmgDK1-MbUihtnSQ3gPgB-A-JKU&m=ri5Ay9ia3NmwAS6P94yjZ7rPIR1dIqrIAIjpslfwcd5ulIIs5BNEG9jmTlq6H6o8&s=KZMxmC8InOxAyxgrDig96R2aKf2hc1C_hk_1413pjCI&e=
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [EXT] Re: resctrl2 - status

[Amit Singh Tomar]

-----Original Message-----
From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
Sent: Tuesday, August 29, 2023 3:53 PM
To: Reinette Chatre <reinette.chatre@intel.com>
Cc: Luck, Tony <tony.luck@intel.com>; Amit Singh Tomar <amitsinght@marvell.com>; Yu, Fenghua <fenghua.yu@intel.com>; james.morse@arm.com; George Cherian <gcherian@marvell.com>; robh@kernel.org; peternewman@google.com; Drew Fustini <dfustini@baylibre.com>; linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
Subject: [EXT] Re: resctrl2 - status

External Email

----------------------------------------------------------------------
On Fri, 25 Aug 2023 13:20:22 -0700
Reinette Chatre <reinette.chatre@intel.com> wrote:

> Hi Tony,
> 
> On 8/25/2023 12:44 PM, Luck, Tony wrote:
> >>>> Alternatively, can user space just take a "load all resctrl 
> >>>> modules and see what sticks" (even modules of different 
> >>>> architectures since a user space may want to be generic) approach?
> >>>
> >>> This mostly works. Except for the cases where different modules 
> >>> access the same underlying hardware, so can't be loaded together.
> >>>
> >>> Examples:
> >>>
> >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> >>> But this is already true ... they have to decide whether to pass 
> >>> the "-o cdp" option to mount.
> >>>
> >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control 
> >>> memory bandwidth with percentages, or with MB/sec values. Again 
> >>> the user already has to make this decision when choosing mount options.
> >>>
> >>>
> >>> Maybe the "What resctrl options does this machine support?" 
> >>> question would be best answered with a small utility?
> >>
> >> A user space utility or a kernel provided utility? If it is a user 
> >> space utility I think it would end up needing to duplicate what the 
> >> kernel is required to do to know if a particular feature is 
> >> supported. It seems appropriate that this could be a kernel utility 
> >> that can share this existing information with user space. resctrl already supports the interface for this via /sys/fs/resctrl/info.
> > 
> > I was imagining a user space utility. Even though /proc/cpuinfo 
> > doesn't show all features, a utility has access to all the CPUID 
> > leaves that contain the details of each feature enumeration.
> 
> For x86 that may work (in some scenarios, see later) for now but as I 
> understand Arm would need a different solution where I believe the 
> information is obtained via ACPI. I think it is unnecessary to require 
> user space to have parsers for CPUID and ACPI if that same information 
> needs to be parsed by the kernel and there already exists an interface 
> with which the information is communicated from kernel to user space. 
> Also, just because information CPUID shows a feature is supported by 
> the hardware does not mean that the kernel has support for that 
> feature. This could be because of a feature mismatch between user 
> space and kernel, or even some features disabled for use via the, for example "rdt=!l3cat", kernel parameter.

James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing. If the description of what MPAM features are supported (before resctrl is running) is necessary, then we should add it to the existing description of the various caches etc.  Maybe somewhere in /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
for the caches
and
/sys/bus/node/devices/nodeX for the DRAM controllers.
[>>] Or may be all the MPAM features list is unified at one place, something like this ?
https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e

I'm sure we can find somewhere sensible for other things covered by MPAM.
The MPAM table parsers and some early code to query features from the hardware should be able to figure out what is needed.

Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.

Jonathan

> 
> >> fyi ... as with previous attempts to discuss this work I find it 
> >> difficult to discuss this work when you are selective about what 
> >> you want to discuss/answer and just wipe the rest. Through this I 
> >> understand that I am not your target audience.
> > 
> > Not my intent. I value your input highly. I'm maybe too avid a 
> > follower of the "trim your replies" school of e-mail etiquette. I 
> > thought I'd covered the gist of your message.
> > 
> > I'll try to be more thorough in responding in the future.  
> 
> Two items from my previous email remain open:
> 
> First, why does making the code modular require everything to be 
> loadable modules?
> I think that it is great that the code is modular. Ideally it will 
> help to support the other architectures. As you explain this modular 
> design also has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> Considering your example of MBA and MBA_MBps support ... if I 
> understand correctly with code being modular it enables changes from 
> one to the other after resctrl mount. User can start with MBA and then 
> switch to MBA_MBps without needing to unmount resctrl. What I do not 
> understand is why does the code being modular require everything to be 
> modules? Why, for example, could a user not interact with a resctrl 
> file that enables the user to make this switch from, for example, MBA 
> to MBA_MBps? With this the existing interfaces can remain to be 
> respected, the existing mount parameters need to remain anyway, while enabling future "more modular" usages.
> 
> Second, copied from my previous email, what is the plan to deal with 
> current users that just mount resctrl and expect to learn from it what 
> features are supported?
> 
> Reinette
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
> g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwICAg&c=nKjWec2b6R0mOyPaz7x
> tfQ&r=V_GK7jRuCHDErm6txmgDK1-MbUihtnSQ3gPgB-A-JKU&m=ri5Ay9ia3NmwAS6P94yjZ7rPIR1dIqrIAIjpslfwcd5ulIIs5BNEG9jmTlq6H6o8&s=KZMxmC8InOxAyxgrDig96R2aKf2hc1C_hk_1413pjCI&e=
> 

Re: [EXT] Re: resctrl2 - status

[Jonathan Cameron]

On Tue, 29 Aug 2023 17:18:53 +0000
Amit Singh Tomar <amitsinght@marvell.com> wrote:

> -----Original Message-----
> From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
> Sent: Tuesday, August 29, 2023 3:53 PM
> To: Reinette Chatre <reinette.chatre@intel.com>
> Cc: Luck, Tony <tony.luck@intel.com>; Amit Singh Tomar <amitsinght@marvell.com>; Yu, Fenghua <fenghua.yu@intel.com>; james.morse@arm.com; George Cherian <gcherian@marvell.com>; robh@kernel.org; peternewman@google.com; Drew Fustini <dfustini@baylibre.com>; linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: [EXT] Re: resctrl2 - status
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Fri, 25 Aug 2023 13:20:22 -0700
> Reinette Chatre <reinette.chatre@intel.com> wrote:
> 
> > Hi Tony,
> > 
> > On 8/25/2023 12:44 PM, Luck, Tony wrote:  
> > >>>> Alternatively, can user space just take a "load all resctrl 
> > >>>> modules and see what sticks" (even modules of different 
> > >>>> architectures since a user space may want to be generic) approach?  
> > >>>
> > >>> This mostly works. Except for the cases where different modules 
> > >>> access the same underlying hardware, so can't be loaded together.
> > >>>
> > >>> Examples:
> > >>>
> > >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> > >>> But this is already true ... they have to decide whether to pass 
> > >>> the "-o cdp" option to mount.
> > >>>
> > >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control 
> > >>> memory bandwidth with percentages, or with MB/sec values. Again 
> > >>> the user already has to make this decision when choosing mount options.
> > >>>
> > >>>
> > >>> Maybe the "What resctrl options does this machine support?" 
> > >>> question would be best answered with a small utility?  
> > >>
> > >> A user space utility or a kernel provided utility? If it is a user 
> > >> space utility I think it would end up needing to duplicate what the 
> > >> kernel is required to do to know if a particular feature is 
> > >> supported. It seems appropriate that this could be a kernel utility 
> > >> that can share this existing information with user space. resctrl already supports the interface for this via /sys/fs/resctrl/info.  
> > > 
> > > I was imagining a user space utility. Even though /proc/cpuinfo 
> > > doesn't show all features, a utility has access to all the CPUID 
> > > leaves that contain the details of each feature enumeration.  
> > 
> > For x86 that may work (in some scenarios, see later) for now but as I 
> > understand Arm would need a different solution where I believe the 
> > information is obtained via ACPI. I think it is unnecessary to require 
> > user space to have parsers for CPUID and ACPI if that same information 
> > needs to be parsed by the kernel and there already exists an interface 
> > with which the information is communicated from kernel to user space. 
> > Also, just because information CPUID shows a feature is supported by 
> > the hardware does not mean that the kernel has support for that 
> > feature. This could be because of a feature mismatch between user 
> > space and kernel, or even some features disabled for use via the, for example "rdt=!l3cat", kernel parameter.  
> 
> James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing. If the description of what MPAM features are supported (before resctrl is running) is necessary, then we should add it to the existing description of the various caches etc.  Maybe somewhere in /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
> for the caches
> and
> /sys/bus/node/devices/nodeX for the DRAM controllers.
> [>>] Or may be all the MPAM features list is unified at one place, something like this ?  
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e

Fine for debug, not for generic tooling.

Might be possible to do something separate but I would expect to see this
info associated with the entities being controlled, not off on it's own
somewhere else in sysfs or similar.

Jonathan


> 
> I'm sure we can find somewhere sensible for other things covered by MPAM.
> The MPAM table parsers and some early code to query features from the hardware should be able to figure out what is needed.
> 
> Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.
> 
> Jonathan
> 
> >   
> > >> fyi ... as with previous attempts to discuss this work I find it 
> > >> difficult to discuss this work when you are selective about what 
> > >> you want to discuss/answer and just wipe the rest. Through this I 
> > >> understand that I am not your target audience.  
> > > 
> > > Not my intent. I value your input highly. I'm maybe too avid a 
> > > follower of the "trim your replies" school of e-mail etiquette. I 
> > > thought I'd covered the gist of your message.
> > > 
> > > I'll try to be more thorough in responding in the future.    
> > 
> > Two items from my previous email remain open:
> > 
> > First, why does making the code modular require everything to be 
> > loadable modules?
> > I think that it is great that the code is modular. Ideally it will 
> > help to support the other architectures. As you explain this modular 
> > design also has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> > Considering your example of MBA and MBA_MBps support ... if I 
> > understand correctly with code being modular it enables changes from 
> > one to the other after resctrl mount. User can start with MBA and then 
> > switch to MBA_MBps without needing to unmount resctrl. What I do not 
> > understand is why does the code being modular require everything to be 
> > modules? Why, for example, could a user not interact with a resctrl 
> > file that enables the user to make this switch from, for example, MBA 
> > to MBA_MBps? With this the existing interfaces can remain to be 
> > respected, the existing mount parameters need to remain anyway, while enabling future "more modular" usages.
> > 
> > Second, copied from my previous email, what is the plan to deal with 
> > current users that just mount resctrl and expect to learn from it what 
> > features are supported?
> > 
> > Reinette
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
> > g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwICAg&c=nKjWec2b6R0mOyPaz7x
> > tfQ&r=V_GK7jRuCHDErm6txmgDK1-MbUihtnSQ3gPgB-A-JKU&m=ri5Ay9ia3NmwAS6P94yjZ7rPIR1dIqrIAIjpslfwcd5ulIIs5BNEG9jmTlq6H6o8&s=KZMxmC8InOxAyxgrDig96R2aKf2hc1C_hk_1413pjCI&e=
> >   
> 


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: resctrl2 - status

[Jonathan Cameron]

On Tue, 29 Aug 2023 17:18:53 +0000
Amit Singh Tomar <amitsinght@marvell.com> wrote:

> -----Original Message-----
> From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
> Sent: Tuesday, August 29, 2023 3:53 PM
> To: Reinette Chatre <reinette.chatre@intel.com>
> Cc: Luck, Tony <tony.luck@intel.com>; Amit Singh Tomar <amitsinght@marvell.com>; Yu, Fenghua <fenghua.yu@intel.com>; james.morse@arm.com; George Cherian <gcherian@marvell.com>; robh@kernel.org; peternewman@google.com; Drew Fustini <dfustini@baylibre.com>; linux-kernel@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: [EXT] Re: resctrl2 - status
> 
> External Email
> 
> ----------------------------------------------------------------------
> On Fri, 25 Aug 2023 13:20:22 -0700
> Reinette Chatre <reinette.chatre@intel.com> wrote:
> 
> > Hi Tony,
> > 
> > On 8/25/2023 12:44 PM, Luck, Tony wrote:  
> > >>>> Alternatively, can user space just take a "load all resctrl 
> > >>>> modules and see what sticks" (even modules of different 
> > >>>> architectures since a user space may want to be generic) approach?  
> > >>>
> > >>> This mostly works. Except for the cases where different modules 
> > >>> access the same underlying hardware, so can't be loaded together.
> > >>>
> > >>> Examples:
> > >>>
> > >>> rdt_l3_cat vs. rdt_l3_cdp - user needs to decide whether they want CDP or not.
> > >>> But this is already true ... they have to decide whether to pass 
> > >>> the "-o cdp" option to mount.
> > >>>
> > >>> rdt_l3_mba vs. rdt_l3_mba_MBps - does the user want to control 
> > >>> memory bandwidth with percentages, or with MB/sec values. Again 
> > >>> the user already has to make this decision when choosing mount options.
> > >>>
> > >>>
> > >>> Maybe the "What resctrl options does this machine support?" 
> > >>> question would be best answered with a small utility?  
> > >>
> > >> A user space utility or a kernel provided utility? If it is a user 
> > >> space utility I think it would end up needing to duplicate what the 
> > >> kernel is required to do to know if a particular feature is 
> > >> supported. It seems appropriate that this could be a kernel utility 
> > >> that can share this existing information with user space. resctrl already supports the interface for this via /sys/fs/resctrl/info.  
> > > 
> > > I was imagining a user space utility. Even though /proc/cpuinfo 
> > > doesn't show all features, a utility has access to all the CPUID 
> > > leaves that contain the details of each feature enumeration.  
> > 
> > For x86 that may work (in some scenarios, see later) for now but as I 
> > understand Arm would need a different solution where I believe the 
> > information is obtained via ACPI. I think it is unnecessary to require 
> > user space to have parsers for CPUID and ACPI if that same information 
> > needs to be parsed by the kernel and there already exists an interface 
> > with which the information is communicated from kernel to user space. 
> > Also, just because information CPUID shows a feature is supported by 
> > the hardware does not mean that the kernel has support for that 
> > feature. This could be because of a feature mismatch between user 
> > space and kernel, or even some features disabled for use via the, for example "rdt=!l3cat", kernel parameter.  
> 
> James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing. If the description of what MPAM features are supported (before resctrl is running) is necessary, then we should add it to the existing description of the various caches etc.  Maybe somewhere in /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/
> for the caches
> and
> /sys/bus/node/devices/nodeX for the DRAM controllers.
> [>>] Or may be all the MPAM features list is unified at one place, something like this ?  
> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e

Fine for debug, not for generic tooling.

Might be possible to do something separate but I would expect to see this
info associated with the entities being controlled, not off on it's own
somewhere else in sysfs or similar.

Jonathan


> 
> I'm sure we can find somewhere sensible for other things covered by MPAM.
> The MPAM table parsers and some early code to query features from the hardware should be able to figure out what is needed.
> 
> Perhaps x86 systems could provide similar descriptions but originating from CPUID etc.
> 
> Jonathan
> 
> >   
> > >> fyi ... as with previous attempts to discuss this work I find it 
> > >> difficult to discuss this work when you are selective about what 
> > >> you want to discuss/answer and just wipe the rest. Through this I 
> > >> understand that I am not your target audience.  
> > > 
> > > Not my intent. I value your input highly. I'm maybe too avid a 
> > > follower of the "trim your replies" school of e-mail etiquette. I 
> > > thought I'd covered the gist of your message.
> > > 
> > > I'll try to be more thorough in responding in the future.    
> > 
> > Two items from my previous email remain open:
> > 
> > First, why does making the code modular require everything to be 
> > loadable modules?
> > I think that it is great that the code is modular. Ideally it will 
> > help to support the other architectures. As you explain this modular 
> > design also has the benefit that "modules" can be loaded and unloaded after resctrl mount.
> > Considering your example of MBA and MBA_MBps support ... if I 
> > understand correctly with code being modular it enables changes from 
> > one to the other after resctrl mount. User can start with MBA and then 
> > switch to MBA_MBps without needing to unmount resctrl. What I do not 
> > understand is why does the code being modular require everything to be 
> > modules? Why, for example, could a user not interact with a resctrl 
> > file that enables the user to make this switch from, for example, MBA 
> > to MBA_MBps? With this the existing interfaces can remain to be 
> > respected, the existing mount parameters need to remain anyway, while enabling future "more modular" usages.
> > 
> > Second, copied from my previous email, what is the plan to deal with 
> > current users that just mount resctrl and expect to learn from it what 
> > features are supported?
> > 
> > Reinette
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.infradead.or
> > g_mailman_listinfo_linux-2Darm-2Dkernel&d=DwICAg&c=nKjWec2b6R0mOyPaz7x
> > tfQ&r=V_GK7jRuCHDErm6txmgDK1-MbUihtnSQ3gPgB-A-JKU&m=ri5Ay9ia3NmwAS6P94yjZ7rPIR1dIqrIAIjpslfwcd5ulIIs5BNEG9jmTlq6H6o8&s=KZMxmC8InOxAyxgrDig96R2aKf2hc1C_hk_1413pjCI&e=
> >   
> 

Re: resctrl2 - status

[Tony Luck]

On Mon, Aug 28, 2023 at 07:50:12AM -0700, Reinette Chatre wrote:
> heh ... sounds familiar to "To transition smoothly it may be required
> for all currently supported features to be loaded by default". It is not
> obvious to me how this also closes the other opens.

Reinette,

It seems unlikley that I'll be able to close every open. But I'll
keep trying as you point them out.

I've just pushed an updated set of patches to:

  git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65

Rebased to v6.5. Includes the module auto-load code discussed in
previous e-mail.

James: 

I'm now hoping for some feedback from ARM folks on whether this is a
useful direction. Is it possible to implement MPAM features on top of
this base architecture independent layer. If not, am I just missing
some simple extensions that you will need. Or is there some fundamental
problem that looks hard/impossible to resolve?

Babu:

Can you try this out on an AMD system. I think I covered most of the
existing AMD resctrl features, but I have no machine to test the code
on, so very likely there are bugs in these code paths.

I'd like to make any needed changes now, before I start breaking this
into reviewable bite-sized patches to avoid too much churn.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Mon, Aug 28, 2023 at 07:50:12AM -0700, Reinette Chatre wrote:
> heh ... sounds familiar to "To transition smoothly it may be required
> for all currently supported features to be loaded by default". It is not
> obvious to me how this also closes the other opens.

Reinette,

It seems unlikley that I'll be able to close every open. But I'll
keep trying as you point them out.

I've just pushed an updated set of patches to:

  git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65

Rebased to v6.5. Includes the module auto-load code discussed in
previous e-mail.

James: 

I'm now hoping for some feedback from ARM folks on whether this is a
useful direction. Is it possible to implement MPAM features on top of
this base architecture independent layer. If not, am I just missing
some simple extensions that you will need. Or is there some fundamental
problem that looks hard/impossible to resolve?

Babu:

Can you try this out on an AMD system. I think I covered most of the
existing AMD resctrl features, but I have no machine to test the code
on, so very likely there are bugs in these code paths.

I'd like to make any needed changes now, before I start breaking this
into reviewable bite-sized patches to avoid too much churn.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Moger, Babu]

Hi Tony,

On 9/6/2023 1:21 PM, Tony Luck wrote:
> On Mon, Aug 28, 2023 at 07:50:12AM -0700, Reinette Chatre wrote:
>> heh ... sounds familiar to "To transition smoothly it may be required
>> for all currently supported features to be loaded by default". It is not
>> obvious to me how this also closes the other opens.
> Reinette,
>
> It seems unlikley that I'll be able to close every open. But I'll
> keep trying as you point them out.
>
> I've just pushed an updated set of patches to:
>
>    git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
>
> Rebased to v6.5. Includes the module auto-load code discussed in
> previous e-mail.
>
> James:
>
> I'm now hoping for some feedback from ARM folks on whether this is a
> useful direction. Is it possible to implement MPAM features on top of
> this base architecture independent layer. If not, am I just missing
> some simple extensions that you will need. Or is there some fundamental
> problem that looks hard/impossible to resolve?
>
> Babu:
>
> Can you try this out on an AMD system. I think I covered most of the
> existing AMD resctrl features, but I have no machine to test the code
> on, so very likely there are bugs in these code paths.
>
> I'd like to make any needed changes now, before I start breaking this
> into reviewable bite-sized patches to avoid too much churn.

I tried your latest code briefly on my system.  Unfortunately, I could 
not get it to work on my AMD system.

# git branch -a
   next
* resctrl2_v65
# ]# uname -r
6.5.0+
#lsmod |grep rdt
rdt_show_ids           12288  0
rdt_mbm_local_bytes    12288  0
rdt_mbm_total_bytes    12288  0
rdt_llc_occupancy      12288  0
rdt_l3_cat             16384  0

# lsmod |grep mbe
amd_mbec               16384  0

I could not get  rdt_l3_mba

# modprobe rdt_l3_mba
modprobe: ERROR: could not insert 'rdt_l3_mba': No such device

I don't see any data for the default group either.

mount  -t resctrl resctrl /sys/fs/resctrl/

cd /sys/fs/resctrl/mon_data/mon_L3_00

cat mbm_summary
      n/a      n/a /

Thanks

Babu


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Moger, Babu]

Hi Tony,

On 9/6/2023 1:21 PM, Tony Luck wrote:
> On Mon, Aug 28, 2023 at 07:50:12AM -0700, Reinette Chatre wrote:
>> heh ... sounds familiar to "To transition smoothly it may be required
>> for all currently supported features to be loaded by default". It is not
>> obvious to me how this also closes the other opens.
> Reinette,
>
> It seems unlikley that I'll be able to close every open. But I'll
> keep trying as you point them out.
>
> I've just pushed an updated set of patches to:
>
>    git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
>
> Rebased to v6.5. Includes the module auto-load code discussed in
> previous e-mail.
>
> James:
>
> I'm now hoping for some feedback from ARM folks on whether this is a
> useful direction. Is it possible to implement MPAM features on top of
> this base architecture independent layer. If not, am I just missing
> some simple extensions that you will need. Or is there some fundamental
> problem that looks hard/impossible to resolve?
>
> Babu:
>
> Can you try this out on an AMD system. I think I covered most of the
> existing AMD resctrl features, but I have no machine to test the code
> on, so very likely there are bugs in these code paths.
>
> I'd like to make any needed changes now, before I start breaking this
> into reviewable bite-sized patches to avoid too much churn.

I tried your latest code briefly on my system.  Unfortunately, I could 
not get it to work on my AMD system.

# git branch -a
   next
* resctrl2_v65
# ]# uname -r
6.5.0+
#lsmod |grep rdt
rdt_show_ids           12288  0
rdt_mbm_local_bytes    12288  0
rdt_mbm_total_bytes    12288  0
rdt_llc_occupancy      12288  0
rdt_l3_cat             16384  0

# lsmod |grep mbe
amd_mbec               16384  0

I could not get  rdt_l3_mba

# modprobe rdt_l3_mba
modprobe: ERROR: could not insert 'rdt_l3_mba': No such device

I don't see any data for the default group either.

mount  -t resctrl resctrl /sys/fs/resctrl/

cd /sys/fs/resctrl/mon_data/mon_L3_00

cat mbm_summary
      n/a      n/a /

Thanks

Babu

RE: resctrl2 - status

[Luck, Tony]

> > Can you try this out on an AMD system. I think I covered most of the
> > existing AMD resctrl features, but I have no machine to test the code
> > on, so very likely there are bugs in these code paths.
> >
> > I'd like to make any needed changes now, before I start breaking this
> > into reviewable bite-sized patches to avoid too much churn.
>
> I tried your latest code briefly on my system.  Unfortunately, I could
> not get it to work on my AMD system.
>
> # git branch -a
>    next
> * resctrl2_v65
> # ]# uname -r
> 6.5.0+
> #lsmod |grep rdt
> rdt_show_ids           12288  0
> rdt_mbm_local_bytes    12288  0
> rdt_mbm_total_bytes    12288  0
> rdt_llc_occupancy      12288  0
> rdt_l3_cat             16384  0
>
> # lsmod |grep mbe
> amd_mbec               16384  0
>
> I could not get  rdt_l3_mba
>
> # modprobe rdt_l3_mba
> modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
>
> I don't see any data for the default group either.
>
> mount  -t resctrl resctrl /sys/fs/resctrl/
>
> cd /sys/fs/resctrl/mon_data/mon_L3_00
>
> cat mbm_summary
>       n/a      n/a /

Babu,

Thank a bunch for taking this for a quick spin. There's several bits of
good news there. Several modules automatically loaded as expected.
Nothing went "OOPS" and crashed the system.

Here’s the code that the rdt_l3_mba module runs that can cause failure
to load with "No such device"

        if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
                pr_debug("No RDT allocation support\n");
                return -ENODEV;
        }

        mba_features = cpuid_ebx(0x10);

        if (!(mba_features & BIT(3))) {
                pr_debug("No RDT MBA allocation\n");
                return -ENODEV;
        }

I assume the first test must have succeeded (same code in rdt_l3_cat, and
that loaded OK). So must be the second. How does AMD enumerate MBA
support?

Less obvious what is the root cause of the mbm_summary file to fail to
show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
monitoring.

-Tony

RE: resctrl2 - status

[Luck, Tony]

> > Can you try this out on an AMD system. I think I covered most of the
> > existing AMD resctrl features, but I have no machine to test the code
> > on, so very likely there are bugs in these code paths.
> >
> > I'd like to make any needed changes now, before I start breaking this
> > into reviewable bite-sized patches to avoid too much churn.
>
> I tried your latest code briefly on my system.  Unfortunately, I could
> not get it to work on my AMD system.
>
> # git branch -a
>    next
> * resctrl2_v65
> # ]# uname -r
> 6.5.0+
> #lsmod |grep rdt
> rdt_show_ids           12288  0
> rdt_mbm_local_bytes    12288  0
> rdt_mbm_total_bytes    12288  0
> rdt_llc_occupancy      12288  0
> rdt_l3_cat             16384  0
>
> # lsmod |grep mbe
> amd_mbec               16384  0
>
> I could not get  rdt_l3_mba
>
> # modprobe rdt_l3_mba
> modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
>
> I don't see any data for the default group either.
>
> mount  -t resctrl resctrl /sys/fs/resctrl/
>
> cd /sys/fs/resctrl/mon_data/mon_L3_00
>
> cat mbm_summary
>       n/a      n/a /

Babu,

Thank a bunch for taking this for a quick spin. There's several bits of
good news there. Several modules automatically loaded as expected.
Nothing went "OOPS" and crashed the system.

Here’s the code that the rdt_l3_mba module runs that can cause failure
to load with "No such device"

        if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
                pr_debug("No RDT allocation support\n");
                return -ENODEV;
        }

        mba_features = cpuid_ebx(0x10);

        if (!(mba_features & BIT(3))) {
                pr_debug("No RDT MBA allocation\n");
                return -ENODEV;
        }

I assume the first test must have succeeded (same code in rdt_l3_cat, and
that loaded OK). So must be the second. How does AMD enumerate MBA
support?

Less obvious what is the root cause of the mbm_summary file to fail to
show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
monitoring.

-Tony


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Moger, Babu]

Hi Tony,


On 9/8/2023 1:51 PM, Luck, Tony wrote:
>>> Can you try this out on an AMD system. I think I covered most of the
>>> existing AMD resctrl features, but I have no machine to test the code
>>> on, so very likely there are bugs in these code paths.
>>>
>>> I'd like to make any needed changes now, before I start breaking this
>>> into reviewable bite-sized patches to avoid too much churn.
>> I tried your latest code briefly on my system.  Unfortunately, I could
>> not get it to work on my AMD system.
>>
>> # git branch -a
>>     next
>> * resctrl2_v65
>> # ]# uname -r
>> 6.5.0+
>> #lsmod |grep rdt
>> rdt_show_ids           12288  0
>> rdt_mbm_local_bytes    12288  0
>> rdt_mbm_total_bytes    12288  0
>> rdt_llc_occupancy      12288  0
>> rdt_l3_cat             16384  0
>>
>> # lsmod |grep mbe
>> amd_mbec               16384  0
>>
>> I could not get  rdt_l3_mba
>>
>> # modprobe rdt_l3_mba
>> modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
>>
>> I don't see any data for the default group either.
>>
>> mount  -t resctrl resctrl /sys/fs/resctrl/
>>
>> cd /sys/fs/resctrl/mon_data/mon_L3_00
>>
>> cat mbm_summary
>>        n/a      n/a /
> Babu,
>
> Thank a bunch for taking this for a quick spin. There's several bits of
> good news there. Several modules automatically loaded as expected.
> Nothing went "OOPS" and crashed the system.
>
> Here’s the code that the rdt_l3_mba module runs that can cause failure
> to load with "No such device"
>
>          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
>                  pr_debug("No RDT allocation support\n");
>                  return -ENODEV;
>          }

Shouldn't this be ?(or similar)

if (!rdt_cpu_has(X86_FEATURE_MBA))
                 return false;

>          mba_features = cpuid_ebx(0x10);
>
>          if (!(mba_features & BIT(3))) {
>                  pr_debug("No RDT MBA allocation\n");
>                  return -ENODEV;
>          }
>
> I assume the first test must have succeeded (same code in rdt_l3_cat, and
> that loaded OK). So must be the second. How does AMD enumerate MBA
> support?
>
> Less obvious what is the root cause of the mbm_summary file to fail to
> show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> monitoring.

I am still not sure if resctrl2 will address all the current gaps in 
resctrl1. We should probably list all issues on the table before we go 
that route.

One of the main issue for AMD is coupling of LLC domains.

For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux 
design assumes that there are globally 16 total CLOSIDs for the whole 
systems. We can only create 16 CLOSID now irrespective of how many 
domains are there.

In reality, we should be able to create "16 x number of LLC domains" 
CLOSIDS in the systems.  This is more evident in AMD. But, same problem 
applies to Intel with multiple sockets.

My 02 cents. Hope to discuss more in our upcoming meeting.

thanks

Re: resctrl2 - status

[Moger, Babu]

Hi Tony,


On 9/8/2023 1:51 PM, Luck, Tony wrote:
>>> Can you try this out on an AMD system. I think I covered most of the
>>> existing AMD resctrl features, but I have no machine to test the code
>>> on, so very likely there are bugs in these code paths.
>>>
>>> I'd like to make any needed changes now, before I start breaking this
>>> into reviewable bite-sized patches to avoid too much churn.
>> I tried your latest code briefly on my system.  Unfortunately, I could
>> not get it to work on my AMD system.
>>
>> # git branch -a
>>     next
>> * resctrl2_v65
>> # ]# uname -r
>> 6.5.0+
>> #lsmod |grep rdt
>> rdt_show_ids           12288  0
>> rdt_mbm_local_bytes    12288  0
>> rdt_mbm_total_bytes    12288  0
>> rdt_llc_occupancy      12288  0
>> rdt_l3_cat             16384  0
>>
>> # lsmod |grep mbe
>> amd_mbec               16384  0
>>
>> I could not get  rdt_l3_mba
>>
>> # modprobe rdt_l3_mba
>> modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
>>
>> I don't see any data for the default group either.
>>
>> mount  -t resctrl resctrl /sys/fs/resctrl/
>>
>> cd /sys/fs/resctrl/mon_data/mon_L3_00
>>
>> cat mbm_summary
>>        n/a      n/a /
> Babu,
>
> Thank a bunch for taking this for a quick spin. There's several bits of
> good news there. Several modules automatically loaded as expected.
> Nothing went "OOPS" and crashed the system.
>
> Here’s the code that the rdt_l3_mba module runs that can cause failure
> to load with "No such device"
>
>          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
>                  pr_debug("No RDT allocation support\n");
>                  return -ENODEV;
>          }

Shouldn't this be ?(or similar)

if (!rdt_cpu_has(X86_FEATURE_MBA))
                 return false;

>          mba_features = cpuid_ebx(0x10);
>
>          if (!(mba_features & BIT(3))) {
>                  pr_debug("No RDT MBA allocation\n");
>                  return -ENODEV;
>          }
>
> I assume the first test must have succeeded (same code in rdt_l3_cat, and
> that loaded OK). So must be the second. How does AMD enumerate MBA
> support?
>
> Less obvious what is the root cause of the mbm_summary file to fail to
> show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> monitoring.

I am still not sure if resctrl2 will address all the current gaps in 
resctrl1. We should probably list all issues on the table before we go 
that route.

One of the main issue for AMD is coupling of LLC domains.

For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux 
design assumes that there are globally 16 total CLOSIDs for the whole 
systems. We can only create 16 CLOSID now irrespective of how many 
domains are there.

In reality, we should be able to create "16 x number of LLC domains" 
CLOSIDS in the systems.  This is more evident in AMD. But, same problem 
applies to Intel with multiple sockets.

My 02 cents. Hope to discuss more in our upcoming meeting.

thanks



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:
> Hi Tony,
> 
> 
> On 9/8/2023 1:51 PM, Luck, Tony wrote:
> > > > Can you try this out on an AMD system. I think I covered most of the
> > > > existing AMD resctrl features, but I have no machine to test the code
> > > > on, so very likely there are bugs in these code paths.
> > > > 
> > > > I'd like to make any needed changes now, before I start breaking this
> > > > into reviewable bite-sized patches to avoid too much churn.
> > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > not get it to work on my AMD system.
> > > 
> > > # git branch -a
> > >     next
> > > * resctrl2_v65
> > > # ]# uname -r
> > > 6.5.0+
> > > #lsmod |grep rdt
> > > rdt_show_ids           12288  0
> > > rdt_mbm_local_bytes    12288  0
> > > rdt_mbm_total_bytes    12288  0
> > > rdt_llc_occupancy      12288  0
> > > rdt_l3_cat             16384  0
> > > 
> > > # lsmod |grep mbe
> > > amd_mbec               16384  0
> > > 
> > > I could not get  rdt_l3_mba
> > > 
> > > # modprobe rdt_l3_mba
> > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > 
> > > I don't see any data for the default group either.
> > > 
> > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > 
> > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > 
> > > cat mbm_summary
> > >        n/a      n/a /
> > Babu,
> > 
> > Thank a bunch for taking this for a quick spin. There's several bits of
> > good news there. Several modules automatically loaded as expected.
> > Nothing went "OOPS" and crashed the system.
> > 
> > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > to load with "No such device"
> > 
> >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> >                  pr_debug("No RDT allocation support\n");
> >                  return -ENODEV;
> >          }
> 
> Shouldn't this be ?(or similar)
> 
> if (!rdt_cpu_has(X86_FEATURE_MBA))
>                 return false;

Yes. I should be using X86_FEATURE bits where they are available
rather than peeking directly at CPUID register bits.

> 
> >          mba_features = cpuid_ebx(0x10);
> > 
> >          if (!(mba_features & BIT(3))) {
> >                  pr_debug("No RDT MBA allocation\n");
> >                  return -ENODEV;
> >          }
> > 
> > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > that loaded OK). So must be the second. How does AMD enumerate MBA
> > support?
> > 
> > Less obvious what is the root cause of the mbm_summary file to fail to
> > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > monitoring.
> 
> I am still not sure if resctrl2 will address all the current gaps in
> resctrl1. We should probably list all issues on the table before we go that
> route.

Indeed yes! I don't want to have to do resctrl3 in a few years to
cover gaps that could have been addressed in resctrl2.

However, fixing resctrl gaps is only one of the motivations for
the rewrite. The bigger one is making life easier for all the
architectures sharing the common code to do what they need to
for their own quirks & differences without cluttering the
common code base, or worrying "did my change just break something
for another CPU architecture".

> One of the main issue for AMD is coupling of LLC domains.
> 
> For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> design assumes that there are globally 16 total CLOSIDs for the whole
> systems. We can only create 16 CLOSID now irrespective of how many domains
> are there.
> 
> In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> in the systems.  This is more evident in AMD. But, same problem applies to
> Intel with multiple sockets.

I think this can be somewhat achieved already with a combination of
resctrl and cpusets (or some other way to set CPU affinity for tasks
to only run on CPUs within a specific domain (or set of domains).
That's why the schemata file allows setting different CBM masks
per domain.

Can you explain how you would use 64 domains on a system with 4 domains
and 16 CLOSID per domain?

> My 02 cents. Hope to discuss more in our upcoming meeting.
Agreed. This will be faster when we can talk instead of type :-)

-Tony

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:
> Hi Tony,
> 
> 
> On 9/8/2023 1:51 PM, Luck, Tony wrote:
> > > > Can you try this out on an AMD system. I think I covered most of the
> > > > existing AMD resctrl features, but I have no machine to test the code
> > > > on, so very likely there are bugs in these code paths.
> > > > 
> > > > I'd like to make any needed changes now, before I start breaking this
> > > > into reviewable bite-sized patches to avoid too much churn.
> > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > not get it to work on my AMD system.
> > > 
> > > # git branch -a
> > >     next
> > > * resctrl2_v65
> > > # ]# uname -r
> > > 6.5.0+
> > > #lsmod |grep rdt
> > > rdt_show_ids           12288  0
> > > rdt_mbm_local_bytes    12288  0
> > > rdt_mbm_total_bytes    12288  0
> > > rdt_llc_occupancy      12288  0
> > > rdt_l3_cat             16384  0
> > > 
> > > # lsmod |grep mbe
> > > amd_mbec               16384  0
> > > 
> > > I could not get  rdt_l3_mba
> > > 
> > > # modprobe rdt_l3_mba
> > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > 
> > > I don't see any data for the default group either.
> > > 
> > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > 
> > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > 
> > > cat mbm_summary
> > >        n/a      n/a /
> > Babu,
> > 
> > Thank a bunch for taking this for a quick spin. There's several bits of
> > good news there. Several modules automatically loaded as expected.
> > Nothing went "OOPS" and crashed the system.
> > 
> > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > to load with "No such device"
> > 
> >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> >                  pr_debug("No RDT allocation support\n");
> >                  return -ENODEV;
> >          }
> 
> Shouldn't this be ?(or similar)
> 
> if (!rdt_cpu_has(X86_FEATURE_MBA))
>                 return false;

Yes. I should be using X86_FEATURE bits where they are available
rather than peeking directly at CPUID register bits.

> 
> >          mba_features = cpuid_ebx(0x10);
> > 
> >          if (!(mba_features & BIT(3))) {
> >                  pr_debug("No RDT MBA allocation\n");
> >                  return -ENODEV;
> >          }
> > 
> > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > that loaded OK). So must be the second. How does AMD enumerate MBA
> > support?
> > 
> > Less obvious what is the root cause of the mbm_summary file to fail to
> > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > monitoring.
> 
> I am still not sure if resctrl2 will address all the current gaps in
> resctrl1. We should probably list all issues on the table before we go that
> route.

Indeed yes! I don't want to have to do resctrl3 in a few years to
cover gaps that could have been addressed in resctrl2.

However, fixing resctrl gaps is only one of the motivations for
the rewrite. The bigger one is making life easier for all the
architectures sharing the common code to do what they need to
for their own quirks & differences without cluttering the
common code base, or worrying "did my change just break something
for another CPU architecture".

> One of the main issue for AMD is coupling of LLC domains.
> 
> For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> design assumes that there are globally 16 total CLOSIDs for the whole
> systems. We can only create 16 CLOSID now irrespective of how many domains
> are there.
> 
> In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> in the systems.  This is more evident in AMD. But, same problem applies to
> Intel with multiple sockets.

I think this can be somewhat achieved already with a combination of
resctrl and cpusets (or some other way to set CPU affinity for tasks
to only run on CPUs within a specific domain (or set of domains).
That's why the schemata file allows setting different CBM masks
per domain.

Can you explain how you would use 64 domains on a system with 4 domains
and 16 CLOSID per domain?

> My 02 cents. Hope to discuss more in our upcoming meeting.
Agreed. This will be faster when we can talk instead of type :-)

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[James Morse]

Hi Tony,

On 06/09/2023 19:21, Tony Luck wrote:
> I've just pushed an updated set of patches to:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> 
> Rebased to v6.5. Includes the module auto-load code discussed in
> previous e-mail.

I've taken a look at your resctrl2_v65rc7 branch. I don't think my reply to your first
series got sent, sorry if I'm actually repeating myself.

It goes without saying that I'd prefer we refactor the existing filesystem into the shape
we need, instead of a wholesale replacement.


All this focusses on the structure of the code, but we can change that anytime. I'd prefer
we work out what user-space needs and go from there.

I think the problems are:
 * Discovering what the platform supports, including possible conflicting schemes.
 * Architecture/Platform specific schema. e.g. SMBA. MPAM will have a handful of these too.
* User-space should know which schema are portable, and which are not. e.g. L2/L3/MB.
* Different control types for the same resource. e.g. Intel uses a percentage for
bandwidth control, AMD an absolute number. We should standardise this sort of thing and
make it discoverable.
* Conflicting schemes for the same hardware. e.g. CDP and mba_MBps

~

I'd really like to keep the 'core' schema in the base driver. This is to ensure that 'L3'
doesn't behave differently due to differing software implementation on Intel/AMD/Arm/riscv.

The L{2.3} schema were really well defined as they reference other things that are already
visible in sysfs, this makes them nice and portable. I think this is the standard we
should aim for with any new schema, and only resort to making them arch-specific if the
hardware is just too strange.
I tried to find a way to do this for SMBA as arm platforms will certainly have something
similar, but the 'any other NUMA node' behaviour was just too specific to what was built.

I suspect this could be done with a frontend/backend split, with a 'common' directory
holding the frontend driver for schema that apply to all architectures if the backend has
shown up.

~

Because I think the types a schema is configured with should be portable across
architectures, I'd prefer the string parsing code lives in the core code, and isn't
duplicated by every submodule! String parsing in the kernel is bad enough!

The obvious types we have so far are: bitmap/percentage/number/bandwidth. I think it makes
sense for user-space to be told what the schema is configured with, and let the core code
do that parsing.

~

I don't have a good answer for conflicting drivers for the same hardware resource. I think
we'd need to find a way of making the existing command-line arguments work, causing the
corresponding module to auto-load.

But this sucks for distros, who would need somewhere to describe the policy of which
modules get loaded. The good news is things like libvirt aren't mounting resctrl themselves.

~

I suspect the CDP schemes should be made arch-specific as x86 needs the odd/even pairing,
whereas MPAM does not. This would allow a scheme where the Iside CLOSIDs can be shared to
avoid halving the available CLOSID.

Having somewhere sensible to put the MPAM driver is useful. It's currently dumped in
drivers/platform as it doesn't really fit anywhere!

Allowing each submodule to add extra files to the info directories looks useful. MPAM's
priority control has a property to say whether 0 is the top of the scale or not, this
would allow it to be exposed to user-space, instead of having to shift the range up/down
to hide the difference in behaviour.

MPAM only needs to update the hardware on a CPU that is part of target domain on some
platforms. The vast majority would allow the MMIO writes to come from anywhere. Having the
applychanges behaviour specific to the submodule would reduce the number of IPI.
From what I've seen, riscv never needs an IPI here.

It looks like all the core code runs at process context without touching the pre-empt
counter or masking interrupts - this is a really nice property. Most of my headaches have
come from this area.

The limbo work isn't arch specific, that is a property of caches, it should really be core
code behaviour to avoid duplication. MPAM needs that, I expect riscv does too.

Making the CLOSID/RMID allocation behaviour arch specific saves some headaches. MPAM is
particularly different in this area. I don't know what riscv's behaviour is here.



> James: 
> 
> I'm now hoping for some feedback from ARM folks on whether this is a
> useful direction. Is it possible to implement MPAM features on top of
> this base architecture independent layer. If not, am I just missing
> some simple extensions that you will need. Or is there some fundamental
> problem that looks hard/impossible to resolve?


You've got an rdt_l2_cat driver which is really a symlink to rdt_l3_cat built with
different pre-processor defines. It's a good trick, but it only works because you know the
full range of hardware that has been built.
MPAM can't do things like this, it really is a bag of bits, and it isn't until the ACPI
tables are parsed to find where the resources are - and then that hardware is probed -
that we have a clue what was built.
For example, the cache controls could be on any cache in the hierarchy, and the number is
derived from the structure of the PPTT table. You've already got a fair amount of
duplication when multiple struct resctrl_resources are defined. MPAM would have to
allocate and populate these dynamically to avoid building a module for 'L5' ... just in
case someone built that...

The mba_MBps logic is really just software, I see no reason to duplicate it on multiple
architectures. This should be in the core of the filesystem. We already have a mount
option to enable it.

I see the arch/submodules can't influence the domain numbers ... this is a problem as it
hardcodes "you can have L2 or L3", which is something you were trying to avoid. MPAM will
want to use NUMA-IDs for memory-side-caches, (I'd hope this to be a core/portable schema),
as well as IOMMU-IDs for the I/O side of this.
I don't think this is really a problem, as I'd like to add the things I need in this area
as core/portable schema.

Arm's IOMMU has support to label traffic with the equivalent to CLOSID/RMID. My prototype
support for this adds IOMMU-groups to the resctrl tasks file so that the devices can be
moved between control/monitor groups as if they were tasks. I think this would work for
other architectures if they had similar support as IOMMU-groups are an existing concept.


Thanks,

James

Re: resctrl2 - status

[James Morse]

Hi Tony,

On 06/09/2023 19:21, Tony Luck wrote:
> I've just pushed an updated set of patches to:
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> 
> Rebased to v6.5. Includes the module auto-load code discussed in
> previous e-mail.

I've taken a look at your resctrl2_v65rc7 branch. I don't think my reply to your first
series got sent, sorry if I'm actually repeating myself.

It goes without saying that I'd prefer we refactor the existing filesystem into the shape
we need, instead of a wholesale replacement.


All this focusses on the structure of the code, but we can change that anytime. I'd prefer
we work out what user-space needs and go from there.

I think the problems are:
 * Discovering what the platform supports, including possible conflicting schemes.
 * Architecture/Platform specific schema. e.g. SMBA. MPAM will have a handful of these too.
* User-space should know which schema are portable, and which are not. e.g. L2/L3/MB.
* Different control types for the same resource. e.g. Intel uses a percentage for
bandwidth control, AMD an absolute number. We should standardise this sort of thing and
make it discoverable.
* Conflicting schemes for the same hardware. e.g. CDP and mba_MBps

~

I'd really like to keep the 'core' schema in the base driver. This is to ensure that 'L3'
doesn't behave differently due to differing software implementation on Intel/AMD/Arm/riscv.

The L{2.3} schema were really well defined as they reference other things that are already
visible in sysfs, this makes them nice and portable. I think this is the standard we
should aim for with any new schema, and only resort to making them arch-specific if the
hardware is just too strange.
I tried to find a way to do this for SMBA as arm platforms will certainly have something
similar, but the 'any other NUMA node' behaviour was just too specific to what was built.

I suspect this could be done with a frontend/backend split, with a 'common' directory
holding the frontend driver for schema that apply to all architectures if the backend has
shown up.

~

Because I think the types a schema is configured with should be portable across
architectures, I'd prefer the string parsing code lives in the core code, and isn't
duplicated by every submodule! String parsing in the kernel is bad enough!

The obvious types we have so far are: bitmap/percentage/number/bandwidth. I think it makes
sense for user-space to be told what the schema is configured with, and let the core code
do that parsing.

~

I don't have a good answer for conflicting drivers for the same hardware resource. I think
we'd need to find a way of making the existing command-line arguments work, causing the
corresponding module to auto-load.

But this sucks for distros, who would need somewhere to describe the policy of which
modules get loaded. The good news is things like libvirt aren't mounting resctrl themselves.

~

I suspect the CDP schemes should be made arch-specific as x86 needs the odd/even pairing,
whereas MPAM does not. This would allow a scheme where the Iside CLOSIDs can be shared to
avoid halving the available CLOSID.

Having somewhere sensible to put the MPAM driver is useful. It's currently dumped in
drivers/platform as it doesn't really fit anywhere!

Allowing each submodule to add extra files to the info directories looks useful. MPAM's
priority control has a property to say whether 0 is the top of the scale or not, this
would allow it to be exposed to user-space, instead of having to shift the range up/down
to hide the difference in behaviour.

MPAM only needs to update the hardware on a CPU that is part of target domain on some
platforms. The vast majority would allow the MMIO writes to come from anywhere. Having the
applychanges behaviour specific to the submodule would reduce the number of IPI.
From what I've seen, riscv never needs an IPI here.

It looks like all the core code runs at process context without touching the pre-empt
counter or masking interrupts - this is a really nice property. Most of my headaches have
come from this area.

The limbo work isn't arch specific, that is a property of caches, it should really be core
code behaviour to avoid duplication. MPAM needs that, I expect riscv does too.

Making the CLOSID/RMID allocation behaviour arch specific saves some headaches. MPAM is
particularly different in this area. I don't know what riscv's behaviour is here.



> James: 
> 
> I'm now hoping for some feedback from ARM folks on whether this is a
> useful direction. Is it possible to implement MPAM features on top of
> this base architecture independent layer. If not, am I just missing
> some simple extensions that you will need. Or is there some fundamental
> problem that looks hard/impossible to resolve?


You've got an rdt_l2_cat driver which is really a symlink to rdt_l3_cat built with
different pre-processor defines. It's a good trick, but it only works because you know the
full range of hardware that has been built.
MPAM can't do things like this, it really is a bag of bits, and it isn't until the ACPI
tables are parsed to find where the resources are - and then that hardware is probed -
that we have a clue what was built.
For example, the cache controls could be on any cache in the hierarchy, and the number is
derived from the structure of the PPTT table. You've already got a fair amount of
duplication when multiple struct resctrl_resources are defined. MPAM would have to
allocate and populate these dynamically to avoid building a module for 'L5' ... just in
case someone built that...

The mba_MBps logic is really just software, I see no reason to duplicate it on multiple
architectures. This should be in the core of the filesystem. We already have a mount
option to enable it.

I see the arch/submodules can't influence the domain numbers ... this is a problem as it
hardcodes "you can have L2 or L3", which is something you were trying to avoid. MPAM will
want to use NUMA-IDs for memory-side-caches, (I'd hope this to be a core/portable schema),
as well as IOMMU-IDs for the I/O side of this.
I don't think this is really a problem, as I'd like to add the things I need in this area
as core/portable schema.

Arm's IOMMU has support to label traffic with the equivalent to CLOSID/RMID. My prototype
support for this adds IOMMU-groups to the resctrl tasks file so that the devices can be
moved between control/monitor groups as if they were tasks. I think this would work for
other architectures if they had similar support as IOMMU-groups are an existing concept.


Thanks,

James

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [EXT] Re: resctrl2 - status

[James Morse]

Hi guys,

On 30/08/2023 11:47, Jonathan Cameron wrote:
> On Tue, 29 Aug 2023 17:18:53 +0000
> Amit Singh Tomar <amitsinght@marvell.com> wrote:
>> From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
>> James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing.

I agree wholeheartedly! Parsing the ACPI tables also wouldn't work: the firmware tables
describe where the MPAM hardware is, but the features that are supported are described by
the hardware, then grouped to find the common features we can make use of.


>> If the description of what MPAM features are supported (before resctrl is running) is necessary, then we
>> should add it to the existing description of the various caches etc.  Maybe somewhere in
>> /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/

>> for the caches
>> and
>> /sys/bus/node/devices/nodeX for the DRAM controllers.

>> [>>] Or may be all the MPAM features list is unified at one place, something like this ?  
>> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e
> 
> Fine for debug, not for generic tooling.

Yeah, this is really just for folk reporting bugs to me so we can work out what their
firmware table described, and what the driver found.


Thanks,

James

Re: [EXT] Re: resctrl2 - status

[James Morse]

Hi guys,

On 30/08/2023 11:47, Jonathan Cameron wrote:
> On Tue, 29 Aug 2023 17:18:53 +0000
> Amit Singh Tomar <amitsinght@marvell.com> wrote:
>> From: Jonathan Cameron <Jonathan.Cameron@Huawei.com> 
>> James probably also has views on this, but I wouldn't expect userspace to go anywhere near ACPI parsing.

I agree wholeheartedly! Parsing the ACPI tables also wouldn't work: the firmware tables
describe where the MPAM hardware is, but the features that are supported are described by
the hardware, then grouped to find the common features we can make use of.


>> If the description of what MPAM features are supported (before resctrl is running) is necessary, then we
>> should add it to the existing description of the various caches etc.  Maybe somewhere in
>> /sys/bus/node/devices/nodeX/cpuY/cache/indexZ/

>> for the caches
>> and
>> /sys/bus/node/devices/nodeX for the DRAM controllers.

>> [>>] Or may be all the MPAM features list is unified at one place, something like this ?  
>> https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/?h=mpam/snapshot/v6.5-rc1&id=c32874cd1d68241c481a9d0c32c8c65d019c247e
> 
> Fine for debug, not for generic tooling.

Yeah, this is really just for folk reporting bugs to me so we can work out what their
firmware table described, and what the driver found.


Thanks,

James

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Drew Fustini]

On Fri, Sep 08, 2023 at 04:13:54PM -0700, Tony Luck wrote:
> On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:
> > Hi Tony,
> > 
> > 
> > On 9/8/2023 1:51 PM, Luck, Tony wrote:
> > > > > Can you try this out on an AMD system. I think I covered most of the
> > > > > existing AMD resctrl features, but I have no machine to test the code
> > > > > on, so very likely there are bugs in these code paths.
> > > > > 
> > > > > I'd like to make any needed changes now, before I start breaking this
> > > > > into reviewable bite-sized patches to avoid too much churn.
> > > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > > not get it to work on my AMD system.
> > > > 
> > > > # git branch -a
> > > >     next
> > > > * resctrl2_v65
> > > > # ]# uname -r
> > > > 6.5.0+
> > > > #lsmod |grep rdt
> > > > rdt_show_ids           12288  0
> > > > rdt_mbm_local_bytes    12288  0
> > > > rdt_mbm_total_bytes    12288  0
> > > > rdt_llc_occupancy      12288  0
> > > > rdt_l3_cat             16384  0
> > > > 
> > > > # lsmod |grep mbe
> > > > amd_mbec               16384  0
> > > > 
> > > > I could not get  rdt_l3_mba
> > > > 
> > > > # modprobe rdt_l3_mba
> > > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > > 
> > > > I don't see any data for the default group either.
> > > > 
> > > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > > 
> > > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > > 
> > > > cat mbm_summary
> > > >        n/a      n/a /
> > > Babu,
> > > 
> > > Thank a bunch for taking this for a quick spin. There's several bits of
> > > good news there. Several modules automatically loaded as expected.
> > > Nothing went "OOPS" and crashed the system.
> > > 
> > > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > > to load with "No such device"
> > > 
> > >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> > >                  pr_debug("No RDT allocation support\n");
> > >                  return -ENODEV;
> > >          }
> > 
> > Shouldn't this be ?(or similar)
> > 
> > if (!rdt_cpu_has(X86_FEATURE_MBA))
> >                 return false;
> 
> Yes. I should be using X86_FEATURE bits where they are available
> rather than peeking directly at CPUID register bits.
> 
> > 
> > >          mba_features = cpuid_ebx(0x10);
> > > 
> > >          if (!(mba_features & BIT(3))) {
> > >                  pr_debug("No RDT MBA allocation\n");
> > >                  return -ENODEV;
> > >          }
> > > 
> > > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > > that loaded OK). So must be the second. How does AMD enumerate MBA
> > > support?
> > > 
> > > Less obvious what is the root cause of the mbm_summary file to fail to
> > > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > > monitoring.
> > 
> > I am still not sure if resctrl2 will address all the current gaps in
> > resctrl1. We should probably list all issues on the table before we go that
> > route.
> 
> Indeed yes! I don't want to have to do resctrl3 in a few years to
> cover gaps that could have been addressed in resctrl2.
> 
> However, fixing resctrl gaps is only one of the motivations for
> the rewrite. The bigger one is making life easier for all the
> architectures sharing the common code to do what they need to
> for their own quirks & differences without cluttering the
> common code base, or worrying "did my change just break something
> for another CPU architecture".
> 
> > One of the main issue for AMD is coupling of LLC domains.
> > 
> > For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> > design assumes that there are globally 16 total CLOSIDs for the whole
> > systems. We can only create 16 CLOSID now irrespective of how many domains
> > are there.
> > 
> > In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> > in the systems.  This is more evident in AMD. But, same problem applies to
> > Intel with multiple sockets.
> 
> I think this can be somewhat achieved already with a combination of
> resctrl and cpusets (or some other way to set CPU affinity for tasks
> to only run on CPUs within a specific domain (or set of domains).
> That's why the schemata file allows setting different CBM masks
> per domain.
> 
> Can you explain how you would use 64 domains on a system with 4 domains
> and 16 CLOSID per domain?
> 
> > My 02 cents. Hope to discuss more in our upcoming meeting.
> Agreed. This will be faster when we can talk instead of type :-)

Is it a meeting that other interested developers can join?

This reminds me that Linux Plumbers Conference [1] is in November and
I think resctrl2 could be a good topic. The CFP is still open for Birds
of a Feather (BoF) proposals [2]. These are free-form get-togethers for
people wishing to discuss a particular topic, and I have had success
hosting them in the past for topics like pinctrl and gpio.

Anyone planning to attend Plumbers?

I'll be going in person but the virtual option works really well in my
experience. I had developers and maintainers attending virtually
participate in my BoF sessions and I felt it was very productive.

thanks,
drew

[1] https://lpc.events/
[2] https://lpc.events/event/17/abstracts/

Re: resctrl2 - status

[Drew Fustini]

On Fri, Sep 08, 2023 at 04:13:54PM -0700, Tony Luck wrote:
> On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:
> > Hi Tony,
> > 
> > 
> > On 9/8/2023 1:51 PM, Luck, Tony wrote:
> > > > > Can you try this out on an AMD system. I think I covered most of the
> > > > > existing AMD resctrl features, but I have no machine to test the code
> > > > > on, so very likely there are bugs in these code paths.
> > > > > 
> > > > > I'd like to make any needed changes now, before I start breaking this
> > > > > into reviewable bite-sized patches to avoid too much churn.
> > > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > > not get it to work on my AMD system.
> > > > 
> > > > # git branch -a
> > > >     next
> > > > * resctrl2_v65
> > > > # ]# uname -r
> > > > 6.5.0+
> > > > #lsmod |grep rdt
> > > > rdt_show_ids           12288  0
> > > > rdt_mbm_local_bytes    12288  0
> > > > rdt_mbm_total_bytes    12288  0
> > > > rdt_llc_occupancy      12288  0
> > > > rdt_l3_cat             16384  0
> > > > 
> > > > # lsmod |grep mbe
> > > > amd_mbec               16384  0
> > > > 
> > > > I could not get  rdt_l3_mba
> > > > 
> > > > # modprobe rdt_l3_mba
> > > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > > 
> > > > I don't see any data for the default group either.
> > > > 
> > > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > > 
> > > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > > 
> > > > cat mbm_summary
> > > >        n/a      n/a /
> > > Babu,
> > > 
> > > Thank a bunch for taking this for a quick spin. There's several bits of
> > > good news there. Several modules automatically loaded as expected.
> > > Nothing went "OOPS" and crashed the system.
> > > 
> > > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > > to load with "No such device"
> > > 
> > >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> > >                  pr_debug("No RDT allocation support\n");
> > >                  return -ENODEV;
> > >          }
> > 
> > Shouldn't this be ?(or similar)
> > 
> > if (!rdt_cpu_has(X86_FEATURE_MBA))
> >                 return false;
> 
> Yes. I should be using X86_FEATURE bits where they are available
> rather than peeking directly at CPUID register bits.
> 
> > 
> > >          mba_features = cpuid_ebx(0x10);
> > > 
> > >          if (!(mba_features & BIT(3))) {
> > >                  pr_debug("No RDT MBA allocation\n");
> > >                  return -ENODEV;
> > >          }
> > > 
> > > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > > that loaded OK). So must be the second. How does AMD enumerate MBA
> > > support?
> > > 
> > > Less obvious what is the root cause of the mbm_summary file to fail to
> > > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > > monitoring.
> > 
> > I am still not sure if resctrl2 will address all the current gaps in
> > resctrl1. We should probably list all issues on the table before we go that
> > route.
> 
> Indeed yes! I don't want to have to do resctrl3 in a few years to
> cover gaps that could have been addressed in resctrl2.
> 
> However, fixing resctrl gaps is only one of the motivations for
> the rewrite. The bigger one is making life easier for all the
> architectures sharing the common code to do what they need to
> for their own quirks & differences without cluttering the
> common code base, or worrying "did my change just break something
> for another CPU architecture".
> 
> > One of the main issue for AMD is coupling of LLC domains.
> > 
> > For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> > design assumes that there are globally 16 total CLOSIDs for the whole
> > systems. We can only create 16 CLOSID now irrespective of how many domains
> > are there.
> > 
> > In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> > in the systems.  This is more evident in AMD. But, same problem applies to
> > Intel with multiple sockets.
> 
> I think this can be somewhat achieved already with a combination of
> resctrl and cpusets (or some other way to set CPU affinity for tasks
> to only run on CPUs within a specific domain (or set of domains).
> That's why the schemata file allows setting different CBM masks
> per domain.
> 
> Can you explain how you would use 64 domains on a system with 4 domains
> and 16 CLOSID per domain?
> 
> > My 02 cents. Hope to discuss more in our upcoming meeting.
> Agreed. This will be faster when we can talk instead of type :-)

Is it a meeting that other interested developers can join?

This reminds me that Linux Plumbers Conference [1] is in November and
I think resctrl2 could be a good topic. The CFP is still open for Birds
of a Feather (BoF) proposals [2]. These are free-form get-togethers for
people wishing to discuss a particular topic, and I have had success
hosting them in the past for topics like pinctrl and gpio.

Anyone planning to attend Plumbers?

I'll be going in person but the virtual option works really well in my
experience. I had developers and maintainers attending virtually
participate in my BoF sessions and I felt it was very productive.

thanks,
drew

[1] https://lpc.events/
[2] https://lpc.events/event/17/abstracts/

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 15, 2023 at 06:16:11PM +0100, James Morse wrote:
> Hi Tony,

Hi James.

> 
> On 06/09/2023 19:21, Tony Luck wrote:
> > I've just pushed an updated set of patches to:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> > 
> > Rebased to v6.5. Includes the module auto-load code discussed in
> > previous e-mail.
> 
> I've taken a look at your resctrl2_v65rc7 branch. I don't think my reply to your first
> series got sent, sorry if I'm actually repeating myself.
> 
> It goes without saying that I'd prefer we refactor the existing filesystem into the shape
> we need, instead of a wholesale replacement.

I'm going to start this e-mail by disagreeing (but there is much
agreement to come below). I think the fastest way to get to where
we want to be (core code architecture independent with a mostly
clear line between that and the architecture specific code) is to
start over from scratch.

> 
> All this focusses on the structure of the code, but we can change that anytime. I'd prefer
> we work out what user-space needs and go from there.
> 
> I think the problems are:
>  * Discovering what the platform supports, including possible conflicting schemes.
>  * Architecture/Platform specific schema. e.g. SMBA. MPAM will have a handful of these too.
> * User-space should know which schema are portable, and which are not. e.g. L2/L3/MB.
> * Different control types for the same resource. e.g. Intel uses a percentage for
> bandwidth control, AMD an absolute number. We should standardise this sort of thing and
> make it discoverable.
> * Conflicting schemes for the same hardware. e.g. CDP and mba_MBps

Good list. In addition to the user-space needs, I'd like to also look
at the developer/maintainer needs ... which is why I think we need a
clear line between core code and architecture code.

> ~
> 
> I'd really like to keep the 'core' schema in the base driver. This is to ensure that 'L3'
> doesn't behave differently due to differing software implementation on Intel/AMD/Arm/riscv.
> 
> The L{2.3} schema were really well defined as they reference other things that are already
> visible in sysfs, this makes them nice and portable. I think this is the standard we
> should aim for with any new schema, and only resort to making them arch-specific if the
> hardware is just too strange.
> I tried to find a way to do this for SMBA as arm platforms will certainly have something
> similar, but the 'any other NUMA node' behaviour was just too specific to what was built.
> 
> I suspect this could be done with a frontend/backend split, with a 'common' directory
> holding the frontend driver for schema that apply to all architectures if the backend has
> shown up.
> 
> ~
> 
> Because I think the types a schema is configured with should be portable across
> architectures, I'd prefer the string parsing code lives in the core code, and isn't
> duplicated by every submodule! String parsing in the kernel is bad enough!
> 
> The obvious types we have so far are: bitmap/percentage/number/bandwidth. I think it makes
> sense for user-space to be told what the schema is configured with, and let the core code
> do that parsing.

My PoC code went full throttle on pushing all the schemata code to the
driver. But you are right that this may be too drastic.  Drivers could
just indicate that they need one of the basic types "bitmap, percentage,
number, bandwidth" together with some parameters (number of bits for
bitmap, granularity for percentage, high/low values for number, ditto
for bandwidth) and have core code handle the inital parsing. Then give
to the module for validation (e.g. for the Intel requirement that
cache bit masks have contiguous "1" bits), and some CPU models require
that a minimum of two bits are set). Also pass to the module to apply
(wrmsr() on a CPU in the domain for x86, MMIO from anywhere for ARM).

For bitmasks - MPAM seems to allow very long masks. Perhaps the kernel
can make use of the "%*pb" format to scanf/printk to cope with those?
It would mean commas in the masks like 0000,00000fff,ffffff00,0000000f

> ~
> 
> I don't have a good answer for conflicting drivers for the same hardware resource. I think
> we'd need to find a way of making the existing command-line arguments work, causing the
> corresponding module to auto-load.
> 
> But this sucks for distros, who would need somewhere to describe the policy of which
> modules get loaded. The good news is things like libvirt aren't mounting resctrl themselves.

This is where I don't think we need to perfectly emulate the system
administrator interface. People deploying resctrl features have to
understand the features. This level of QoS control isn't going to
be a "one-size-fits-all" proposition (even between different generations
of the same CPU architecture, much less across different architectures.

> ~
> 
> I suspect the CDP schemes should be made arch-specific as x86 needs the odd/even pairing,
> whereas MPAM does not. This would allow a scheme where the Iside CLOSIDs can be shared to
> avoid halving the available CLOSID.
> 
> Having somewhere sensible to put the MPAM driver is useful. It's currently dumped in
> drivers/platform as it doesn't really fit anywhere!
> 
> Allowing each submodule to add extra files to the info directories looks useful. MPAM's
> priority control has a property to say whether 0 is the top of the scale or not, this
> would allow it to be exposed to user-space, instead of having to shift the range up/down
> to hide the difference in behaviour.

I'm contemplating allowing submodules to have one extra level of
directory structure below info. E.g. so they could make
info/FEATURE/socket{0,1,...}/param{1,2,...} as I have a feature
that may be easier to describe this way.

> MPAM only needs to update the hardware on a CPU that is part of target domain on some
> platforms. The vast majority would allow the MMIO writes to come from anywhere. Having the
> applychanges behaviour specific to the submodule would reduce the number of IPI.
> >From what I've seen, riscv never needs an IPI here.

See above. The "applychanges" phase of a schemata update is necessarily
CPU architecuture specific. IPI or not is an architecture decision.

> It looks like all the core code runs at process context without touching the pre-empt
> counter or masking interrupts - this is a really nice property. Most of my headaches have
> come from this area.
> 
> The limbo work isn't arch specific, that is a property of caches, it should really be core
> code behaviour to avoid duplication. MPAM needs that, I expect riscv does too.

Yes. I'm not super happy with how much of the monitor code I stuffed
down into resctrl2/arch/x86. Partly it was done because of how many
bits need to be shared by different modules (e.g. the MBmpbs module
needs to read the MBM counters).

But note that if the user doesn't care about cache occupancy monitoring,
they can skip loading the module that reports that, and then they can
skip the whole limbo dance with periodic interrupts sampling cache
occupancy until the monitor counter gets to a low enough value to put
it back on the free list.

So while the code might be in the core, it should be aware of whether
it needs to do anything.

> Making the CLOSID/RMID allocation behaviour arch specific saves some headaches. MPAM is
> particularly different in this area. I don't know what riscv's behaviour is here.

Heh! This was my initial motivation for the rewrite. I saw where your
patches had to pass closid + rmid around to functions that only needed
one or the other on X86 and thought "we should just have a neutral
"resctrl_ids_t" typedef for that (I'm moving away from "u64" to this
typedef as I continue work on this).
> 
> 
> 
> > James: 
> > 
> > I'm now hoping for some feedback from ARM folks on whether this is a
> > useful direction. Is it possible to implement MPAM features on top of
> > this base architecture independent layer. If not, am I just missing
> > some simple extensions that you will need. Or is there some fundamental
> > problem that looks hard/impossible to resolve?
> 
> 
> You've got an rdt_l2_cat driver which is really a symlink to rdt_l3_cat built with
> different pre-processor defines. It's a good trick, but it only works because you know the
> full range of hardware that has been built.
> MPAM can't do things like this, it really is a bag of bits, and it isn't until the ACPI
> tables are parsed to find where the resources are - and then that hardware is probed -
> that we have a clue what was built.
> For example, the cache controls could be on any cache in the hierarchy, and the number is
> derived from the structure of the PPTT table. You've already got a fair amount of
> duplication when multiple struct resctrl_resources are defined. MPAM would have to
> allocate and populate these dynamically to avoid building a module for 'L5' ... just in
> case someone built that...

In my code that's an arch/x86 driver. You can do something completely
different for ARM/MPAM that meets your needs. Perhaps that's easier to
see with the changes proposed above where core code handles the basic
parsing of schemata bitmasks/numbers/...

> The mba_MBps logic is really just software, I see no reason to duplicate it on multiple
> architectures. This should be in the core of the filesystem. We already have a mount
> option to enable it.

Maybe. But it needs access to both the monitoring data, and the MBA
control registers. Would need suitable interfaces between arch specific
code that can get that data and tweak those control registers.

> I see the arch/submodules can't influence the domain numbers ... this is a problem as it
> hardcodes "you can have L2 or L3", which is something you were trying to avoid. MPAM will
> want to use NUMA-IDs for memory-side-caches, (I'd hope this to be a core/portable schema),
> as well as IOMMU-IDs for the I/O side of this.
> I don't think this is really a problem, as I'd like to add the things I need in this area
> as core/portable schema.

The modules can supply a "scope" which could be any of CPU, core,
module, socket, or tied to some specific cache domain level. If
there is something more exotic, we could add a callback function
so the module could provide its own custom CPU->domain mapping.

> Arm's IOMMU has support to label traffic with the equivalent to CLOSID/RMID. My prototype
> support for this adds IOMMU-groups to the resctrl tasks file so that the devices can be
> moved between control/monitor groups as if they were tasks. I think this would work for
> other architectures if they had similar support as IOMMU-groups are an existing concept.

So "tasks" accepts and reports non-numbers for these IOMMU-groups? And
those tie to something in /sys that tells the user which set of devices
they are monitoring/controlling?

> 
> Thanks,
> 
> James

Thanks for your detailed reply. It's given me a lot to think about.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 15, 2023 at 06:16:11PM +0100, James Morse wrote:
> Hi Tony,

Hi James.

> 
> On 06/09/2023 19:21, Tony Luck wrote:
> > I've just pushed an updated set of patches to:
> > 
> >   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> > 
> > Rebased to v6.5. Includes the module auto-load code discussed in
> > previous e-mail.
> 
> I've taken a look at your resctrl2_v65rc7 branch. I don't think my reply to your first
> series got sent, sorry if I'm actually repeating myself.
> 
> It goes without saying that I'd prefer we refactor the existing filesystem into the shape
> we need, instead of a wholesale replacement.

I'm going to start this e-mail by disagreeing (but there is much
agreement to come below). I think the fastest way to get to where
we want to be (core code architecture independent with a mostly
clear line between that and the architecture specific code) is to
start over from scratch.

> 
> All this focusses on the structure of the code, but we can change that anytime. I'd prefer
> we work out what user-space needs and go from there.
> 
> I think the problems are:
>  * Discovering what the platform supports, including possible conflicting schemes.
>  * Architecture/Platform specific schema. e.g. SMBA. MPAM will have a handful of these too.
> * User-space should know which schema are portable, and which are not. e.g. L2/L3/MB.
> * Different control types for the same resource. e.g. Intel uses a percentage for
> bandwidth control, AMD an absolute number. We should standardise this sort of thing and
> make it discoverable.
> * Conflicting schemes for the same hardware. e.g. CDP and mba_MBps

Good list. In addition to the user-space needs, I'd like to also look
at the developer/maintainer needs ... which is why I think we need a
clear line between core code and architecture code.

> ~
> 
> I'd really like to keep the 'core' schema in the base driver. This is to ensure that 'L3'
> doesn't behave differently due to differing software implementation on Intel/AMD/Arm/riscv.
> 
> The L{2.3} schema were really well defined as they reference other things that are already
> visible in sysfs, this makes them nice and portable. I think this is the standard we
> should aim for with any new schema, and only resort to making them arch-specific if the
> hardware is just too strange.
> I tried to find a way to do this for SMBA as arm platforms will certainly have something
> similar, but the 'any other NUMA node' behaviour was just too specific to what was built.
> 
> I suspect this could be done with a frontend/backend split, with a 'common' directory
> holding the frontend driver for schema that apply to all architectures if the backend has
> shown up.
> 
> ~
> 
> Because I think the types a schema is configured with should be portable across
> architectures, I'd prefer the string parsing code lives in the core code, and isn't
> duplicated by every submodule! String parsing in the kernel is bad enough!
> 
> The obvious types we have so far are: bitmap/percentage/number/bandwidth. I think it makes
> sense for user-space to be told what the schema is configured with, and let the core code
> do that parsing.

My PoC code went full throttle on pushing all the schemata code to the
driver. But you are right that this may be too drastic.  Drivers could
just indicate that they need one of the basic types "bitmap, percentage,
number, bandwidth" together with some parameters (number of bits for
bitmap, granularity for percentage, high/low values for number, ditto
for bandwidth) and have core code handle the inital parsing. Then give
to the module for validation (e.g. for the Intel requirement that
cache bit masks have contiguous "1" bits), and some CPU models require
that a minimum of two bits are set). Also pass to the module to apply
(wrmsr() on a CPU in the domain for x86, MMIO from anywhere for ARM).

For bitmasks - MPAM seems to allow very long masks. Perhaps the kernel
can make use of the "%*pb" format to scanf/printk to cope with those?
It would mean commas in the masks like 0000,00000fff,ffffff00,0000000f

> ~
> 
> I don't have a good answer for conflicting drivers for the same hardware resource. I think
> we'd need to find a way of making the existing command-line arguments work, causing the
> corresponding module to auto-load.
> 
> But this sucks for distros, who would need somewhere to describe the policy of which
> modules get loaded. The good news is things like libvirt aren't mounting resctrl themselves.

This is where I don't think we need to perfectly emulate the system
administrator interface. People deploying resctrl features have to
understand the features. This level of QoS control isn't going to
be a "one-size-fits-all" proposition (even between different generations
of the same CPU architecture, much less across different architectures.

> ~
> 
> I suspect the CDP schemes should be made arch-specific as x86 needs the odd/even pairing,
> whereas MPAM does not. This would allow a scheme where the Iside CLOSIDs can be shared to
> avoid halving the available CLOSID.
> 
> Having somewhere sensible to put the MPAM driver is useful. It's currently dumped in
> drivers/platform as it doesn't really fit anywhere!
> 
> Allowing each submodule to add extra files to the info directories looks useful. MPAM's
> priority control has a property to say whether 0 is the top of the scale or not, this
> would allow it to be exposed to user-space, instead of having to shift the range up/down
> to hide the difference in behaviour.

I'm contemplating allowing submodules to have one extra level of
directory structure below info. E.g. so they could make
info/FEATURE/socket{0,1,...}/param{1,2,...} as I have a feature
that may be easier to describe this way.

> MPAM only needs to update the hardware on a CPU that is part of target domain on some
> platforms. The vast majority would allow the MMIO writes to come from anywhere. Having the
> applychanges behaviour specific to the submodule would reduce the number of IPI.
> >From what I've seen, riscv never needs an IPI here.

See above. The "applychanges" phase of a schemata update is necessarily
CPU architecuture specific. IPI or not is an architecture decision.

> It looks like all the core code runs at process context without touching the pre-empt
> counter or masking interrupts - this is a really nice property. Most of my headaches have
> come from this area.
> 
> The limbo work isn't arch specific, that is a property of caches, it should really be core
> code behaviour to avoid duplication. MPAM needs that, I expect riscv does too.

Yes. I'm not super happy with how much of the monitor code I stuffed
down into resctrl2/arch/x86. Partly it was done because of how many
bits need to be shared by different modules (e.g. the MBmpbs module
needs to read the MBM counters).

But note that if the user doesn't care about cache occupancy monitoring,
they can skip loading the module that reports that, and then they can
skip the whole limbo dance with periodic interrupts sampling cache
occupancy until the monitor counter gets to a low enough value to put
it back on the free list.

So while the code might be in the core, it should be aware of whether
it needs to do anything.

> Making the CLOSID/RMID allocation behaviour arch specific saves some headaches. MPAM is
> particularly different in this area. I don't know what riscv's behaviour is here.

Heh! This was my initial motivation for the rewrite. I saw where your
patches had to pass closid + rmid around to functions that only needed
one or the other on X86 and thought "we should just have a neutral
"resctrl_ids_t" typedef for that (I'm moving away from "u64" to this
typedef as I continue work on this).
> 
> 
> 
> > James: 
> > 
> > I'm now hoping for some feedback from ARM folks on whether this is a
> > useful direction. Is it possible to implement MPAM features on top of
> > this base architecture independent layer. If not, am I just missing
> > some simple extensions that you will need. Or is there some fundamental
> > problem that looks hard/impossible to resolve?
> 
> 
> You've got an rdt_l2_cat driver which is really a symlink to rdt_l3_cat built with
> different pre-processor defines. It's a good trick, but it only works because you know the
> full range of hardware that has been built.
> MPAM can't do things like this, it really is a bag of bits, and it isn't until the ACPI
> tables are parsed to find where the resources are - and then that hardware is probed -
> that we have a clue what was built.
> For example, the cache controls could be on any cache in the hierarchy, and the number is
> derived from the structure of the PPTT table. You've already got a fair amount of
> duplication when multiple struct resctrl_resources are defined. MPAM would have to
> allocate and populate these dynamically to avoid building a module for 'L5' ... just in
> case someone built that...

In my code that's an arch/x86 driver. You can do something completely
different for ARM/MPAM that meets your needs. Perhaps that's easier to
see with the changes proposed above where core code handles the basic
parsing of schemata bitmasks/numbers/...

> The mba_MBps logic is really just software, I see no reason to duplicate it on multiple
> architectures. This should be in the core of the filesystem. We already have a mount
> option to enable it.

Maybe. But it needs access to both the monitoring data, and the MBA
control registers. Would need suitable interfaces between arch specific
code that can get that data and tweak those control registers.

> I see the arch/submodules can't influence the domain numbers ... this is a problem as it
> hardcodes "you can have L2 or L3", which is something you were trying to avoid. MPAM will
> want to use NUMA-IDs for memory-side-caches, (I'd hope this to be a core/portable schema),
> as well as IOMMU-IDs for the I/O side of this.
> I don't think this is really a problem, as I'd like to add the things I need in this area
> as core/portable schema.

The modules can supply a "scope" which could be any of CPU, core,
module, socket, or tied to some specific cache domain level. If
there is something more exotic, we could add a callback function
so the module could provide its own custom CPU->domain mapping.

> Arm's IOMMU has support to label traffic with the equivalent to CLOSID/RMID. My prototype
> support for this adds IOMMU-groups to the resctrl tasks file so that the devices can be
> moved between control/monitor groups as if they were tasks. I think this would work for
> other architectures if they had similar support as IOMMU-groups are an existing concept.

So "tasks" accepts and reports non-numbers for these IOMMU-groups? And
those tie to something in /sys that tells the user which set of devices
they are monitoring/controlling?

> 
> Thanks,
> 
> James

Thanks for your detailed reply. It's given me a lot to think about.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Jonathan Cameron]

On Fri, 15 Sep 2023 10:55:58 -0700
Drew Fustini <dfustini@baylibre.com> wrote:

> On Fri, Sep 08, 2023 at 04:13:54PM -0700, Tony Luck wrote:
> > On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:  
> > > Hi Tony,
> > > 
> > > 
> > > On 9/8/2023 1:51 PM, Luck, Tony wrote:  
> > > > > > Can you try this out on an AMD system. I think I covered most of the
> > > > > > existing AMD resctrl features, but I have no machine to test the code
> > > > > > on, so very likely there are bugs in these code paths.
> > > > > > 
> > > > > > I'd like to make any needed changes now, before I start breaking this
> > > > > > into reviewable bite-sized patches to avoid too much churn.  
> > > > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > > > not get it to work on my AMD system.
> > > > > 
> > > > > # git branch -a
> > > > >     next
> > > > > * resctrl2_v65
> > > > > # ]# uname -r
> > > > > 6.5.0+
> > > > > #lsmod |grep rdt
> > > > > rdt_show_ids           12288  0
> > > > > rdt_mbm_local_bytes    12288  0
> > > > > rdt_mbm_total_bytes    12288  0
> > > > > rdt_llc_occupancy      12288  0
> > > > > rdt_l3_cat             16384  0
> > > > > 
> > > > > # lsmod |grep mbe
> > > > > amd_mbec               16384  0
> > > > > 
> > > > > I could not get  rdt_l3_mba
> > > > > 
> > > > > # modprobe rdt_l3_mba
> > > > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > > > 
> > > > > I don't see any data for the default group either.
> > > > > 
> > > > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > > > 
> > > > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > > > 
> > > > > cat mbm_summary
> > > > >        n/a      n/a /  
> > > > Babu,
> > > > 
> > > > Thank a bunch for taking this for a quick spin. There's several bits of
> > > > good news there. Several modules automatically loaded as expected.
> > > > Nothing went "OOPS" and crashed the system.
> > > > 
> > > > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > > > to load with "No such device"
> > > > 
> > > >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> > > >                  pr_debug("No RDT allocation support\n");
> > > >                  return -ENODEV;
> > > >          }  
> > > 
> > > Shouldn't this be ?(or similar)
> > > 
> > > if (!rdt_cpu_has(X86_FEATURE_MBA))
> > >                 return false;  
> > 
> > Yes. I should be using X86_FEATURE bits where they are available
> > rather than peeking directly at CPUID register bits.
> >   
> > >   
> > > >          mba_features = cpuid_ebx(0x10);
> > > > 
> > > >          if (!(mba_features & BIT(3))) {
> > > >                  pr_debug("No RDT MBA allocation\n");
> > > >                  return -ENODEV;
> > > >          }
> > > > 
> > > > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > > > that loaded OK). So must be the second. How does AMD enumerate MBA
> > > > support?
> > > > 
> > > > Less obvious what is the root cause of the mbm_summary file to fail to
> > > > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > > > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > > > monitoring.  
> > > 
> > > I am still not sure if resctrl2 will address all the current gaps in
> > > resctrl1. We should probably list all issues on the table before we go that
> > > route.  
> > 
> > Indeed yes! I don't want to have to do resctrl3 in a few years to
> > cover gaps that could have been addressed in resctrl2.
> > 
> > However, fixing resctrl gaps is only one of the motivations for
> > the rewrite. The bigger one is making life easier for all the
> > architectures sharing the common code to do what they need to
> > for their own quirks & differences without cluttering the
> > common code base, or worrying "did my change just break something
> > for another CPU architecture".
> >   
> > > One of the main issue for AMD is coupling of LLC domains.
> > > 
> > > For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> > > design assumes that there are globally 16 total CLOSIDs for the whole
> > > systems. We can only create 16 CLOSID now irrespective of how many domains
> > > are there.
> > > 
> > > In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> > > in the systems.  This is more evident in AMD. But, same problem applies to
> > > Intel with multiple sockets.  
> > 
> > I think this can be somewhat achieved already with a combination of
> > resctrl and cpusets (or some other way to set CPU affinity for tasks
> > to only run on CPUs within a specific domain (or set of domains).
> > That's why the schemata file allows setting different CBM masks
> > per domain.
> > 
> > Can you explain how you would use 64 domains on a system with 4 domains
> > and 16 CLOSID per domain?
> >   
> > > My 02 cents. Hope to discuss more in our upcoming meeting.  
> > Agreed. This will be faster when we can talk instead of type :-)  
> 
> Is it a meeting that other interested developers can join?
> 
> This reminds me that Linux Plumbers Conference [1] is in November and
> I think resctrl2 could be a good topic. The CFP is still open for Birds
> of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> people wishing to discuss a particular topic, and I have had success
> hosting them in the past for topics like pinctrl and gpio.
> 
> Anyone planning to attend Plumbers?
> 
> I'll be going in person but the virtual option works really well in my
> experience. I had developers and maintainers attending virtually
> participate in my BoF sessions and I felt it was very productive.

FWIW I'm keen and should be there in person.  However, I'm not on the must
be available list for this one ;)   Agree that hybrid worked fine for BoF last
year.

Jonathan


> 
> thanks,
> drew
> 
> [1] https://lpc.events/
> [2] https://lpc.events/event/17/abstracts/
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Jonathan Cameron]

On Fri, 15 Sep 2023 10:55:58 -0700
Drew Fustini <dfustini@baylibre.com> wrote:

> On Fri, Sep 08, 2023 at 04:13:54PM -0700, Tony Luck wrote:
> > On Fri, Sep 08, 2023 at 04:35:05PM -0500, Moger, Babu wrote:  
> > > Hi Tony,
> > > 
> > > 
> > > On 9/8/2023 1:51 PM, Luck, Tony wrote:  
> > > > > > Can you try this out on an AMD system. I think I covered most of the
> > > > > > existing AMD resctrl features, but I have no machine to test the code
> > > > > > on, so very likely there are bugs in these code paths.
> > > > > > 
> > > > > > I'd like to make any needed changes now, before I start breaking this
> > > > > > into reviewable bite-sized patches to avoid too much churn.  
> > > > > I tried your latest code briefly on my system.  Unfortunately, I could
> > > > > not get it to work on my AMD system.
> > > > > 
> > > > > # git branch -a
> > > > >     next
> > > > > * resctrl2_v65
> > > > > # ]# uname -r
> > > > > 6.5.0+
> > > > > #lsmod |grep rdt
> > > > > rdt_show_ids           12288  0
> > > > > rdt_mbm_local_bytes    12288  0
> > > > > rdt_mbm_total_bytes    12288  0
> > > > > rdt_llc_occupancy      12288  0
> > > > > rdt_l3_cat             16384  0
> > > > > 
> > > > > # lsmod |grep mbe
> > > > > amd_mbec               16384  0
> > > > > 
> > > > > I could not get  rdt_l3_mba
> > > > > 
> > > > > # modprobe rdt_l3_mba
> > > > > modprobe: ERROR: could not insert 'rdt_l3_mba': No such device
> > > > > 
> > > > > I don't see any data for the default group either.
> > > > > 
> > > > > mount  -t resctrl resctrl /sys/fs/resctrl/
> > > > > 
> > > > > cd /sys/fs/resctrl/mon_data/mon_L3_00
> > > > > 
> > > > > cat mbm_summary
> > > > >        n/a      n/a /  
> > > > Babu,
> > > > 
> > > > Thank a bunch for taking this for a quick spin. There's several bits of
> > > > good news there. Several modules automatically loaded as expected.
> > > > Nothing went "OOPS" and crashed the system.
> > > > 
> > > > Here’s the code that the rdt_l3_mba module runs that can cause failure
> > > > to load with "No such device"
> > > > 
> > > >          if (!boot_cpu_has(X86_FEATURE_RDT_A)) {
> > > >                  pr_debug("No RDT allocation support\n");
> > > >                  return -ENODEV;
> > > >          }  
> > > 
> > > Shouldn't this be ?(or similar)
> > > 
> > > if (!rdt_cpu_has(X86_FEATURE_MBA))
> > >                 return false;  
> > 
> > Yes. I should be using X86_FEATURE bits where they are available
> > rather than peeking directly at CPUID register bits.
> >   
> > >   
> > > >          mba_features = cpuid_ebx(0x10);
> > > > 
> > > >          if (!(mba_features & BIT(3))) {
> > > >                  pr_debug("No RDT MBA allocation\n");
> > > >                  return -ENODEV;
> > > >          }
> > > > 
> > > > I assume the first test must have succeeded (same code in rdt_l3_cat, and
> > > > that loaded OK). So must be the second. How does AMD enumerate MBA
> > > > support?
> > > > 
> > > > Less obvious what is the root cause of the mbm_summary file to fail to
> > > > show any data. rdt_mbm_local_bytes  and rdt_mbm_total_bytes  modules
> > > > loaded OK. So I'm looking for the right CPUID bits to detect memory bandwidth
> > > > monitoring.  
> > > 
> > > I am still not sure if resctrl2 will address all the current gaps in
> > > resctrl1. We should probably list all issues on the table before we go that
> > > route.  
> > 
> > Indeed yes! I don't want to have to do resctrl3 in a few years to
> > cover gaps that could have been addressed in resctrl2.
> > 
> > However, fixing resctrl gaps is only one of the motivations for
> > the rewrite. The bigger one is making life easier for all the
> > architectures sharing the common code to do what they need to
> > for their own quirks & differences without cluttering the
> > common code base, or worrying "did my change just break something
> > for another CPU architecture".
> >   
> > > One of the main issue for AMD is coupling of LLC domains.
> > > 
> > > For example, AMD hardware supports 16 CLOSids per LLC domain. But Linux
> > > design assumes that there are globally 16 total CLOSIDs for the whole
> > > systems. We can only create 16 CLOSID now irrespective of how many domains
> > > are there.
> > > 
> > > In reality, we should be able to create "16 x number of LLC domains" CLOSIDS
> > > in the systems.  This is more evident in AMD. But, same problem applies to
> > > Intel with multiple sockets.  
> > 
> > I think this can be somewhat achieved already with a combination of
> > resctrl and cpusets (or some other way to set CPU affinity for tasks
> > to only run on CPUs within a specific domain (or set of domains).
> > That's why the schemata file allows setting different CBM masks
> > per domain.
> > 
> > Can you explain how you would use 64 domains on a system with 4 domains
> > and 16 CLOSID per domain?
> >   
> > > My 02 cents. Hope to discuss more in our upcoming meeting.  
> > Agreed. This will be faster when we can talk instead of type :-)  
> 
> Is it a meeting that other interested developers can join?
> 
> This reminds me that Linux Plumbers Conference [1] is in November and
> I think resctrl2 could be a good topic. The CFP is still open for Birds
> of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> people wishing to discuss a particular topic, and I have had success
> hosting them in the past for topics like pinctrl and gpio.
> 
> Anyone planning to attend Plumbers?
> 
> I'll be going in person but the virtual option works really well in my
> experience. I had developers and maintainers attending virtually
> participate in my BoF sessions and I felt it was very productive.

FWIW I'm keen and should be there in person.  However, I'm not on the must
be available list for this one ;)   Agree that hybrid worked fine for BoF last
year.

Jonathan


> 
> thanks,
> drew
> 
> [1] https://lpc.events/
> [2] https://lpc.events/event/17/abstracts/
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Peter Newman]

Hi Tony,

On Wed, Sep 6, 2023 at 8:21 PM Tony Luck <tony.luck@intel.com> wrote:
> I've just pushed an updated set of patches to:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65

I'm trying to understand the purpose of the resctrl_resource struct
defined here:

https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/tree/include/linux/resctrl.h?h=resctrl2_v65#n356

From the common fields, it seems to be something with a name and some
info files that can be divided into domains and should be told when
CPUs are hotplugged. (I skipped the scope field because I couldn't
find any references outside fs/resctrl2/arch). The remaining fields
are explicitly disjoint depending on whether we're talking about
monitoring or allocation.

From this I don't get a strong sense that a "resource" is really a
thing and I think James's resctrl_schema struct only for the purpose
of resource allocation was more the right direction for common code.
Outwardly, resctrl groups seem to be the only thing relating
monitoring to allocation.

Is there a good reason for the common code to understand relationships
between monitors and schemas with the same name? If not, I think it
would make more sense to register monitors and control schemata
independently.

-Peter

Re: resctrl2 - status

[Peter Newman]

Hi Tony,

On Wed, Sep 6, 2023 at 8:21 PM Tony Luck <tony.luck@intel.com> wrote:
> I've just pushed an updated set of patches to:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65

I'm trying to understand the purpose of the resctrl_resource struct
defined here:

https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/tree/include/linux/resctrl.h?h=resctrl2_v65#n356

From the common fields, it seems to be something with a name and some
info files that can be divided into domains and should be told when
CPUs are hotplugged. (I skipped the scope field because I couldn't
find any references outside fs/resctrl2/arch). The remaining fields
are explicitly disjoint depending on whether we're talking about
monitoring or allocation.

From this I don't get a strong sense that a "resource" is really a
thing and I think James's resctrl_schema struct only for the purpose
of resource allocation was more the right direction for common code.
Outwardly, resctrl groups seem to be the only thing relating
monitoring to allocation.

Is there a good reason for the common code to understand relationships
between monitors and schemas with the same name? If not, I think it
would make more sense to register monitors and control schemata
independently.

-Peter

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Tue, Sep 19, 2023 at 02:53:07PM +0200, Peter Newman wrote:
> Hi Tony,
> 
> On Wed, Sep 6, 2023 at 8:21 PM Tony Luck <tony.luck@intel.com> wrote:
> > I've just pushed an updated set of patches to:
> >
> >   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> 
> I'm trying to understand the purpose of the resctrl_resource struct
> defined here:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/tree/include/linux/resctrl.h?h=resctrl2_v65#n356
> 
> >From the common fields, it seems to be something with a name and some
> info files that can be divided into domains and should be told when
> CPUs are hotplugged. (I skipped the scope field because I couldn't
> find any references outside fs/resctrl2/arch). The remaining fields
> are explicitly disjoint depending on whether we're talking about
> monitoring or allocation.
> 
> >From this I don't get a strong sense that a "resource" is really a
> thing and I think James's resctrl_schema struct only for the purpose
> of resource allocation was more the right direction for common code.
> Outwardly, resctrl groups seem to be the only thing relating
> monitoring to allocation.
> 

Peter,

It's a good question. I started out with separate control_resource and
monitor_resource structures, but combined them early on when it was
looking like the overall size wasn't all that big (for a structure that
will only have a dozen or so instances) and there were a bunch of common
fields. It would be easy to separate them again if consensus is that is
cleaner. There are only two places (currently) where code walks
resources of all types (mount and unmount). So no big hassle to replace
those with separate for_each_control_resource() and for_each_monitor_resource()

> I skipped the scope field ...

The scope field is for the module to tell the core code the granularity
of the control/monitor resource so it can build a custom domain list
based on L2 cache scope, L3 cache scope, NUMA ndoe scope, core scope,
or anything else that h/w folks dream up.  This means only the common
code needs to register a CPU hotplug notifier.

Note that there is no sharing of domains between modules to allow each
module to be fully independent of any other module. This also means
that the domain structure can have a common header, with some module
specific data allocated directly after ... though the need for that
might be going away as I implement James suggestion to keep the common
schemata parsing in the generic code.

> Is there a good reason for the common code to understand relationships
> between monitors and schemas with the same name? If not, I think it
> would make more sense to register monitors and control schemata
> independently.

I'm going back through my code now, breaking it into a patch series
that builds a piece at a time. One of many things to become clear as
I work through this is that the ".name" field in the resource isn't
really useful. It may disappear to be replaced more specific
fields based on usage. There's already one for the name of the "info/"
directory for the resource. I'm adding a "schemata_name" for control
resources. When I get to the monitor section I'll likley add a
field that gets used to construct the names of directories under
the "mon_data/" directories instead of using ".name". Doing that
would remove any apparent relationship between monitors and
schemas. There's no reason on X86 for them to be connected.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Tue, Sep 19, 2023 at 02:53:07PM +0200, Peter Newman wrote:
> Hi Tony,
> 
> On Wed, Sep 6, 2023 at 8:21 PM Tony Luck <tony.luck@intel.com> wrote:
> > I've just pushed an updated set of patches to:
> >
> >   git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl_v65
> 
> I'm trying to understand the purpose of the resctrl_resource struct
> defined here:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git/tree/include/linux/resctrl.h?h=resctrl2_v65#n356
> 
> >From the common fields, it seems to be something with a name and some
> info files that can be divided into domains and should be told when
> CPUs are hotplugged. (I skipped the scope field because I couldn't
> find any references outside fs/resctrl2/arch). The remaining fields
> are explicitly disjoint depending on whether we're talking about
> monitoring or allocation.
> 
> >From this I don't get a strong sense that a "resource" is really a
> thing and I think James's resctrl_schema struct only for the purpose
> of resource allocation was more the right direction for common code.
> Outwardly, resctrl groups seem to be the only thing relating
> monitoring to allocation.
> 

Peter,

It's a good question. I started out with separate control_resource and
monitor_resource structures, but combined them early on when it was
looking like the overall size wasn't all that big (for a structure that
will only have a dozen or so instances) and there were a bunch of common
fields. It would be easy to separate them again if consensus is that is
cleaner. There are only two places (currently) where code walks
resources of all types (mount and unmount). So no big hassle to replace
those with separate for_each_control_resource() and for_each_monitor_resource()

> I skipped the scope field ...

The scope field is for the module to tell the core code the granularity
of the control/monitor resource so it can build a custom domain list
based on L2 cache scope, L3 cache scope, NUMA ndoe scope, core scope,
or anything else that h/w folks dream up.  This means only the common
code needs to register a CPU hotplug notifier.

Note that there is no sharing of domains between modules to allow each
module to be fully independent of any other module. This also means
that the domain structure can have a common header, with some module
specific data allocated directly after ... though the need for that
might be going away as I implement James suggestion to keep the common
schemata parsing in the generic code.

> Is there a good reason for the common code to understand relationships
> between monitors and schemas with the same name? If not, I think it
> would make more sense to register monitors and control schemata
> independently.

I'm going back through my code now, breaking it into a patch series
that builds a piece at a time. One of many things to become clear as
I work through this is that the ".name" field in the resource isn't
really useful. It may disappear to be replaced more specific
fields based on usage. There's already one for the name of the "info/"
directory for the resource. I'm adding a "schemata_name" for control
resources. When I get to the monitor section I'll likley add a
field that gets used to construct the names of directories under
the "mon_data/" directories instead of using ".name". Doing that
would remove any apparent relationship between monitors and
schemas. There's no reason on X86 for them to be connected.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 15, 2023 at 01:38:16PM -0700, Tony Luck wrote:
> My PoC code went full throttle on pushing all the schemata code to the
> driver. But you are right that this may be too drastic.  Drivers could
> just indicate that they need one of the basic types "bitmap, percentage,
> number, bandwidth" together with some parameters (number of bits for
> bitmap, granularity for percentage, high/low values for number, ditto
> for bandwidth) and have core code handle the inital parsing. Then give
> to the module for validation (e.g. for the Intel requirement that
> cache bit masks have contiguous "1" bits), and some CPU models require
> that a minimum of two bits are set). Also pass to the module to apply
> (wrmsr() on a CPU in the domain for x86, MMIO from anywhere for ARM).
> 
> For bitmasks - MPAM seems to allow very long masks. Perhaps the kernel
> can make use of the "%*pb" format to scanf/printk to cope with those?
> It would mean commas in the masks like 0000,00000fff,ffffff00,0000000f

James,

Your earlier e-mail came at a fortuitous moment. I am breaking my
giant patches into smaller pieces. I had just got to the "add the
schemata file" section of the series. I took your advice and put
the majority of the parsing code into the core, leaving the arch
specific modules just a hook to handle weird quirks.

Modules specify the "type" of their schemata entries, together with
a parameter that the core can use. I implemented two types so far.
But should be easy to add additional types like "bandwidth" as you
had suggested.

1) Bitmask. Parameter is the number of bits.
2) Unsigned long. Parameter is the max value.

Examples of quirks:
1) Intel L3 CAT requires all "1" bits to be in a consecutive block.
   In this case core code will accept any bitmask that doesn't exceed
   the number of bits. The module can reject if the user doesn't
   follow the consecutive bits rule.
2) Intel L3 MBA requires that throttle percentage values be a multiple of the
   h/w enumerated granularity (e.g. 10%). Here the module does a
   roundup() to quietly fix invalid input (same as legacy resctrl).

I've only got through the "control" section of converting to patches.
But I think this is well past the halfway point.

Current snapshot of code is here:


git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_patches


* Comes with just the rdt_l3_cat and rdt_l3_mba modules.
  Without the AMD support bits at this point.
* No "cpus", "mode", "size" files yet.
* Just CTRL_MON directories.
* No monitoring support yet.

-Tony

Re: resctrl2 - status

[Tony Luck]

On Fri, Sep 15, 2023 at 01:38:16PM -0700, Tony Luck wrote:
> My PoC code went full throttle on pushing all the schemata code to the
> driver. But you are right that this may be too drastic.  Drivers could
> just indicate that they need one of the basic types "bitmap, percentage,
> number, bandwidth" together with some parameters (number of bits for
> bitmap, granularity for percentage, high/low values for number, ditto
> for bandwidth) and have core code handle the inital parsing. Then give
> to the module for validation (e.g. for the Intel requirement that
> cache bit masks have contiguous "1" bits), and some CPU models require
> that a minimum of two bits are set). Also pass to the module to apply
> (wrmsr() on a CPU in the domain for x86, MMIO from anywhere for ARM).
> 
> For bitmasks - MPAM seems to allow very long masks. Perhaps the kernel
> can make use of the "%*pb" format to scanf/printk to cope with those?
> It would mean commas in the masks like 0000,00000fff,ffffff00,0000000f

James,

Your earlier e-mail came at a fortuitous moment. I am breaking my
giant patches into smaller pieces. I had just got to the "add the
schemata file" section of the series. I took your advice and put
the majority of the parsing code into the core, leaving the arch
specific modules just a hook to handle weird quirks.

Modules specify the "type" of their schemata entries, together with
a parameter that the core can use. I implemented two types so far.
But should be easy to add additional types like "bandwidth" as you
had suggested.

1) Bitmask. Parameter is the number of bits.
2) Unsigned long. Parameter is the max value.

Examples of quirks:
1) Intel L3 CAT requires all "1" bits to be in a consecutive block.
   In this case core code will accept any bitmask that doesn't exceed
   the number of bits. The module can reject if the user doesn't
   follow the consecutive bits rule.
2) Intel L3 MBA requires that throttle percentage values be a multiple of the
   h/w enumerated granularity (e.g. 10%). Here the module does a
   roundup() to quietly fix invalid input (same as legacy resctrl).

I've only got through the "control" section of converting to patches.
But I think this is well past the halfway point.

Current snapshot of code is here:


git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux.git resctrl2_patches


* Comes with just the rdt_l3_cat and rdt_l3_mba modules.
  Without the AMD support bits at this point.
* No "cpus", "mode", "size" files yet.
* Just CTRL_MON directories.
* No monitoring support yet.

-Tony

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Peter Newman]

On Mon, Sep 18, 2023 at 12:44 PM Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
> Drew Fustini <dfustini@baylibre.com> wrote:
> >
> > This reminds me that Linux Plumbers Conference [1] is in November and
> > I think resctrl2 could be a good topic. The CFP is still open for Birds
> > of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> > people wishing to discuss a particular topic, and I have had success
> > hosting them in the past for topics like pinctrl and gpio.
> >
> > Anyone planning to attend Plumbers?
> >
> > I'll be going in person but the virtual option works really well in my
> > experience. I had developers and maintainers attending virtually
> > participate in my BoF sessions and I felt it was very productive.
>
> FWIW I'm keen and should be there in person.  However, I'm not on the must
> be available list for this one ;)   Agree that hybrid worked fine for BoF last
> year.
>
> Jonathan

I got us a BoF session to talk about resctrl and porting it to new
architectures:

https://lpc.events/event/17/contributions/1567/

To anyone planning to attend, while the schedule is still easy to
adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
doesn't work.

Also I got one registration code for scheduling this. Let me know if
you want it.

Thanks!
-Peter

Re: resctrl2 - status

[Peter Newman]

On Mon, Sep 18, 2023 at 12:44 PM Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
> Drew Fustini <dfustini@baylibre.com> wrote:
> >
> > This reminds me that Linux Plumbers Conference [1] is in November and
> > I think resctrl2 could be a good topic. The CFP is still open for Birds
> > of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> > people wishing to discuss a particular topic, and I have had success
> > hosting them in the past for topics like pinctrl and gpio.
> >
> > Anyone planning to attend Plumbers?
> >
> > I'll be going in person but the virtual option works really well in my
> > experience. I had developers and maintainers attending virtually
> > participate in my BoF sessions and I felt it was very productive.
>
> FWIW I'm keen and should be there in person.  However, I'm not on the must
> be available list for this one ;)   Agree that hybrid worked fine for BoF last
> year.
>
> Jonathan

I got us a BoF session to talk about resctrl and porting it to new
architectures:

https://lpc.events/event/17/contributions/1567/

To anyone planning to attend, while the schedule is still easy to
adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
doesn't work.

Also I got one registration code for scheduling this. Let me know if
you want it.

Thanks!
-Peter

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: resctrl2 - status

[Luck, Tony]

> I got us a BoF session to talk about resctrl and porting it to new
> architectures:

Thanks for doing this.

> https://lpc.events/event/17/contributions/1567/
>
> To anyone planning to attend, while the schedule is still easy to
> adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> doesn't work.

Do you know if the BoF sessions are going to be remote accessible?
Personal conflicts mean I can't attend in person.

-Tony

RE: resctrl2 - status

[Luck, Tony]

> I got us a BoF session to talk about resctrl and porting it to new
> architectures:

Thanks for doing this.

> https://lpc.events/event/17/contributions/1567/
>
> To anyone planning to attend, while the schedule is still easy to
> adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> doesn't work.

Do you know if the BoF sessions are going to be remote accessible?
Personal conflicts mean I can't attend in person.

-Tony
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Jonathan Cameron]

On Thu, 28 Sep 2023 14:47:13 +0000
"Luck, Tony" <tony.luck@intel.com> wrote:

> > I got us a BoF session to talk about resctrl and porting it to new
> > architectures:  
> 
> Thanks for doing this.
Great!

> 
> > https://lpc.events/event/17/contributions/1567/
> >
> > To anyone planning to attend, while the schedule is still easy to
> > adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> > doesn't work.  
> 
> Do you know if the BoF sessions are going to be remote accessible?
> Personal conflicts mean I can't attend in person.

They were last year (it worked surprisingly well). If there are any problems
we can set up a jabra in the room.

I'm keen to attend this. That time currently works for me though I got
the impression things are still very fluid on schedule (I sent a list
of uconfs to avoid for the PCI device attestation BoF that is just before
this in the posted schedule.)

So if there are obvious clashes to avoid, (I can't immediately spot any
general ones) then pass those on.

Jonathan

> 
> -Tony

Re: resctrl2 - status

[Jonathan Cameron]

On Thu, 28 Sep 2023 14:47:13 +0000
"Luck, Tony" <tony.luck@intel.com> wrote:

> > I got us a BoF session to talk about resctrl and porting it to new
> > architectures:  
> 
> Thanks for doing this.
Great!

> 
> > https://lpc.events/event/17/contributions/1567/
> >
> > To anyone planning to attend, while the schedule is still easy to
> > adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> > doesn't work.  
> 
> Do you know if the BoF sessions are going to be remote accessible?
> Personal conflicts mean I can't attend in person.

They were last year (it worked surprisingly well). If there are any problems
we can set up a jabra in the room.

I'm keen to attend this. That time currently works for me though I got
the impression things are still very fluid on schedule (I sent a list
of uconfs to avoid for the PCI device attestation BoF that is just before
this in the posted schedule.)

So if there are obvious clashes to avoid, (I can't immediately spot any
general ones) then pass those on.

Jonathan

> 
> -Tony


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: resctrl2 - status

[Drew Fustini]

On Thu, Sep 28, 2023 at 10:47:22AM +0200, Peter Newman wrote:
> On Mon, Sep 18, 2023 at 12:44 PM Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> > Drew Fustini <dfustini@baylibre.com> wrote:
> > >
> > > This reminds me that Linux Plumbers Conference [1] is in November and
> > > I think resctrl2 could be a good topic. The CFP is still open for Birds
> > > of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> > > people wishing to discuss a particular topic, and I have had success
> > > hosting them in the past for topics like pinctrl and gpio.
> > >
> > > Anyone planning to attend Plumbers?
> > >
> > > I'll be going in person but the virtual option works really well in my
> > > experience. I had developers and maintainers attending virtually
> > > participate in my BoF sessions and I felt it was very productive.
> >
> > FWIW I'm keen and should be there in person.  However, I'm not on the must
> > be available list for this one ;)   Agree that hybrid worked fine for BoF last
> > year.
> >
> > Jonathan
> 
> I got us a BoF session to talk about resctrl and porting it to new
> architectures:
> 
> https://lpc.events/event/17/contributions/1567/
> 
> To anyone planning to attend, while the schedule is still easy to
> adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> doesn't work.
> 
> Also I got one registration code for scheduling this. Let me know if
> you want it.

Great, thanks for setting up a session for resctrl. The only thing I
need to attend is the RISC-V MC. Hopefully it will not be during that
time. I've cc'd Atish and Palmer in case they know. They may also be
interested in resctrl, too.

thanks,
drew

Re: resctrl2 - status

[Drew Fustini]

On Thu, Sep 28, 2023 at 10:47:22AM +0200, Peter Newman wrote:
> On Mon, Sep 18, 2023 at 12:44 PM Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> > Drew Fustini <dfustini@baylibre.com> wrote:
> > >
> > > This reminds me that Linux Plumbers Conference [1] is in November and
> > > I think resctrl2 could be a good topic. The CFP is still open for Birds
> > > of a Feather (BoF) proposals [2]. These are free-form get-togethers for
> > > people wishing to discuss a particular topic, and I have had success
> > > hosting them in the past for topics like pinctrl and gpio.
> > >
> > > Anyone planning to attend Plumbers?
> > >
> > > I'll be going in person but the virtual option works really well in my
> > > experience. I had developers and maintainers attending virtually
> > > participate in my BoF sessions and I felt it was very productive.
> >
> > FWIW I'm keen and should be there in person.  However, I'm not on the must
> > be available list for this one ;)   Agree that hybrid worked fine for BoF last
> > year.
> >
> > Jonathan
> 
> I got us a BoF session to talk about resctrl and porting it to new
> architectures:
> 
> https://lpc.events/event/17/contributions/1567/
> 
> To anyone planning to attend, while the schedule is still easy to
> adjust, please let me know if the proposed time (Nov 15, 4:30-5:15pm)
> doesn't work.
> 
> Also I got one registration code for scheduling this. Let me know if
> you want it.

Great, thanks for setting up a session for resctrl. The only thing I
need to attend is the RISC-V MC. Hopefully it will not be during that
time. I've cc'd Atish and Palmer in case they know. They may also be
interested in resctrl, too.

thanks,
drew

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel