Re: [PATCHv12 0/3] rdmacg: IB/core: rdma controller support

[Parav Pandit <pandit.parav-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

Hi Leon,

On Wed, Oct 5, 2016 at 4:52 PM, Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Wed, Aug 31, 2016 at 02:07:24PM +0530, Parav Pandit wrote:
>> rdmacg: IB/core: rdma controller support
>>
>> Overview:
>> Currently user space applications can easily take away all the rdma
>> device specific resources such as AH, CQ, QP, MR etc. Due to which other
>> applications in other cgroup or kernel space ULPs may not even get chance
>> to allocate any rdma resources. This results into service unavailibility.
>>
>> RDMA cgroup addresses this issue by allowing resource accounting,
>> limit enforcement on per cgroup, per rdma device basis.
>>
>> RDMA uverbs layer will enforce limits on well defined RDMA verb
>> resources without any HCA vendor device driver involvement.
>>
>> RDMA uverbs layer will not do limit enforcement of HCA hw vendor
>> specific resources. Instead rdma cgroup provides set of APIs
>> through which vendor specific drivers can do resource accounting
>> by making use of rdma cgroup.
>
> Hi Parav,
> I want to propose an extension to the RDMA cgroup which can be done as
> follow-up patches.
>
> Let's add new global type, which will control whole HCA (for example in percentages). It will
> allow natively define new objects without need to introduce them to the user.
>
In other cgroup such as CPU, this is done using cpu.weight API. Where
percentage or weight is configured by the user.
In this mode, resources taken away from other cgroup proportionately.
It works for cpu because its mainly stateless resource unlike rdma
resources.
So if we want to simplify user configuration similarly,
percentage/weight configuration can be extended.
This way they need not be introduced to users.
I hope your definition of "user" is actual end-user and not rdma cgroup. Right?
In other words, new object should be still added as new enum value in
rdma_cgroup.h?
Only than it can be overwritten by specific UVERBs type as you
described below. I think thats what you meant as you described below.

Otherwise charging/uncharging this new percentage resource can get messy.

> This HCA share will be overwritten by specific UVERBS types which you
> already defined.
>
> What do you think?

So to refine your proposal from cgroup perspective, instead of adding
new resource type in rdma_cgroup.h for percentage, I prefer to have

Existing
1. rdma.max
2. rdma.current
New,
3. rdma.weight
This ABI will have similar API to say
echo "mlx4_0 50" > rdma.weight.
Where 50 is weight of the resources.
For example,
for one cgroup instance weight=sum=100% resource for a given cgroup.
for three cgroup instances percentage=(weight/sum)% = 50/(50+50+50) = 33%.
One cgroup gets 33% resource.

Weight can be in range of 1 to 10,000 similar to cpu cgroup.

This might work if applications running in all cgroups are similar.
But weight doesn't do justice, when there are different type of
applications running in each cgroup. Such as few running libfabric
based apps, few running MPI, others directly using ibverbs.
So as you said rdma.max configuration would be required for management
plane to override weight (percentage) for certain resources.


>
> Except this proposal,
> Reviewed-by: Leon Romanovsky <leonro-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>
> Thanks.