[Question] Do we need remote charging for cpu and cpuacct subsys?

[Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

Hi,

memcg currently has a remote charging mechanism that can charge usage to other
memcg instead of the one the task belongs to.

In our environment, we need to account the cpu usage consumed by some kworkers
to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
cpu and cpuacct subsys in our kernel.

I want to know if the community has a plan to do this?
What will the community approach look like?

I think we need to move the active_memcg to a separated active_cgroup struct,
and the latter will contain active_memcg, active_tg, and active_cpuacct.


Regards,
Hao Lee

[Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

Hi,

memcg currently has a remote charging mechanism that can charge usage to other
memcg instead of the one the task belongs to.

In our environment, we need to account the cpu usage consumed by some kworkers
to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
cpu and cpuacct subsys in our kernel.

I want to know if the community has a plan to do this?
What will the community approach look like?

I think we need to move the active_memcg to a separated active_cgroup struct,
and the latter will contain active_memcg, active_tg, and active_cpuacct.


Regards,
Hao Lee

[Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

Hi,

memcg currently has a remote charging mechanism that can charge usage to other
memcg instead of the one the task belongs to.

In our environment, we need to account the cpu usage consumed by some kworkers
to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
cpu and cpuacct subsys in our kernel.

I want to know if the community has a plan to do this?
What will the community approach look like?

I think we need to move the active_memcg to a separated active_cgroup struct,
and the latter will contain active_memcg, active_tg, and active_cpuacct.


Regards,
Hao Lee

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Daniel Jordan]

Hello,

On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> memcg currently has a remote charging mechanism that can charge usage to other
> memcg instead of the one the task belongs to.
> 
> In our environment, we need to account the cpu usage consumed by some kworkers
> to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> cpu and cpuacct subsys in our kernel.

I also want to see this upstream, and am actually working on it right
now, have been for some time.

So far, this is needed to properly account multithreaded padata jobs,
memory reclaim, and net rx.  Android folks have raised this issue in the
past too, though I'm not aware of the specific kthreads that are giving
them problems.

So naturally, I'm curious about your use case and how it may be
different from these others.  What kworkers would you like to account?

> I want to know if the community has a plan to do this?
> What will the community approach look like?

There has been discussion about this here,

   https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3@ca-dmjordan1.us.oracle.com/

more recently here,

   https://lore.kernel.org/lkml/YGxjwKbec68sCcqo@slm.duckdns.org/

and we may talk about it at LPC:

   https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform

> I think we need to move the active_memcg to a separated active_cgroup struct,
> and the latter will contain active_memcg, active_tg, and active_cpuacct.

I'm not seeing how that could work for cases that don't know the cgroup
when the remote charging period begins.  The only one I'm aware of
that's like that is net rx, where the work to process packets has to
start before their ultimate destination, and therefore cgroup, is known.

thanks,
Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Daniel Jordan]

+ Android folks

On Fri, Jul 02, 2021 at 04:07:42PM -0400, Daniel Jordan wrote:
> Hello,
> 
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> > 
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
> 
> I also want to see this upstream, and am actually working on it right
> now, have been for some time.
> 
> So far, this is needed to properly account multithreaded padata jobs,
> memory reclaim, and net rx.  Android folks have raised this issue in the
> past too, though I'm not aware of the specific kthreads that are giving
> them problems.

Pavan, Wei, do you have any details about this?

> So naturally, I'm curious about your use case and how it may be
> different from these others.  What kworkers would you like to account?
> 
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
> 
> There has been discussion about this here,
> 
>    https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3@ca-dmjordan1.us.oracle.com/
> 
> more recently here,
> 
>    https://lore.kernel.org/lkml/YGxjwKbec68sCcqo@slm.duckdns.org/
> 
> and we may talk about it at LPC:
> 
>    https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform
> 
> > I think we need to move the active_memcg to a separated active_cgroup struct,
> > and the latter will contain active_memcg, active_tg, and active_cpuacct.
> 
> I'm not seeing how that could work for cases that don't know the cgroup
> when the remote charging period begins.  The only one I'm aware of
> that's like that is net rx, where the work to process packets has to
> start before their ultimate destination, and therefore cgroup, is known.
> 
> thanks,
> Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Daniel Jordan]

+ Android folks

On Fri, Jul 02, 2021 at 04:07:42PM -0400, Daniel Jordan wrote:
> Hello,
> 
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> > 
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
> 
> I also want to see this upstream, and am actually working on it right
> now, have been for some time.
> 
> So far, this is needed to properly account multithreaded padata jobs,
> memory reclaim, and net rx.  Android folks have raised this issue in the
> past too, though I'm not aware of the specific kthreads that are giving
> them problems.

Pavan, Wei, do you have any details about this?

> So naturally, I'm curious about your use case and how it may be
> different from these others.  What kworkers would you like to account?
> 
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
> 
> There has been discussion about this here,
> 
>    https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
> 
> more recently here,
> 
>    https://lore.kernel.org/lkml/YGxjwKbec68sCcqo-NiLfg/pYEd1N0TnZuCh8vA@public.gmane.org/
> 
> and we may talk about it at LPC:
> 
>    https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform
> 
> > I think we need to move the active_memcg to a separated active_cgroup struct,
> > and the latter will contain active_memcg, active_tg, and active_cpuacct.
> 
> I'm not seeing how that could work for cases that don't know the cgroup
> when the remote charging period begins.  The only one I'm aware of
> that's like that is net rx, where the work to process packets has to
> start before their ultimate destination, and therefore cgroup, is known.
> 
> thanks,
> Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

On Sat, Jul 3, 2021 at 4:07 AM Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
>
> Hello,
>
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> >
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
>
> I also want to see this upstream, and am actually working on it right
> now, have been for some time.
>
> So far, this is needed to properly account multithreaded padata jobs,
> memory reclaim, and net rx. Android folks have raised this issue in the
> past too, though I'm not aware of the specific kthreads that are giving
> them problems.
>
> So naturally, I'm curious about your use case and how it may be
> different from these others. What kworkers would you like to account?

Thanks. We use a workqueue to perform asynchronous reclaim for cgroups.
The kworker may consume lots of CPU cycles if the cgroup memory pressure
is extremely high, so we want to charge the cpu usage to the related
cgroup for which the kworker works. Otherwise, the reclaim kworker will
steal cpu time from the system level, which breaks the resource isolation.

>
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
>
> There has been discussion about this here,
>
>    https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3@ca-dmjordan1.us.oracle.com/
>

Yes, our work is very similar to Johannes'.

> more recently here,
>
>    https://lore.kernel.org/lkml/YGxjwKbec68sCcqo@slm.duckdns.org/
>
> and we may talk about it at LPC:
>
>    https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform

I also found that you posted a patchset in 2019 to introduce a
cgroup-aware workqueue. In that discussion, back-charging is considered
to be a suitable solution.
    https://lore.kernel.org/lkml/20190611195549.GL3341036@devbig004.ftw2.facebook.com/

I also have a question here. Are the back-charging and remote charging
the same thing?

>
> > I think we need to move the active_memcg to a separated active_cgroup struct,
> > and the latter will contain active_memcg, active_tg, and active_cpuacct.
>
> I'm not seeing how that could work for cases that don't know the cgroup
> when the remote charging period begins.

This is indeed a problem. Neither cgroup-aware kthread nor remote
charging can address this. Maybe this is the biggest obstacle hindering
fine-grained charging.

> The only one I'm aware of
> that's like that is net rx, where the work to process packets has to
> start before their ultimate destination, and therefore cgroup, is known.

Sorry. Is this a typo? It seems the word "known" should be "unknown"...


Regards,
Hao Lee

>
> thanks,
> Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

On Sat, Jul 3, 2021 at 4:07 AM Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>
> Hello,
>
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> >
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
>
> I also want to see this upstream, and am actually working on it right
> now, have been for some time.
>
> So far, this is needed to properly account multithreaded padata jobs,
> memory reclaim, and net rx. Android folks have raised this issue in the
> past too, though I'm not aware of the specific kthreads that are giving
> them problems.
>
> So naturally, I'm curious about your use case and how it may be
> different from these others. What kworkers would you like to account?

Thanks. We use a workqueue to perform asynchronous reclaim for cgroups.
The kworker may consume lots of CPU cycles if the cgroup memory pressure
is extremely high, so we want to charge the cpu usage to the related
cgroup for which the kworker works. Otherwise, the reclaim kworker will
steal cpu time from the system level, which breaks the resource isolation.

>
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
>
> There has been discussion about this here,
>
>    https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/
>

Yes, our work is very similar to Johannes'.

> more recently here,
>
>    https://lore.kernel.org/lkml/YGxjwKbec68sCcqo-NiLfg/pYEd1N0TnZuCh8vA@public.gmane.org/
>
> and we may talk about it at LPC:
>
>    https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform

I also found that you posted a patchset in 2019 to introduce a
cgroup-aware workqueue. In that discussion, back-charging is considered
to be a suitable solution.
    https://lore.kernel.org/lkml/20190611195549.GL3341036-LpCCV3molIbIZ9tKgghJQw2O0Ztt9esIQQ4Iyu8u01E@public.gmane.org/

I also have a question here. Are the back-charging and remote charging
the same thing?

>
> > I think we need to move the active_memcg to a separated active_cgroup struct,
> > and the latter will contain active_memcg, active_tg, and active_cpuacct.
>
> I'm not seeing how that could work for cases that don't know the cgroup
> when the remote charging period begins.

This is indeed a problem. Neither cgroup-aware kthread nor remote
charging can address this. Maybe this is the biggest obstacle hindering
fine-grained charging.

> The only one I'm aware of
> that's like that is net rx, where the work to process packets has to
> start before their ultimate destination, and therefore cgroup, is known.

Sorry. Is this a typo? It seems the word "known" should be "unknown"...


Regards,
Hao Lee

>
> thanks,
> Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

On Sat, Jul 3, 2021 at 4:07 AM Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
>
> Hello,
>
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> >
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
>
> I also want to see this upstream, and am actually working on it right
> now, have been for some time.
>
> So far, this is needed to properly account multithreaded padata jobs,
> memory reclaim, and net rx. Android folks have raised this issue in the
> past too, though I'm not aware of the specific kthreads that are giving
> them problems.
>
> So naturally, I'm curious about your use case and how it may be
> different from these others. What kworkers would you like to account?

Thanks. We use a workqueue to perform asynchronous reclaim for cgroups.
The kworker may consume lots of CPU cycles if the cgroup memory pressure
is extremely high, so we want to charge the cpu usage to the related
cgroup for which the kworker works. Otherwise, the reclaim kworker will
steal cpu time from the system level, which breaks the resource isolation.

>
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
>
> There has been discussion about this here,
>
>    https://lore.kernel.org/lkml/20200219214112.4kt573kyzbvmbvn3@ca-dmjordan1.us.oracle.com/
>

Yes, our work is very similar to Johannes'.

> more recently here,
>
>    https://lore.kernel.org/lkml/YGxjwKbec68sCcqo@slm.duckdns.org/
>
> and we may talk about it at LPC:
>
>    https://www.linuxplumbersconf.org/event/11/page/104-accepted-microconferences#cont-perform

I also found that you posted a patchset in 2019 to introduce a
cgroup-aware workqueue. In that discussion, back-charging is considered
to be a suitable solution.
    https://lore.kernel.org/lkml/20190611195549.GL3341036@devbig004.ftw2.facebook.com/

I also have a question here. Are the back-charging and remote charging
the same thing?

>
> > I think we need to move the active_memcg to a separated active_cgroup struct,
> > and the latter will contain active_memcg, active_tg, and active_cpuacct.
>
> I'm not seeing how that could work for cases that don't know the cgroup
> when the remote charging period begins.

This is indeed a problem. Neither cgroup-aware kthread nor remote
charging can address this. Maybe this is the biggest obstacle hindering
fine-grained charging.

> The only one I'm aware of
> that's like that is net rx, where the work to process packets has to
> start before their ultimate destination, and therefore cgroup, is known.

Sorry. Is this a typo? It seems the word "known" should be "unknown"...


Regards,
Hao Lee

>
> thanks,
> Daniel

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Daniel Jordan]

Hi,

[only replying now b/c I was out most of this week]

On Sun, Jul 04, 2021 at 12:18:22AM +0800, Hao Lee wrote:
> On Sat, Jul 3, 2021 at 4:07 AM Daniel Jordan <daniel.m.jordan@oracle.com> wrote:
> > So naturally, I'm curious about your use case and how it may be
> > different from these others. What kworkers would you like to account?
> 
> Thanks. We use a workqueue to perform asynchronous reclaim for cgroups.
> The kworker may consume lots of CPU cycles if the cgroup memory pressure
> is extremely high, so we want to charge the cpu usage to the related
> cgroup for which the kworker works. Otherwise, the reclaim kworker will
> steal cpu time from the system level, which breaks the resource isolation.

Ok, that helps.

> I also have a question here. Are the back-charging and remote charging
> the same thing?

Basically yes.

> > The only one I'm aware of
> > that's like that is net rx, where the work to process packets has to
> > start before their ultimate destination, and therefore cgroup, is known.
> 
> Sorry. Is this a typo?  It seems the word "known" should be "unknown"...

What I'm saying is, the work to process packets starts when the cgroup
is unknown.

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Daniel Jordan]

Hi,

[only replying now b/c I was out most of this week]

On Sun, Jul 04, 2021 at 12:18:22AM +0800, Hao Lee wrote:
> On Sat, Jul 3, 2021 at 4:07 AM Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> > So naturally, I'm curious about your use case and how it may be
> > different from these others. What kworkers would you like to account?
> 
> Thanks. We use a workqueue to perform asynchronous reclaim for cgroups.
> The kworker may consume lots of CPU cycles if the cgroup memory pressure
> is extremely high, so we want to charge the cpu usage to the related
> cgroup for which the kworker works. Otherwise, the reclaim kworker will
> steal cpu time from the system level, which breaks the resource isolation.

Ok, that helps.

> I also have a question here. Are the back-charging and remote charging
> the same thing?

Basically yes.

> > The only one I'm aware of
> > that's like that is net rx, where the work to process packets has to
> > start before their ultimate destination, and therefore cgroup, is known.
> 
> Sorry. Is this a typo?  It seems the word "known" should be "unknown"...

What I'm saying is, the work to process packets starts when the cgroup
is unknown.

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Tejun Heo]

Hello,

On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> memcg currently has a remote charging mechanism that can charge usage to other
> memcg instead of the one the task belongs to.
> 
> In our environment, we need to account the cpu usage consumed by some kworkers
> to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> cpu and cpuacct subsys in our kernel.
> 
> I want to know if the community has a plan to do this?
> What will the community approach look like?

Yeah, we need sth like this to account for cpu cycles spent in commont path
but caused by specific cgroups - e.g. memory reclaim, net packet rx and so
on. There were some mentions of needing somthing like that but haven't been
any patchsets or concrete efforts that I know of.

Thanks.

-- 
tejun

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Tejun Heo]

Hello,

On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> memcg currently has a remote charging mechanism that can charge usage to other
> memcg instead of the one the task belongs to.
> 
> In our environment, we need to account the cpu usage consumed by some kworkers
> to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> cpu and cpuacct subsys in our kernel.
> 
> I want to know if the community has a plan to do this?
> What will the community approach look like?

Yeah, we need sth like this to account for cpu cycles spent in commont path
but caused by specific cgroups - e.g. memory reclaim, net packet rx and so
on. There were some mentions of needing somthing like that but haven't been
any patchsets or concrete efforts that I know of.

Thanks.

-- 
tejun

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

On Sat, Jul 3, 2021 at 6:26 AM Tejun Heo <tj@kernel.org> wrote:
>
> Hello,
>
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> >
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
> >
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
>
> Yeah, we need sth like this to account for cpu cycles spent in commont path
> but caused by specific cgroups - e.g. memory reclaim, net packet rx and so
> on. There were some mentions of needing somthing like that but haven't been
> any patchsets or concrete efforts that I know of.

Thanks for this reply. I am interested in this area because this feature
would be very useful in our containerized environment. I will track related
discussions and want to do some work related to remote charging.

>
> Thanks.
>
> --
> tejun

Re: [Question] Do we need remote charging for cpu and cpuacct subsys?

[Hao Lee]

On Sat, Jul 3, 2021 at 6:26 AM Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>
> Hello,
>
> On Fri, Jul 02, 2021 at 08:26:27AM -0000, Hao Lee wrote:
> > memcg currently has a remote charging mechanism that can charge usage to other
> > memcg instead of the one the task belongs to.
> >
> > In our environment, we need to account the cpu usage consumed by some kworkers
> > to a specific cgroup. Thus, we want to introduce a remote-charging mechanism to
> > cpu and cpuacct subsys in our kernel.
> >
> > I want to know if the community has a plan to do this?
> > What will the community approach look like?
>
> Yeah, we need sth like this to account for cpu cycles spent in commont path
> but caused by specific cgroups - e.g. memory reclaim, net packet rx and so
> on. There were some mentions of needing somthing like that but haven't been
> any patchsets or concrete efforts that I know of.

Thanks for this reply. I am interested in this area because this feature
would be very useful in our containerized environment. I will track related
discussions and want to do some work related to remote charging.

>
> Thanks.
>
> --
> tejun