linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] mm: memcg/slab: optimize objcg stock draining
@ 2021-01-06  4:22 Roman Gushchin
  2021-01-06  6:05 ` Shakeel Butt
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: Roman Gushchin @ 2021-01-06  4:22 UTC (permalink / raw)
  To: Andrew Morton, linux-mm
  Cc: Michal Hocko, Johannes Weiner, Shakeel Butt, linux-kernel,
	kernel-team, Roman Gushchin, Imran Khan

Imran Khan reported a regression in hackbench results caused by the
commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
instead of pages"). The regression is noticeable in the case of
a consequent allocation of several relatively large slab objects,
e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
to a number of atomic operations in page_counter_uncharge().

The corresponding call graph is below (provided by Imran Khan):
  |__alloc_skb
  |    |
  |    |__kmalloc_reserve.isra.61
  |    |    |
  |    |    |__kmalloc_node_track_caller
  |    |    |    |
  |    |    |    |slab_pre_alloc_hook.constprop.88
  |    |    |     obj_cgroup_charge
  |    |    |    |    |
  |    |    |    |    |__memcg_kmem_charge
  |    |    |    |    |    |
  |    |    |    |    |    |page_counter_try_charge
  |    |    |    |    |
  |    |    |    |    |refill_obj_stock
  |    |    |    |    |    |
  |    |    |    |    |    |drain_obj_stock.isra.68
  |    |    |    |    |    |    |
  |    |    |    |    |    |    |__memcg_kmem_uncharge
  |    |    |    |    |    |    |    |
  |    |    |    |    |    |    |    |page_counter_uncharge
  |    |    |    |    |    |    |    |    |
  |    |    |    |    |    |    |    |    |page_counter_cancel
  |    |    |    |
  |    |    |    |
  |    |    |    |__slab_alloc
  |    |    |    |    |
  |    |    |    |    |___slab_alloc
  |    |    |    |    |
  |    |    |    |slab_post_alloc_hook

Instead of directly uncharging the accounted kernel memory, it's
possible to refill the generic page-sized per-cpu stock instead.
It's a much faster operation, especially on a default hierarchy.
As a bonus, __memcg_kmem_uncharge_page() will also get faster,
so the freeing of page-sized kernel allocations (e.g. large kmallocs)
will become faster.

A similar change has been done earlier for the socket memory by
the commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for
socket memory uncharging").

Signed-off-by: Roman Gushchin <guro@fb.com>
Reported-by: Imran Khan <imran.f.khan@oracle.com>
---
 mm/memcontrol.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 0d74b80fa4de..8148c1df3aff 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
 	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		page_counter_uncharge(&memcg->kmem, nr_pages);
 
-	page_counter_uncharge(&memcg->memory, nr_pages);
-	if (do_memsw_account())
-		page_counter_uncharge(&memcg->memsw, nr_pages);
+	refill_stock(memcg, nr_pages);
 }
 
 /**
-- 
2.26.2


^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06  4:22 [PATCH] mm: memcg/slab: optimize objcg stock draining Roman Gushchin
@ 2021-01-06  6:05 ` Shakeel Butt
  2021-01-06 16:42   ` Roman Gushchin
  2021-01-06  6:19 ` Imran Khan
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 7+ messages in thread
From: Shakeel Butt @ 2021-01-06  6:05 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, Linux MM, Michal Hocko, Johannes Weiner, LKML,
	Kernel Team, Imran Khan

On Tue, Jan 5, 2021 at 8:22 PM Roman Gushchin <guro@fb.com> wrote:
>
> Imran Khan reported a regression in hackbench results caused by the
> commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> instead of pages"). The regression is noticeable in the case of
> a consequent allocation of several relatively large slab objects,
> e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
> drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
> to a number of atomic operations in page_counter_uncharge().
>
> The corresponding call graph is below (provided by Imran Khan):
>   |__alloc_skb
>   |    |
>   |    |__kmalloc_reserve.isra.61
>   |    |    |
>   |    |    |__kmalloc_node_track_caller
>   |    |    |    |
>   |    |    |    |slab_pre_alloc_hook.constprop.88
>   |    |    |     obj_cgroup_charge
>   |    |    |    |    |
>   |    |    |    |    |__memcg_kmem_charge
>   |    |    |    |    |    |
>   |    |    |    |    |    |page_counter_try_charge
>   |    |    |    |    |
>   |    |    |    |    |refill_obj_stock
>   |    |    |    |    |    |
>   |    |    |    |    |    |drain_obj_stock.isra.68
>   |    |    |    |    |    |    |
>   |    |    |    |    |    |    |__memcg_kmem_uncharge
>   |    |    |    |    |    |    |    |
>   |    |    |    |    |    |    |    |page_counter_uncharge
>   |    |    |    |    |    |    |    |    |
>   |    |    |    |    |    |    |    |    |page_counter_cancel
>   |    |    |    |
>   |    |    |    |
>   |    |    |    |__slab_alloc
>   |    |    |    |    |
>   |    |    |    |    |___slab_alloc
>   |    |    |    |    |
>   |    |    |    |slab_post_alloc_hook
>
> Instead of directly uncharging the accounted kernel memory, it's
> possible to refill the generic page-sized per-cpu stock instead.
> It's a much faster operation, especially on a default hierarchy.
> As a bonus, __memcg_kmem_uncharge_page() will also get faster,
> so the freeing of page-sized kernel allocations (e.g. large kmallocs)
> will become faster.
>
> A similar change has been done earlier for the socket memory by
> the commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for
> socket memory uncharging").
>
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Reported-by: Imran Khan <imran.f.khan@oracle.com>

I remember seeing this somewhere
https://lore.kernel.org/linux-mm/20190423154405.259178-1-shakeelb@google.com/

Reviewed-by: Shakeel Butt <shakeelb@google.com>

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06  4:22 [PATCH] mm: memcg/slab: optimize objcg stock draining Roman Gushchin
  2021-01-06  6:05 ` Shakeel Butt
@ 2021-01-06  6:19 ` Imran Khan
  2021-01-06 19:50 ` Andrew Morton
  2021-01-20 16:34 ` Michal Koutný
  3 siblings, 0 replies; 7+ messages in thread
From: Imran Khan @ 2021-01-06  6:19 UTC (permalink / raw)
  To: Roman Gushchin, Andrew Morton, linux-mm
  Cc: Michal Hocko, Johannes Weiner, Shakeel Butt, linux-kernel, kernel-team



On 6/1/21 3:22 pm, Roman Gushchin wrote:
> Imran Khan reported a regression in hackbench results caused by the
> commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> instead of pages"). The regression is noticeable in the case of
> a consequent allocation of several relatively large slab objects,
> e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
> drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
> to a number of atomic operations in page_counter_uncharge().
> 
> The corresponding call graph is below (provided by Imran Khan):
>    |__alloc_skb
>    |    |
>    |    |__kmalloc_reserve.isra.61
>    |    |    |
>    |    |    |__kmalloc_node_track_caller
>    |    |    |    |
>    |    |    |    |slab_pre_alloc_hook.constprop.88
>    |    |    |     obj_cgroup_charge
>    |    |    |    |    |
>    |    |    |    |    |__memcg_kmem_charge
>    |    |    |    |    |    |
>    |    |    |    |    |    |page_counter_try_charge
>    |    |    |    |    |
>    |    |    |    |    |refill_obj_stock
>    |    |    |    |    |    |
>    |    |    |    |    |    |drain_obj_stock.isra.68
>    |    |    |    |    |    |    |
>    |    |    |    |    |    |    |__memcg_kmem_uncharge
>    |    |    |    |    |    |    |    |
>    |    |    |    |    |    |    |    |page_counter_uncharge
>    |    |    |    |    |    |    |    |    |
>    |    |    |    |    |    |    |    |    |page_counter_cancel
>    |    |    |    |
>    |    |    |    |
>    |    |    |    |__slab_alloc
>    |    |    |    |    |
>    |    |    |    |    |___slab_alloc
>    |    |    |    |    |
>    |    |    |    |slab_post_alloc_hook
> 
> Instead of directly uncharging the accounted kernel memory, it's
> possible to refill the generic page-sized per-cpu stock instead.
> It's a much faster operation, especially on a default hierarchy.
> As a bonus, __memcg_kmem_uncharge_page() will also get faster,
> so the freeing of page-sized kernel allocations (e.g. large kmallocs)
> will become faster.
> 
> A similar change has been done earlier for the socket memory by
> the commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for
> socket memory uncharging").
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
> Reported-by: Imran Khan <imran.f.khan@oracle.com>

Tested-by: Imran Khan <imran.f.khan@oracle.com>

> ---
>   mm/memcontrol.c | 4 +---
>   1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 0d74b80fa4de..8148c1df3aff 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
>   	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
>   		page_counter_uncharge(&memcg->kmem, nr_pages);
>   
> -	page_counter_uncharge(&memcg->memory, nr_pages);
> -	if (do_memsw_account())
> -		page_counter_uncharge(&memcg->memsw, nr_pages);
> +	refill_stock(memcg, nr_pages);
>   }
>   
>   /**
> 

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06  6:05 ` Shakeel Butt
@ 2021-01-06 16:42   ` Roman Gushchin
  0 siblings, 0 replies; 7+ messages in thread
From: Roman Gushchin @ 2021-01-06 16:42 UTC (permalink / raw)
  To: Shakeel Butt
  Cc: Andrew Morton, Linux MM, Michal Hocko, Johannes Weiner, LKML,
	Kernel Team, Imran Khan

On Tue, Jan 05, 2021 at 10:05:20PM -0800, Shakeel Butt wrote:
> On Tue, Jan 5, 2021 at 8:22 PM Roman Gushchin <guro@fb.com> wrote:
> >
> > Imran Khan reported a regression in hackbench results caused by the
> > commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> > instead of pages"). The regression is noticeable in the case of
> > a consequent allocation of several relatively large slab objects,
> > e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
> > drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
> > to a number of atomic operations in page_counter_uncharge().
> >
> > The corresponding call graph is below (provided by Imran Khan):
> >   |__alloc_skb
> >   |    |
> >   |    |__kmalloc_reserve.isra.61
> >   |    |    |
> >   |    |    |__kmalloc_node_track_caller
> >   |    |    |    |
> >   |    |    |    |slab_pre_alloc_hook.constprop.88
> >   |    |    |     obj_cgroup_charge
> >   |    |    |    |    |
> >   |    |    |    |    |__memcg_kmem_charge
> >   |    |    |    |    |    |
> >   |    |    |    |    |    |page_counter_try_charge
> >   |    |    |    |    |
> >   |    |    |    |    |refill_obj_stock
> >   |    |    |    |    |    |
> >   |    |    |    |    |    |drain_obj_stock.isra.68
> >   |    |    |    |    |    |    |
> >   |    |    |    |    |    |    |__memcg_kmem_uncharge
> >   |    |    |    |    |    |    |    |
> >   |    |    |    |    |    |    |    |page_counter_uncharge
> >   |    |    |    |    |    |    |    |    |
> >   |    |    |    |    |    |    |    |    |page_counter_cancel
> >   |    |    |    |
> >   |    |    |    |
> >   |    |    |    |__slab_alloc
> >   |    |    |    |    |
> >   |    |    |    |    |___slab_alloc
> >   |    |    |    |    |
> >   |    |    |    |slab_post_alloc_hook
> >
> > Instead of directly uncharging the accounted kernel memory, it's
> > possible to refill the generic page-sized per-cpu stock instead.
> > It's a much faster operation, especially on a default hierarchy.
> > As a bonus, __memcg_kmem_uncharge_page() will also get faster,
> > so the freeing of page-sized kernel allocations (e.g. large kmallocs)
> > will become faster.
> >
> > A similar change has been done earlier for the socket memory by
> > the commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for
> > socket memory uncharging").
> >
> > Signed-off-by: Roman Gushchin <guro@fb.com>
> > Reported-by: Imran Khan <imran.f.khan@oracle.com>
> 
> I remember seeing this somewhere
> https://lore.kernel.org/linux-mm/20190423154405.259178-1-shakeelb@google.com/

Yes, we've discussed it a couple of times, as I remember. Looks like now
we finally have a good reasoning/benchmark, thanks to Imran.

> 
> Reviewed-by: Shakeel Butt <shakeelb@google.com>

Thank you for the review!

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06  4:22 [PATCH] mm: memcg/slab: optimize objcg stock draining Roman Gushchin
  2021-01-06  6:05 ` Shakeel Butt
  2021-01-06  6:19 ` Imran Khan
@ 2021-01-06 19:50 ` Andrew Morton
  2021-01-06 20:04   ` Roman Gushchin
  2021-01-20 16:34 ` Michal Koutný
  3 siblings, 1 reply; 7+ messages in thread
From: Andrew Morton @ 2021-01-06 19:50 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: linux-mm, Michal Hocko, Johannes Weiner, Shakeel Butt,
	linux-kernel, kernel-team, Imran Khan

On Tue, 5 Jan 2021 20:22:39 -0800 Roman Gushchin <guro@fb.com> wrote:

> Imran Khan reported a regression in hackbench results caused by the
> commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> instead of pages").

How large was the regression?

> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
>  	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
>  		page_counter_uncharge(&memcg->kmem, nr_pages);
>  
> -	page_counter_uncharge(&memcg->memory, nr_pages);
> -	if (do_memsw_account())
> -		page_counter_uncharge(&memcg->memsw, nr_pages);
> +	refill_stock(memcg, nr_pages);
>  }

IOW, which kernel version(s) should we be patching?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06 19:50 ` Andrew Morton
@ 2021-01-06 20:04   ` Roman Gushchin
  0 siblings, 0 replies; 7+ messages in thread
From: Roman Gushchin @ 2021-01-06 20:04 UTC (permalink / raw)
  To: Andrew Morton
  Cc: linux-mm, Michal Hocko, Johannes Weiner, Shakeel Butt,
	linux-kernel, kernel-team, Imran Khan

On Wed, Jan 06, 2021 at 11:50:44AM -0800, Andrew Morton wrote:
> On Tue, 5 Jan 2021 20:22:39 -0800 Roman Gushchin <guro@fb.com> wrote:
> 
> > Imran Khan reported a regression in hackbench results caused by the
> > commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> > instead of pages").
> 
> How large was the regression?

~16% according to Imran's data.

> 
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
> >  	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
> >  		page_counter_uncharge(&memcg->kmem, nr_pages);
> >  
> > -	page_counter_uncharge(&memcg->memory, nr_pages);
> > -	if (do_memsw_account())
> > -		page_counter_uncharge(&memcg->memsw, nr_pages);
> > +	refill_stock(memcg, nr_pages);
> >  }
> 
> IOW, which kernel version(s) should we be patching?

5.9+

Thanks!

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH] mm: memcg/slab: optimize objcg stock draining
  2021-01-06  4:22 [PATCH] mm: memcg/slab: optimize objcg stock draining Roman Gushchin
                   ` (2 preceding siblings ...)
  2021-01-06 19:50 ` Andrew Morton
@ 2021-01-20 16:34 ` Michal Koutný
  3 siblings, 0 replies; 7+ messages in thread
From: Michal Koutný @ 2021-01-20 16:34 UTC (permalink / raw)
  To: Roman Gushchin
  Cc: Andrew Morton, linux-mm, Michal Hocko, Johannes Weiner,
	Shakeel Butt, linux-kernel, kernel-team, Imran Khan

[-- Attachment #1: Type: text/plain, Size: 2286 bytes --]

On Tue, Jan 05, 2021 at 08:22:39PM -0800, Roman Gushchin <guro@fb.com> wrote:
> Imran Khan reported a regression in hackbench results caused by the
> commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
> instead of pages"). The regression is noticeable in the case of
> a consequent allocation of several relatively large slab objects,
> e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
> drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
> to a number of atomic operations in page_counter_uncharge().
> 
> The corresponding call graph is below (provided by Imran Khan):
>   |__alloc_skb
>   |    |
>   |    |__kmalloc_reserve.isra.61
>   |    |    |
>   |    |    |__kmalloc_node_track_caller
>   |    |    |    |
>   |    |    |    |slab_pre_alloc_hook.constprop.88
>   |    |    |     obj_cgroup_charge
>   |    |    |    |    |
>   |    |    |    |    |__memcg_kmem_charge
>   |    |    |    |    |    |
>   |    |    |    |    |    |page_counter_try_charge
>   |    |    |    |    |
>   |    |    |    |    |refill_obj_stock
>   |    |    |    |    |    |
>   |    |    |    |    |    |drain_obj_stock.isra.68   <--- draining old memcg
>   |    |    |    |    |    |    |
>   |    |    |    |    |    |    |__memcg_kmem_uncharge
>   |    |    |    |    |    |    |    |
>   |    |    |    |    |    |    |    |page_counter_uncharge
>   |    |    |    |    |    |    |    |    |
>   |    |    |    |    |    |    |    |    |page_counter_cancel
> [...]
> -	page_counter_uncharge(&memcg->memory, nr_pages);
> -	if (do_memsw_account())
> -		page_counter_uncharge(&memcg->memsw, nr_pages);
> +	refill_stock(memcg, nr_pages);
I noticed that refill_stock(memcg,...) switches the local stock to
memcg.

In this particular call chain, the uncharged memcg is the old memcg of
stock->cached_objcg. The refill_stock() then may switch stock->cached to
the old memcg too. If the patch leads to better performance, then the
switch probably doesn't happen at this moment (and I guess
stock->cached_objcg and stock->cached can be independent to some extent,
so the old memcg in one needn't be the old in the latter).

In conclusion
Reviewed-by: Michal Koutný <mkoutny@suse.com>


Michal

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2021-01-20 16:37 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-06  4:22 [PATCH] mm: memcg/slab: optimize objcg stock draining Roman Gushchin
2021-01-06  6:05 ` Shakeel Butt
2021-01-06 16:42   ` Roman Gushchin
2021-01-06  6:19 ` Imran Khan
2021-01-06 19:50 ` Andrew Morton
2021-01-06 20:04   ` Roman Gushchin
2021-01-20 16:34 ` Michal Koutný

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).