From: Johannes Weiner <hannes@cmpxchg.org> To: Waiman Long <llong@redhat.com> Cc: Michal Hocko <mhocko@kernel.org>, Vladimir Davydov <vdavydov.dev@gmail.com>, Andrew Morton <akpm@linux-foundation.org>, Tejun Heo <tj@kernel.org>, Christoph Lameter <cl@linux.com>, Pekka Enberg <penberg@kernel.org>, David Rientjes <rientjes@google.com>, Joonsoo Kim <iamjoonsoo.kim@lge.com>, Vlastimil Babka <vbabka@suse.cz>, Roman Gushchin <guro@fb.com>, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt <shakeelb@google.com>, Muchun Song <songmuchun@bytedance.com>, Alex Shi <alex.shi@linux.alibaba.com>, Chris Down <chris@chrisdown.name>, Yafang Shao <laoar.shao@gmail.com>, Wei Yang <richard.weiyang@gmail.com>, Masayoshi Mizuma <msys.mizuma@gmail.com>, Xing Zhengjun <zhengjun.xing@linux.intel.com> Subject: Re: [PATCH v3 5/5] mm/memcg: Optimize user context object stock access Date: Thu, 15 Apr 2021 14:53:15 -0400 [thread overview] Message-ID: <YHiLmxE9oCOfmbS3@cmpxchg.org> (raw) In-Reply-To: <8dbd3505-9c51-362f-82d8-5efa5773e020@redhat.com> On Thu, Apr 15, 2021 at 02:16:17PM -0400, Waiman Long wrote: > On 4/15/21 1:53 PM, Johannes Weiner wrote: > > On Tue, Apr 13, 2021 at 09:20:27PM -0400, Waiman Long wrote: > > > Most kmem_cache_alloc() calls are from user context. With instrumentation > > > enabled, the measured amount of kmem_cache_alloc() calls from non-task > > > context was about 0.01% of the total. > > > > > > The irq disable/enable sequence used in this case to access content > > > from object stock is slow. To optimize for user context access, there > > > are now two object stocks for task context and interrupt context access > > > respectively. > > > > > > The task context object stock can be accessed after disabling preemption > > > which is cheap in non-preempt kernel. The interrupt context object stock > > > can only be accessed after disabling interrupt. User context code can > > > access interrupt object stock, but not vice versa. > > > > > > The mod_objcg_state() function is also modified to make sure that memcg > > > and lruvec stat updates are done with interrupted disabled. > > > > > > The downside of this change is that there are more data stored in local > > > object stocks and not reflected in the charge counter and the vmstat > > > arrays. However, this is a small price to pay for better performance. > > > > > > Signed-off-by: Waiman Long <longman@redhat.com> > > > Acked-by: Roman Gushchin <guro@fb.com> > > > Reviewed-by: Shakeel Butt <shakeelb@google.com> > > This makes sense, and also explains the previous patch a bit > > better. But please merge those two. > The reason I broke it into two is so that the patches are individually > easier to review. I prefer to update the commit log of patch 4 to explain > why the obj_stock structure is introduced instead of merging the two. Well I did not find them easier to review separately. > > > @@ -2327,7 +2365,9 @@ static void drain_local_stock(struct work_struct *dummy) > > > local_irq_save(flags); > > > stock = this_cpu_ptr(&memcg_stock); > > > - drain_obj_stock(&stock->obj); > > > + drain_obj_stock(&stock->irq_obj); > > > + if (in_task()) > > > + drain_obj_stock(&stock->task_obj); > > > drain_stock(stock); > > > clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); > > > @@ -3183,7 +3223,7 @@ static inline void mod_objcg_state(struct obj_cgroup *objcg, > > > memcg = obj_cgroup_memcg(objcg); > > > if (pgdat) > > > lruvec = mem_cgroup_lruvec(memcg, pgdat); > > > - __mod_memcg_lruvec_state(memcg, lruvec, idx, nr); > > > + mod_memcg_lruvec_state(memcg, lruvec, idx, nr); > > > rcu_read_unlock(); > > This is actually a bug introduced in the earlier patch, isn't it? > > Calling __mod_memcg_lruvec_state() without irqs disabled... > > > Not really, in patch 3, mod_objcg_state() is called only in the stock update > context where interrupt had already been disabled. But now, that is no > longer the case, that is why i need to update mod_objcg_state() to make sure > irq is disabled before updating vmstat data array. Oh, I see it now. Man, that's subtle. We've had several very hard to track down preemption bugs in those stats, because they manifest as counter imbalances and you have no idea if there is a leak somewhere. The convention for these functions is that the __ prefix indicates that preemption has been suitably disabled. Please always follow this convention, even if the semantic change is temporary. Btw, is there a reason why the stock caching isn't just part of mod_objcg_state()? Why does the user need to choose if they want the caching or not? It's not like we ask for this when charging, either.
WARNING: multiple messages have this Message-ID (diff)
From: Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org> To: Waiman Long <llong-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Cc: Michal Hocko <mhocko-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Vladimir Davydov <vdavydov.dev-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Andrew Morton <akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Christoph Lameter <cl-vYTEC60ixJUAvxtiuMwx3w@public.gmane.org>, Pekka Enberg <penberg-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, David Rientjes <rientjes-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>, Joonsoo Kim <iamjoonsoo.kim-Hm3cg6mZ9cc@public.gmane.org>, Vlastimil Babka <vbabka-AlSwsSmVLrQ@public.gmane.org>, Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org>, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, Shakeel Butt <shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>, Muchun Song <songmuchun-EC8Uxl6Npydl57MIdRCFDg@public.gmane.org>, Alex Shi <alex.shi-KPsoFbNs7GizrGE5bRqYAgC/G2K4zDHf@public.gmane.org>, Chris Down <chris-6Bi1550iOqEnzZ6mRAm98g@public.gmane.org>, Yafang Shao <laoar.shao-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Wei Yang <richard.weiyang-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Masayoshi Mizuma <msys.mizuma-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>, Xing Zhengjun <zhengjun.xing-VuQAYsv1563Yd54FQh9/CA@public.gmane.org> Subject: Re: [PATCH v3 5/5] mm/memcg: Optimize user context object stock access Date: Thu, 15 Apr 2021 14:53:15 -0400 [thread overview] Message-ID: <YHiLmxE9oCOfmbS3@cmpxchg.org> (raw) In-Reply-To: <8dbd3505-9c51-362f-82d8-5efa5773e020-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> On Thu, Apr 15, 2021 at 02:16:17PM -0400, Waiman Long wrote: > On 4/15/21 1:53 PM, Johannes Weiner wrote: > > On Tue, Apr 13, 2021 at 09:20:27PM -0400, Waiman Long wrote: > > > Most kmem_cache_alloc() calls are from user context. With instrumentation > > > enabled, the measured amount of kmem_cache_alloc() calls from non-task > > > context was about 0.01% of the total. > > > > > > The irq disable/enable sequence used in this case to access content > > > from object stock is slow. To optimize for user context access, there > > > are now two object stocks for task context and interrupt context access > > > respectively. > > > > > > The task context object stock can be accessed after disabling preemption > > > which is cheap in non-preempt kernel. The interrupt context object stock > > > can only be accessed after disabling interrupt. User context code can > > > access interrupt object stock, but not vice versa. > > > > > > The mod_objcg_state() function is also modified to make sure that memcg > > > and lruvec stat updates are done with interrupted disabled. > > > > > > The downside of this change is that there are more data stored in local > > > object stocks and not reflected in the charge counter and the vmstat > > > arrays. However, this is a small price to pay for better performance. > > > > > > Signed-off-by: Waiman Long <longman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> > > > Acked-by: Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org> > > > Reviewed-by: Shakeel Butt <shakeelb-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> > > This makes sense, and also explains the previous patch a bit > > better. But please merge those two. > The reason I broke it into two is so that the patches are individually > easier to review. I prefer to update the commit log of patch 4 to explain > why the obj_stock structure is introduced instead of merging the two. Well I did not find them easier to review separately. > > > @@ -2327,7 +2365,9 @@ static void drain_local_stock(struct work_struct *dummy) > > > local_irq_save(flags); > > > stock = this_cpu_ptr(&memcg_stock); > > > - drain_obj_stock(&stock->obj); > > > + drain_obj_stock(&stock->irq_obj); > > > + if (in_task()) > > > + drain_obj_stock(&stock->task_obj); > > > drain_stock(stock); > > > clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); > > > @@ -3183,7 +3223,7 @@ static inline void mod_objcg_state(struct obj_cgroup *objcg, > > > memcg = obj_cgroup_memcg(objcg); > > > if (pgdat) > > > lruvec = mem_cgroup_lruvec(memcg, pgdat); > > > - __mod_memcg_lruvec_state(memcg, lruvec, idx, nr); > > > + mod_memcg_lruvec_state(memcg, lruvec, idx, nr); > > > rcu_read_unlock(); > > This is actually a bug introduced in the earlier patch, isn't it? > > Calling __mod_memcg_lruvec_state() without irqs disabled... > > > Not really, in patch 3, mod_objcg_state() is called only in the stock update > context where interrupt had already been disabled. But now, that is no > longer the case, that is why i need to update mod_objcg_state() to make sure > irq is disabled before updating vmstat data array. Oh, I see it now. Man, that's subtle. We've had several very hard to track down preemption bugs in those stats, because they manifest as counter imbalances and you have no idea if there is a leak somewhere. The convention for these functions is that the __ prefix indicates that preemption has been suitably disabled. Please always follow this convention, even if the semantic change is temporary. Btw, is there a reason why the stock caching isn't just part of mod_objcg_state()? Why does the user need to choose if they want the caching or not? It's not like we ask for this when charging, either.
next prev parent reply other threads:[~2021-04-15 18:53 UTC|newest] Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-04-14 1:20 [PATCH v3 0/5] mm/memcg: Reduce kmemcache memory accounting overhead Waiman Long 2021-04-14 1:20 ` Waiman Long 2021-04-14 1:20 ` [PATCH v3 1/5] mm/memcg: Pass both memcg and lruvec to mod_memcg_lruvec_state() Waiman Long 2021-04-14 1:20 ` Waiman Long 2021-04-15 3:27 ` Masayoshi Mizuma 2021-04-15 3:27 ` Masayoshi Mizuma 2021-04-15 16:40 ` Johannes Weiner 2021-04-15 16:40 ` Johannes Weiner 2021-04-15 16:59 ` Waiman Long 2021-04-15 16:59 ` Waiman Long 2021-04-16 15:48 ` Johannes Weiner 2021-04-16 15:48 ` Johannes Weiner 2021-04-14 1:20 ` [PATCH v3 2/5] mm/memcg: Introduce obj_cgroup_uncharge_mod_state() Waiman Long 2021-04-14 1:20 ` Waiman Long 2021-04-15 3:27 ` Masayoshi Mizuma 2021-04-15 3:27 ` Masayoshi Mizuma 2021-04-15 16:30 ` Johannes Weiner 2021-04-15 16:30 ` Johannes Weiner 2021-04-15 16:35 ` Waiman Long 2021-04-15 16:35 ` Waiman Long 2021-04-15 18:10 ` Johannes Weiner 2021-04-15 18:10 ` Johannes Weiner 2021-04-15 18:47 ` Waiman Long 2021-04-15 18:47 ` Waiman Long 2021-04-15 19:40 ` Johannes Weiner 2021-04-15 19:40 ` Johannes Weiner 2021-04-15 19:44 ` Waiman Long 2021-04-15 20:19 ` Johannes Weiner 2021-04-15 20:19 ` Johannes Weiner 2021-04-14 1:20 ` [PATCH v3 3/5] mm/memcg: Cache vmstat data in percpu memcg_stock_pcp Waiman Long 2021-04-14 1:20 ` Waiman Long 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 16:50 ` Johannes Weiner 2021-04-15 17:08 ` Waiman Long 2021-04-15 17:08 ` Waiman Long 2021-04-15 18:13 ` Johannes Weiner 2021-04-15 18:13 ` Johannes Weiner 2021-04-14 1:20 ` [PATCH v3 4/5] mm/memcg: Separate out object stock data into its own struct Waiman Long 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 16:57 ` Johannes Weiner 2021-04-15 16:57 ` Johannes Weiner 2021-04-14 1:20 ` [PATCH v3 5/5] mm/memcg: Optimize user context object stock access Waiman Long 2021-04-14 1:20 ` Waiman Long 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 3:28 ` Masayoshi Mizuma 2021-04-15 9:44 ` Christoph Lameter 2021-04-15 9:44 ` Christoph Lameter 2021-04-15 12:16 ` Masayoshi Mizuma 2021-04-15 12:16 ` Masayoshi Mizuma 2021-04-15 17:53 ` Johannes Weiner 2021-04-15 18:16 ` Waiman Long 2021-04-15 18:16 ` Waiman Long 2021-04-15 18:53 ` Johannes Weiner [this message] 2021-04-15 18:53 ` Johannes Weiner 2021-04-15 19:06 ` Waiman Long 2021-04-15 19:06 ` Waiman Long 2021-04-15 3:26 ` [PATCH v3 0/5] mm/memcg: Reduce kmemcache memory accounting overhead Masayoshi Mizuma 2021-04-15 3:26 ` Masayoshi Mizuma 2021-04-15 13:17 ` Waiman Long 2021-04-15 13:17 ` Waiman Long 2021-04-15 15:47 ` Masayoshi Mizuma 2021-04-15 15:47 ` Masayoshi Mizuma 2021-04-15 17:10 ` Matthew Wilcox 2021-04-15 17:10 ` Matthew Wilcox 2021-04-15 17:41 ` Waiman Long 2021-04-15 17:41 ` Waiman Long
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=YHiLmxE9oCOfmbS3@cmpxchg.org \ --to=hannes@cmpxchg.org \ --cc=akpm@linux-foundation.org \ --cc=alex.shi@linux.alibaba.com \ --cc=cgroups@vger.kernel.org \ --cc=chris@chrisdown.name \ --cc=cl@linux.com \ --cc=guro@fb.com \ --cc=iamjoonsoo.kim@lge.com \ --cc=laoar.shao@gmail.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=llong@redhat.com \ --cc=mhocko@kernel.org \ --cc=msys.mizuma@gmail.com \ --cc=penberg@kernel.org \ --cc=richard.weiyang@gmail.com \ --cc=rientjes@google.com \ --cc=shakeelb@google.com \ --cc=songmuchun@bytedance.com \ --cc=tj@kernel.org \ --cc=vbabka@suse.cz \ --cc=vdavydov.dev@gmail.com \ --cc=zhengjun.xing@linux.intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.