[PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Tejun Heo]

memcg_offline_kmem() may be called from memcg_free_kmem() after a css
init failure.  memcg_free_kmem() is a ->css_free callback which is
called without cgroup_mutex and memcg_offline_kmem() ends up using
css_for_each_descendant_pre() without any locking.  Fix it by adding
rcu read locking around it.

 mkdir: cannot create directory a??65530a??: No space left on device
 [  527.241361] ===============================
 [  527.241845] [ INFO: suspicious RCU usage. ]
 [  527.242367] 4.6.0-work+ #321 Not tainted
 [  527.242730] -------------------------------
 [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
 [  527.243970]
 [  527.243970] other info that might help us debug this:
 [  527.243970]
 [  527.244715]
 [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
 [  527.245463] 2 locks held by kworker/0:5/1664:
 [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
 [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
 [  527.248098]
 [  527.248098] stack backtrace:
 [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
 [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
 [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
 [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
 [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
 [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
 [  527.250555] Call Trace:
 [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
 [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
 [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
 [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
 [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
 [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
 [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
 [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
 [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
 [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
 [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
 [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
 [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
 [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200

Signed-off-by: Tejun Heo <tj@kernel.org>
---
 mm/memcontrol.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index cf428d7..8d42c6d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2892,6 +2892,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 	 * ordering is imposed by list_lru_node->lock taken by
 	 * memcg_drain_all_list_lrus().
 	 */
+	rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */
 	css_for_each_descendant_pre(css, &memcg->css) {
 		child = mem_cgroup_from_css(css);
 		BUG_ON(child->kmemcg_id != kmemcg_id);
@@ -2899,6 +2900,8 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 		if (!memcg->use_hierarchy)
 			break;
 	}
+	rcu_read_unlock();
+
 	memcg_drain_all_list_lrus(kmemcg_id, parent->kmemcg_id);
 
 	memcg_free_cache_id(kmemcg_id);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Tejun Heo]

memcg_offline_kmem() may be called from memcg_free_kmem() after a css
init failure.  memcg_free_kmem() is a ->css_free callback which is
called without cgroup_mutex and memcg_offline_kmem() ends up using
css_for_each_descendant_pre() without any locking.  Fix it by adding
rcu read locking around it.

 mkdir: cannot create directory ‘65530’: No space left on device
 [  527.241361] ===============================
 [  527.241845] [ INFO: suspicious RCU usage. ]
 [  527.242367] 4.6.0-work+ #321 Not tainted
 [  527.242730] -------------------------------
 [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
 [  527.243970]
 [  527.243970] other info that might help us debug this:
 [  527.243970]
 [  527.244715]
 [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
 [  527.245463] 2 locks held by kworker/0:5/1664:
 [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
 [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
 [  527.248098]
 [  527.248098] stack backtrace:
 [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
 [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
 [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
 [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
 [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
 [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
 [  527.250555] Call Trace:
 [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
 [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
 [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
 [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
 [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
 [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
 [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
 [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
 [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
 [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
 [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
 [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
 [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
 [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200

Signed-off-by: Tejun Heo <tj@kernel.org>
---
 mm/memcontrol.c |    3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index cf428d7..8d42c6d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2892,6 +2892,7 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 	 * ordering is imposed by list_lru_node->lock taken by
 	 * memcg_drain_all_list_lrus().
 	 */
+	rcu_read_lock(); /* can be called from css_free w/o cgroup_mutex */
 	css_for_each_descendant_pre(css, &memcg->css) {
 		child = mem_cgroup_from_css(css);
 		BUG_ON(child->kmemcg_id != kmemcg_id);
@@ -2899,6 +2900,8 @@ static void memcg_offline_kmem(struct mem_cgroup *memcg)
 		if (!memcg->use_hierarchy)
 			break;
 	}
+	rcu_read_unlock();
+
 	memcg_drain_all_list_lrus(kmemcg_id, parent->kmemcg_id);
 
 	memcg_free_cache_id(kmemcg_id);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Andrew Morton]

On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj@kernel.org> wrote:

> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory ___65530___: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!

cc:stable?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Andrew Morton]

On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory ___65530___: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!

cc:stable?

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Vladimir Davydov]

On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote:
> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory a??65530a??: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
>  [  527.243970]
>  [  527.243970] other info that might help us debug this:
>  [  527.243970]
>  [  527.244715]
>  [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
>  [  527.245463] 2 locks held by kworker/0:5/1664:
>  [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.248098]
>  [  527.248098] stack backtrace:
>  [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
>  [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
>  [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
>  [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
>  [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
>  [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
>  [  527.250555] Call Trace:
>  [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
>  [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
>  [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
>  [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
>  [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
>  [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
>  [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
>  [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
>  [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
>  [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
>  [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>

Acked-by: Vladimir Davydov <vdavydov@virtuozzo.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Vladimir Davydov]

On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote:
> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory ‘65530’: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
>  [  527.243970]
>  [  527.243970] other info that might help us debug this:
>  [  527.243970]
>  [  527.244715]
>  [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
>  [  527.245463] 2 locks held by kworker/0:5/1664:
>  [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.248098]
>  [  527.248098] stack backtrace:
>  [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
>  [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
>  [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
>  [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
>  [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
>  [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
>  [  527.250555] Call Trace:
>  [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
>  [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
>  [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
>  [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
>  [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
>  [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
>  [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
>  [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
>  [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
>  [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
>  [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>

Acked-by: Vladimir Davydov <vdavydov@virtuozzo.com>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Michal Hocko]

On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj@kernel.org> wrote:
> 
> > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > rcu read locking around it.
> > 
> >  mkdir: cannot create directory ___65530___: No space left on device
> >  [  527.241361] ===============================
> >  [  527.241845] [ INFO: suspicious RCU usage. ]
> >  [  527.242367] 4.6.0-work+ #321 Not tainted
> >  [  527.242730] -------------------------------
> >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> 
> cc:stable?

Also which kernel versions would be affected? I have tried to look and
got lost in the indirection of the css_free path.

Thanks!

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Michal Hocko]

On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > rcu read locking around it.
> > 
> >  mkdir: cannot create directory ___65530___: No space left on device
> >  [  527.241361] ===============================
> >  [  527.241845] [ INFO: suspicious RCU usage. ]
> >  [  527.242367] 4.6.0-work+ #321 Not tainted
> >  [  527.242730] -------------------------------
> >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> 
> cc:stable?

Also which kernel versions would be affected? I have tried to look and
got lost in the indirection of the css_free path.

Thanks!

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Tejun Heo]

On Fri, May 27, 2016 at 05:31:24PM +0200, Michal Hocko wrote:
> On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> > On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj@kernel.org> wrote:
> > 
> > > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > > rcu read locking around it.
> > > 
> > >  mkdir: cannot create directory ___65530___: No space left on device
> > >  [  527.241361] ===============================
> > >  [  527.241845] [ INFO: suspicious RCU usage. ]
> > >  [  527.242367] 4.6.0-work+ #321 Not tainted
> > >  [  527.242730] -------------------------------
> > >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> > 
> > cc:stable?
> 
> Also which kernel versions would be affected? I have tried to look and
> got lost in the indirection of the css_free path.

I think it's actually from 0b8f73e10428 ("mm: memcontrol: clean up
alloc, online, offline, free functions") which got merged during this
cycle, so no need for -stable.

Thanks.

-- 
tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Tejun Heo]

On Fri, May 27, 2016 at 05:31:24PM +0200, Michal Hocko wrote:
> On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> > On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > 
> > > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > > rcu read locking around it.
> > > 
> > >  mkdir: cannot create directory ___65530___: No space left on device
> > >  [  527.241361] ===============================
> > >  [  527.241845] [ INFO: suspicious RCU usage. ]
> > >  [  527.242367] 4.6.0-work+ #321 Not tainted
> > >  [  527.242730] -------------------------------
> > >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> > 
> > cc:stable?
> 
> Also which kernel versions would be affected? I have tried to look and
> got lost in the indirection of the css_free path.

I think it's actually from 0b8f73e10428 ("mm: memcontrol: clean up
alloc, online, offline, free functions") which got merged during this
cycle, so no need for -stable.

Thanks.

-- 
tejun

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Johannes Weiner]

On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote:
> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory a??65530a??: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
>  [  527.243970]
>  [  527.243970] other info that might help us debug this:
>  [  527.243970]
>  [  527.244715]
>  [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
>  [  527.245463] 2 locks held by kworker/0:5/1664:
>  [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.248098]
>  [  527.248098] stack backtrace:
>  [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
>  [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
>  [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
>  [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
>  [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
>  [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
>  [  527.250555] Call Trace:
>  [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
>  [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
>  [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
>  [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
>  [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
>  [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
>  [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
>  [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
>  [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
>  [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
>  [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200
> 
> Signed-off-by: Tejun Heo <tj@kernel.org>

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Johannes Weiner]

On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote:
> memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> init failure.  memcg_free_kmem() is a ->css_free callback which is
> called without cgroup_mutex and memcg_offline_kmem() ends up using
> css_for_each_descendant_pre() without any locking.  Fix it by adding
> rcu read locking around it.
> 
>  mkdir: cannot create directory ‘65530’: No space left on device
>  [  527.241361] ===============================
>  [  527.241845] [ INFO: suspicious RCU usage. ]
>  [  527.242367] 4.6.0-work+ #321 Not tainted
>  [  527.242730] -------------------------------
>  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
>  [  527.243970]
>  [  527.243970] other info that might help us debug this:
>  [  527.243970]
>  [  527.244715]
>  [  527.244715] rcu_scheduler_active = 1, debug_locks = 0
>  [  527.245463] 2 locks held by kworker/0:5/1664:
>  [  527.245939]  #0:  ("cgroup_destroy"){.+.+..}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.246958]  #1:  ((&css->destroy_work)#3){+.+...}, at: [<ffffffff81060ab5>] process_one_work+0x165/0x4a0
>  [  527.248098]
>  [  527.248098] stack backtrace:
>  [  527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321
>  [  527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
>  [  527.250555] Workqueue: cgroup_destroy css_free_work_fn
>  [  527.250555]  0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40
>  [  527.250555]  0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000
>  [  527.250555]  ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0
>  [  527.250555] Call Trace:
>  [  527.250555]  [<ffffffff8128bfc7>] dump_stack+0x68/0xa1
>  [  527.250555]  [<ffffffff8108c297>] lockdep_rcu_suspicious+0xd7/0x110
>  [  527.250555]  [<ffffffff810ca03d>] css_next_descendant_pre+0x7d/0xb0
>  [  527.250555]  [<ffffffff8114d14a>] memcg_offline_kmem.part.44+0x4a/0xc0
>  [  527.250555]  [<ffffffff8114d3ac>] mem_cgroup_css_free+0x1ec/0x200
>  [  527.250555]  [<ffffffff810ccdc9>] css_free_work_fn+0x49/0x5e0
>  [  527.250555]  [<ffffffff81060b15>] process_one_work+0x1c5/0x4a0
>  [  527.250555]  [<ffffffff81060ab5>] ? process_one_work+0x165/0x4a0
>  [  527.250555]  [<ffffffff81060e39>] worker_thread+0x49/0x490
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff81060df0>] ? process_one_work+0x4a0/0x4a0
>  [  527.250555]  [<ffffffff810672ba>] kthread+0xea/0x100
>  [  527.250555]  [<ffffffff814cbcff>] ret_from_fork+0x1f/0x40
>  [  527.250555]  [<ffffffff810671d0>] ? kthread_create_on_node+0x200/0x200
> 
> Signed-off-by: Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

Acked-by: Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Michal Hocko]

Sorry for a late response.

On Fri 27-05-16 11:51:40, Tejun Heo wrote:
> On Fri, May 27, 2016 at 05:31:24PM +0200, Michal Hocko wrote:
> > On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> > > On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj@kernel.org> wrote:
> > > 
> > > > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > > > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > > > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > > > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > > > rcu read locking around it.
> > > > 
> > > >  mkdir: cannot create directory ___65530___: No space left on device
> > > >  [  527.241361] ===============================
> > > >  [  527.241845] [ INFO: suspicious RCU usage. ]
> > > >  [  527.242367] 4.6.0-work+ #321 Not tainted
> > > >  [  527.242730] -------------------------------
> > > >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> > > 
> > > cc:stable?
> > 
> > Also which kernel versions would be affected? I have tried to look and
> > got lost in the indirection of the css_free path.
> 
> I think it's actually from 0b8f73e10428 ("mm: memcontrol: clean up
> alloc, online, offline, free functions") which got merged during this
> cycle, so no need for -stable.

yes you are right! memcg_free_kmem didn't call memcg_offline_kmem before
that commit. Thanks for the clarification.

Anyway
$ git describe --contains 0b8f73e10428
v4.5-rc1~30^2~11

So it would be stable # 4.5+
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem()

[Michal Hocko]

Sorry for a late response.

On Fri 27-05-16 11:51:40, Tejun Heo wrote:
> On Fri, May 27, 2016 at 05:31:24PM +0200, Michal Hocko wrote:
> > On Thu 26-05-16 14:02:02, Andrew Morton wrote:
> > > On Thu, 26 May 2016 16:30:18 -0400 Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > > 
> > > > memcg_offline_kmem() may be called from memcg_free_kmem() after a css
> > > > init failure.  memcg_free_kmem() is a ->css_free callback which is
> > > > called without cgroup_mutex and memcg_offline_kmem() ends up using
> > > > css_for_each_descendant_pre() without any locking.  Fix it by adding
> > > > rcu read locking around it.
> > > > 
> > > >  mkdir: cannot create directory ___65530___: No space left on device
> > > >  [  527.241361] ===============================
> > > >  [  527.241845] [ INFO: suspicious RCU usage. ]
> > > >  [  527.242367] 4.6.0-work+ #321 Not tainted
> > > >  [  527.242730] -------------------------------
> > > >  [  527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required!
> > > 
> > > cc:stable?
> > 
> > Also which kernel versions would be affected? I have tried to look and
> > got lost in the indirection of the css_free path.
> 
> I think it's actually from 0b8f73e10428 ("mm: memcontrol: clean up
> alloc, online, offline, free functions") which got merged during this
> cycle, so no need for -stable.

yes you are right! memcg_free_kmem didn't call memcg_offline_kmem before
that commit. Thanks for the clarification.

Anyway
$ git describe --contains 0b8f73e10428
v4.5-rc1~30^2~11

So it would be stable # 4.5+
-- 
Michal Hocko
SUSE Labs