From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f69.google.com (mail-oi0-f69.google.com [209.85.218.69]) by kanga.kvack.org (Postfix) with ESMTP id 262D96B007E for ; Fri, 27 May 2016 04:17:42 -0400 (EDT) Received: by mail-oi0-f69.google.com with SMTP id w143so157503813oiw.3 for ; Fri, 27 May 2016 01:17:42 -0700 (PDT) Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-he1eur01on0091.outbound.protection.outlook.com. [104.47.0.91]) by mx.google.com with ESMTPS id s67si6487010oia.89.2016.05.27.01.17.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 27 May 2016 01:17:41 -0700 (PDT) Date: Fri, 27 May 2016 11:17:32 +0300 From: Vladimir Davydov Subject: Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem() Message-ID: <20160527081732.GB26059@esperanza> References: <20160526203018.GG23194@mtj.duckdns.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20160526203018.GG23194@mtj.duckdns.org> Sender: owner-linux-mm@kvack.org List-ID: To: Tejun Heo Cc: Johannes Weiner , Michal Hocko , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, kernel-team@fb.com On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote: > memcg_offline_kmem() may be called from memcg_free_kmem() after a css > init failure. memcg_free_kmem() is a ->css_free callback which is > called without cgroup_mutex and memcg_offline_kmem() ends up using > css_for_each_descendant_pre() without any locking. Fix it by adding > rcu read locking around it. > > mkdir: cannot create directory a??65530a??: No space left on device > [ 527.241361] =============================== > [ 527.241845] [ INFO: suspicious RCU usage. ] > [ 527.242367] 4.6.0-work+ #321 Not tainted > [ 527.242730] ------------------------------- > [ 527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock required! > [ 527.243970] > [ 527.243970] other info that might help us debug this: > [ 527.243970] > [ 527.244715] > [ 527.244715] rcu_scheduler_active = 1, debug_locks = 0 > [ 527.245463] 2 locks held by kworker/0:5/1664: > [ 527.245939] #0: ("cgroup_destroy"){.+.+..}, at: [] process_one_work+0x165/0x4a0 > [ 527.246958] #1: ((&css->destroy_work)#3){+.+...}, at: [] process_one_work+0x165/0x4a0 > [ 527.248098] > [ 527.248098] stack backtrace: > [ 527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-work+ #321 > [ 527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014 > [ 527.250555] Workqueue: cgroup_destroy css_free_work_fn > [ 527.250555] 0000000000000000 ffff880178747c68 ffffffff8128bfc7 ffff880178b8ac40 > [ 527.250555] 0000000000000001 ffff880178747c98 ffffffff8108c297 0000000000000000 > [ 527.250555] ffff88010de54138 000000000000fffb ffff88010de537e8 ffff880178747cc0 > [ 527.250555] Call Trace: > [ 527.250555] [] dump_stack+0x68/0xa1 > [ 527.250555] [] lockdep_rcu_suspicious+0xd7/0x110 > [ 527.250555] [] css_next_descendant_pre+0x7d/0xb0 > [ 527.250555] [] memcg_offline_kmem.part.44+0x4a/0xc0 > [ 527.250555] [] mem_cgroup_css_free+0x1ec/0x200 > [ 527.250555] [] css_free_work_fn+0x49/0x5e0 > [ 527.250555] [] process_one_work+0x1c5/0x4a0 > [ 527.250555] [] ? process_one_work+0x165/0x4a0 > [ 527.250555] [] worker_thread+0x49/0x490 > [ 527.250555] [] ? process_one_work+0x4a0/0x4a0 > [ 527.250555] [] ? process_one_work+0x4a0/0x4a0 > [ 527.250555] [] kthread+0xea/0x100 > [ 527.250555] [] ret_from_fork+0x1f/0x40 > [ 527.250555] [] ? kthread_create_on_node+0x200/0x200 > > Signed-off-by: Tejun Heo Acked-by: Vladimir Davydov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vladimir Davydov Subject: Re: [PATCH] memcg: add RCU locking around css_for_each_descendant_pre() in memcg_offline_kmem() Date: Fri, 27 May 2016 11:17:32 +0300 Message-ID: <20160527081732.GB26059@esperanza> References: <20160526203018.GG23194@mtj.duckdns.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=NlLwEGAGRpjcrqPqEUtq3KmsJADmCTYVzvKPIq1NVX0=; b=PJ7X4OYKBPiDuB/XQKN4pl6vY13tl1QRRmLxk7cx4smxm61jETRGuZ8gEw+x3dWBRiiWJ/mGQ4Up8Zc+YnNXE7uaHg5W1nhaiV5c0ywDpWre6AQ8dhjgEoNf9BRxGceznD107aAc2WWo+7+3n119ukxWrjfqyYcf1MKIkLO0B28= Content-Disposition: inline In-Reply-To: <20160526203018.GG23194@mtj.duckdns.org> Sender: owner-linux-mm@kvack.org List-ID: To: Tejun Heo Cc: Johannes Weiner , Michal Hocko , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, kernel-team@fb.com On Thu, May 26, 2016 at 04:30:18PM -0400, Tejun Heo wrote: > memcg_offline_kmem() may be called from memcg_free_kmem() after a css > init failure. memcg_free_kmem() is a ->css_free callback which is > called without cgroup_mutex and memcg_offline_kmem() ends up using > css_for_each_descendant_pre() without any locking. Fix it by adding > rcu read locking around it. >=20 > mkdir: cannot create directory =E2=80=9865530=E2=80=99: No space left = on device > [ 527.241361] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > [ 527.241845] [ INFO: suspicious RCU usage. ] > [ 527.242367] 4.6.0-work+ #321 Not tainted > [ 527.242730] ------------------------------- > [ 527.243220] kernel/cgroup.c:4008 cgroup_mutex or RCU read lock requ= ired! > [ 527.243970] > [ 527.243970] other info that might help us debug this: > [ 527.243970] > [ 527.244715] > [ 527.244715] rcu_scheduler_active =3D 1, debug_locks =3D 0 > [ 527.245463] 2 locks held by kworker/0:5/1664: > [ 527.245939] #0: ("cgroup_destroy"){.+.+..}, at: [] process_one_work+0x165/0x4a0 > [ 527.246958] #1: ((&css->destroy_work)#3){+.+...}, at: [] process_one_work+0x165/0x4a0 > [ 527.248098] > [ 527.248098] stack backtrace: > [ 527.249565] CPU: 0 PID: 1664 Comm: kworker/0:5 Not tainted 4.6.0-wo= rk+ #321 > [ 527.250429] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), = BIOS 1.9.1-1.fc24 04/01/2014 > [ 527.250555] Workqueue: cgroup_destroy css_free_work_fn > [ 527.250555] 0000000000000000 ffff880178747c68 ffffffff8128bfc7 fff= f880178b8ac40 > [ 527.250555] 0000000000000001 ffff880178747c98 ffffffff8108c297 000= 0000000000000 > [ 527.250555] ffff88010de54138 000000000000fffb ffff88010de537e8 fff= f880178747cc0 > [ 527.250555] Call Trace: > [ 527.250555] [] dump_stack+0x68/0xa1 > [ 527.250555] [] lockdep_rcu_suspicious+0xd7/0x110 > [ 527.250555] [] css_next_descendant_pre+0x7d/0xb0 > [ 527.250555] [] memcg_offline_kmem.part.44+0x4a/0= xc0 > [ 527.250555] [] mem_cgroup_css_free+0x1ec/0x200 > [ 527.250555] [] css_free_work_fn+0x49/0x5e0 > [ 527.250555] [] process_one_work+0x1c5/0x4a0 > [ 527.250555] [] ? process_one_work+0x165/0x4a0 > [ 527.250555] [] worker_thread+0x49/0x490 > [ 527.250555] [] ? process_one_work+0x4a0/0x4a0 > [ 527.250555] [] ? process_one_work+0x4a0/0x4a0 > [ 527.250555] [] kthread+0xea/0x100 > [ 527.250555] [] ret_from_fork+0x1f/0x40 > [ 527.250555] [] ? kthread_create_on_node+0x200/0x= 200 >=20 > Signed-off-by: Tejun Heo Acked-by: Vladimir Davydov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org