From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f197.google.com (mail-qk0-f197.google.com [209.85.220.197]) by kanga.kvack.org (Postfix) with ESMTP id AFA556B0005 for ; Wed, 10 Aug 2016 15:43:09 -0400 (EDT) Received: by mail-qk0-f197.google.com with SMTP id g67so101517438qkf.1 for ; Wed, 10 Aug 2016 12:43:09 -0700 (PDT) Received: from mail-qk0-x244.google.com (mail-qk0-x244.google.com. [2607:f8b0:400d:c09::244]) by mx.google.com with ESMTPS id 2si27736880qtv.138.2016.08.10.12.43.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Aug 2016 12:43:08 -0700 (PDT) Received: by mail-qk0-x244.google.com with SMTP id v123so885997qkh.3 for ; Wed, 10 Aug 2016 12:43:08 -0700 (PDT) Date: Wed, 10 Aug 2016 15:43:06 -0400 From: Tejun Heo Subject: Re: [RFC][PATCH] cgroup_threadgroup_rwsem - affects scalability and OOM Message-ID: <20160810194306.GP25053@mtj.duckdns.org> References: <4717ef90-ca86-4a34-c63a-94b8b4bfaaec@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4717ef90-ca86-4a34-c63a-94b8b4bfaaec@gmail.com> Sender: owner-linux-mm@kvack.org List-ID: To: Balbir Singh Cc: cgroups@vger.kernel.org, Oleg Nesterov , Andrew Morton , "linux-mm@kvack.org" Hello, Edited subject and description and applied the patch to cgroup/for-4.8-fixes w/ stable cc'd. Thanks. ------ 8< ------ From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tejun Heo Subject: Re: [RFC][PATCH] cgroup_threadgroup_rwsem - affects scalability and OOM Date: Wed, 10 Aug 2016 15:43:06 -0400 Message-ID: <20160810194306.GP25053@mtj.duckdns.org> References: <4717ef90-ca86-4a34-c63a-94b8b4bfaaec@gmail.com> Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=sQ6uhI4u03E6pKTG1PJiti6hisYkXUx6AxXU6LSzOVs=; b=ef7/zQjMWeNgHYsb26PGTy0doDWB8TIaHMjK5YQpk5aQl/2ZD9nsORyN4D8m6XZxik tqceT9D9FV/vbGB0rJOIgywBWtxYJBIccYQ/G/F57YMne/b08PNU3QnudmXUsYCNSvxK cAMF3b3rwn9m9gmoyjEjstJXPVhpVg+I5rhAuTp3fSR0HwI2pk1h+5TPknAz8n4ffOaQ F19rXXnou1y8kcpy/hpgQFvrNuE3iAD/ST9NWuYxFrV8y6EVcSajCIVU+c1Km525prl1 FjTYdU71EpkbIHdVJSzaxe8nGvNSaiUvGYi+f64OiR7Sb9DBcb7hAQ+/RYTPo+ofGJo2 UFhw== Content-Disposition: inline In-Reply-To: <4717ef90-ca86-4a34-c63a-94b8b4bfaaec@gmail.com> Sender: owner-linux-mm@kvack.org List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Balbir Singh Cc: cgroups@vger.kernel.org, Oleg Nesterov , Andrew Morton , "linux-mm@kvack.org" Hello, Edited subject and description and applied the patch to cgroup/for-4.8-fixes w/ stable cc'd. Thanks. ------ 8< ------ >From b570e91aaa563673dc6cca58c14388d68c767353 Mon Sep 17 00:00:00 2001 From: Balbir Singh Date: Tue, 9 Aug 2016 14:19:01 +1000 Subject: [PATCH] cgroup: reduce read locked section of cgroup_threadgroup_rwsem during fork cgroup_threadgroup_rwsem is acquired in read mode during process exit and fork. It is also grabbed in write mode during __cgroups_proc_write(). I've recently run into a scenario with lots of memory pressure and OOM and I am beginning to see systemd __switch_to+0x1f8/0x350 __schedule+0x30c/0x990 schedule+0x48/0xc0 percpu_down_write+0x114/0x170 __cgroup_procs_write.isra.12+0xb8/0x3c0 cgroup_file_write+0x74/0x1a0 kernfs_fop_write+0x188/0x200 __vfs_write+0x6c/0xe0 vfs_write+0xc0/0x230 SyS_write+0x6c/0x110 system_call+0x38/0xb4 This thread is waiting on the reader of cgroup_threadgroup_rwsem to exit. The reader itself is under memory pressure and has gone into reclaim after fork. There are times the reader also ends up waiting on oom_lock as well. __switch_to+0x1f8/0x350 __schedule+0x30c/0x990 schedule+0x48/0xc0 jbd2_log_wait_commit+0xd4/0x180 ext4_evict_inode+0x88/0x5c0 evict+0xf8/0x2a0 dispose_list+0x50/0x80 prune_icache_sb+0x6c/0x90 super_cache_scan+0x190/0x210 shrink_slab.part.15+0x22c/0x4c0 shrink_zone+0x288/0x3c0 do_try_to_free_pages+0x1dc/0x590 try_to_free_pages+0xdc/0x260 __alloc_pages_nodemask+0x72c/0xc90 alloc_pages_current+0xb4/0x1a0 page_table_alloc+0xc0/0x170 __pte_alloc+0x58/0x1f0 copy_page_range+0x4ec/0x950 copy_process.isra.5+0x15a0/0x1870 _do_fork+0xa8/0x4b0 ppc_clone+0x8/0xc In the meanwhile, all processes exiting/forking are blocked almost stalling the system. This patch moves the threadgroup_change_begin from before cgroup_fork() to just before cgroup_canfork(). There is no nee to worry about threadgroup changes till the task is actually added to the threadgroup. This avoids having to call reclaim with cgroup_threadgroup_rwsem held. tj: Subject and description edits. Signed-off-by: Balbir Singh Acked-by: Zefan Li Cc: Oleg Nesterov Cc: Andrew Morton Cc: stable@vger.kernel.org # v4.2+ Signed-off-by: Tejun Heo --- kernel/fork.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 52e725d..aaf7823 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1404,7 +1404,6 @@ static struct task_struct *copy_process(unsigned long clone_flags, p->real_start_time = ktime_get_boot_ns(); p->io_context = NULL; p->audit_context = NULL; - threadgroup_change_begin(current); cgroup_fork(p); #ifdef CONFIG_NUMA p->mempolicy = mpol_dup(p->mempolicy); @@ -1556,6 +1555,7 @@ static struct task_struct *copy_process(unsigned long clone_flags, INIT_LIST_HEAD(&p->thread_group); p->task_works = NULL; + threadgroup_change_begin(current); /* * Ensure that the cgroup subsystem policies allow the new process to be * forked. It should be noted the the new process's css_set can be changed @@ -1656,6 +1656,7 @@ static struct task_struct *copy_process(unsigned long clone_flags, bad_fork_cancel_cgroup: cgroup_cancel_fork(p); bad_fork_free_pid: + threadgroup_change_end(current); if (pid != &init_struct_pid) free_pid(pid); bad_fork_cleanup_thread: @@ -1688,7 +1689,6 @@ static struct task_struct *copy_process(unsigned long clone_flags, mpol_put(p->mempolicy); bad_fork_cleanup_threadgroup_lock: #endif - threadgroup_change_end(current); delayacct_tsk_free(p); bad_fork_cleanup_count: atomic_dec(&p->cred->user->processes); -- 2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org